In a Nassau tournament, the player (or team) winning the front nine wins a prize, the player (or team) winning the back nine gets a prize, and the player or team with the low 18-hole total wins a prize.
As a bet, the most common form is the $2 Nassau. The front nine is worth $2, the back nine is worth $2 and the 18-hole total is worth $2. A player or team sweeping all three wins $6.
The type of scoring is really up the individuals. Stroke play or match play? Scramble, alternate shot, best ball? Full handicaps, partial handicaps, no handicaps? You decide. It's not like there are "official" rules for this sort of thing.
While the $2 Nassau sounds innocent enough, winnings can pile up if a higher initial bet is made, or if a lot of "pressing" takes place.
A player or team that is trailing in a Nassau can "press the bet" - opening a new bet to run concurrently with the bet that has been pressed. A Nassau that has been pressed and re-pressed and double-pressed and pressed even more can wind up costing someone a lot of money.

