The name "dots" or "dot game" derives from the fact that golfers mark their scorecards with dots to denote having won one of the side bets on any given hole. At the end of the round, each player counts up his dots - each of which is worth a set amount - and the players settle up.
So, what side bets are involved in dots? As stated above, that's entirely up to the members of the group. But among the most popular dots bets are greens in regulation, closest to the pin, scoring par or better on a hole, long drive, sandies, Arnies (a k a Seves) and 1-putts.
On each hole, a golfer achieving one of the agreed-upon "dots" marks a dot on his scorecard, in the same box where he writes his score for that hole.
There are a seemingly endless variety of ways to play dots, and many groups add in double points, triple points and more for more elaborate bets (e.g., making a birdie one hole after you hit a par-5 green in two might earn you triple dots). Again, the variations are up to the members of the group.
This type of game goes by several other names, too, the most common of which is Garbage.