Assassins
Their color is black
"You will see the black banner of the Caste of Assassins"
Tarnsman of Gor, page 175

Bakers
Their colors are yellow and brown
"I stayed for four days in the rooms above the shop of Dina of Turia, there I dyed my hair black and exchanged the robes of the merchant for the yellow and brown tunic of the Bakers; to which caste her father and two brothers had belonged."
Nomads of Gor, page 237

Bleachers
"...its libraries, its records and files; its cubicles for Smiths, Bakers, Cosmeticians, Bleachers, Dyers, Weavers, and Leather Workers..."
Assassin of Gor, page 111

Cosmeticians
"...its libraries, its records and files; its cubicles for Smiths, Bakers, Cosmeticians, Bleachers, Dyers, Weavers, and Leather Workers..."
Assassin of Gor, page 111

Charcoal Makers
"His stature and burden proclaimed him a member of the Caste of Carriers of Wood, or Woodsmen, that Gorean caste which, with the caste of Charcoal Makers, provides most of the common fuel for the Gorean cities."
Outlaw of Gor, page 27

Cloth Workers 
"The Carders and Dyers, incidentally, are subcastes separate from the Weavers. All are subcastes of the Rug Makers, which itself, interestingly, perhaps surprisingly, is accounted generally as a subcaste of the Cloth Workers. Rug Makers themselves, however usually regard themselves in their various subcastes, as being independent of the Cloth Workers. A rug maker would not care to be confused with a maker of kaftans, turbans, or djellabas."
Tribesmen of Gor, pages 49-50

"I was the daughter of a Cloth Worker..."
Assassin of Gor, page 317

Dyers
"...its libraries, its records and files; its cubicles for Smiths, Bakers, Cosmeticians, Bleachers, Dyers, Weavers, and Leather Workers..."
Assassin of Gor, page 111

Goat Keepers
"I was pleased to see that the men of other castes....and even castes as low as the Peasants, Saddle-Makers, the Weavers, the Goat-Keepers, the Poets, and the Merchants...."
Priest-Kings of Gor, pages 294-295

Leather Workers
"It was there that she had been purchased by Barus, of the Leather Workers."
Captive of Gor, page 234

"...its libraries, its records and files; its cubicles for Smiths, Bakers, Cosmeticians, Bleachers, Dyers, Weavers, and Leather Workers..."
Assassin of Gor, page 111

Magicians
"The magicians of Anango are famed on Gor. If you wish to have someone turned into a turtle or something, those are the fellows to see."
Magicians of Gor, page 259

Merchants
Their colors are gold and white
"Down the stairway slowly, in trailing white silk, bordered with gold, the colors of the Merchants... ...Aphris of Turia, then, was of the caste of merchants."
Nomads of Gor, page 91

"I wore a white robe, woven of the wool of the Hurt, imported from distant Ar, trimmed with golden cloth, from Tor, the colors of the Merchant."
Hunters of Gor, page 7

Metal Workers
"My opponent was not Andreas, but a squat, powerful man with short-clipped yellow hair, Kron of Tharna, of the Caste of Metal Workers."
Outlaw of Gor, page 113

Musicians
"A similar decree, in most cities, stands against the enslavement of one who is in the Caste of Musicians."
Beasts of Gor, page 44

Peasants
"Economically, the base of the Gorean life was the free peasant, which was perhaps the lowest but undoubtedly the most fundamental caste..."
Tarnsman of Gor, page 43

Players
Their colors are red and yellow
"In most cities it is regarded, incidentally, as a criminal offense to enslave one of the Caste of Players."
Beasts of Gor, page 44

"...he wore the garb of the Player, but his garb was rich and the squares of the finest red and yellow silk..."
Assassin of Gor, page 322

Poets
"It could have been worse, I thought. After all, though the Caste of Singers, or Poets, was not a high caste, it had more prestige than, for example, the Caste of Pot-Makers or Saddle-Makers, with which it was sometimes compared."
Outlaw of Gor, page 103

Pot-Makers(Potters)
"Hup's rag might once have been of the Caste of Potters."
Assassin of Gor, page 10

Saddle-Makers
(See Poets for quote mentioning the existence of the Caste of Saddle-Makers)

Rencers
"The Rencers are often thought to be a higher caste than the Peasants."
Raiders of Gor, page 94

Rope-Makers
This is more of a guild, or union, not specifically a caste and are mentioned here as exceptions to the caste system.
"These organizations, such as the sail-makers, almost guildlike, not castes, have dues...."
Raiders of Gor, page 134

Slavers
Their colors are blue and yellow (sub caste of the Merchants)
"I saw that the cover of the thalarion wagon, which had been rolled back, was of blue and yellow silk. It was the camp of a slaver."
Outlaw of Gor, page 185

"The Slavers, incidentally, are of the Merchant caste, though, in virtue of their merchandise and practices, their robes are different."
Assassin of Gor, page 208

"Samos wore the blue and yellow robes of the Slaver."
Hunters of Gor, page 7

"From among the domed tents, wearing a swirling robe of broadly striped blue and yellow silk, with a headband of the same material, there approached a short, fat man, Targo the Slaver..."
Outlaw of Gor, page 191

Sleen Trainers: Their colors are brown and black (may be a sub castes or a specialty)
"He now no longer wore the brown and black common to professional sleen trainers..."
Beasts of Gor, page 78

Smiths
"...its libraries, its records and files; its cubicles for Smiths, Bakers, Cosmeticians, Bleachers, Dyers, Weavers, and Leather Workers..."
Assassin of Gor, page 111

Tarn Keepers
"...was purchased by a fat, odious fellow, of the Caste of Tarn Keepers."
Tarnsman of Gor, page 132

Thieves
"There is even, in Port Kar, a recognized Caste of Thieves, the only such I know of on Gor...They are recognized by the Thiefs Scar, which they wear as a caste mark, a tiny, three-pronged brand burned into the face, in back of and below the eye, over the right cheekbone."
Raiders of Gor, page 104

"On his right cheek, over the cheekbone was the Thief brand of the Caste of Thieves of Port Kar, who use the small brand to identify their members."
Assassin of Gor, page 96

Vintners
Their colors are white and green (to represent leaves)
"...of the Caste of Vintners...wearing a white tunic with a representation in green cloth of leaves about the collar and down the sleeves of the garment..."
Assassin of Gor, page 29

Weavers:
"I was pleased to see that the men of other castes....and even castes as low as the Peasants, Saddle-Makers, the Weavers, the Goat-Keepers, the Poets, and the Merchants...."
Priest-Kings of Gor, pages 294-295

Wood Carriers/Woodsmen
"His stature and burden proclaimed him a member of the Caste of Carriers of Wood, or Woodsmen, that Gorean caste which, with the caste of Charcoal Makers, provides most of the common fuel for the Gorean cities."