More importantly, the garrison sallies out against your 20-stack and they have Handgunners, Outriders and Crossbowmen which I find all difficult to deal with with the Lizardmen early game (Skink skirmishers just seem to get zoned out by 1 unit of Empire Swordsmen on each flank).

Unless you go for a lightning quick battle with New World Colonies, they fortify very quickly. I play on Very Hard/Hard most of the times.