Math: If you have 20 farms and 20 woodmills at L9 then they will need 18'000 workers. L9 cottages with a tax at 100% gives 40'500 citizens. Basicly you will be able to have workers and idle population for your military with 9 cottages only.
My suggestion is to rebuild your city, tare down some cottages if you have "to many" and build barracks instead, maybe even granaries for protection against plunder.
