I want to make compact mode work but it's only shows like 10 clothing items per category.

Depending on the age of the sim in CAS, the amount of items can differ.
How many rows do you get normally without compact mode, please compare both with like female, all-day, top?

F.i. if you normally have 10 rows, each line is one item showing 3 various colour styles in standard form,
in compact form you will see only 4 rows in compact form, each icon is a different item, you have to click to see the other coloured variants.

So if you see 10 different items in compact form, this would be 30 item rows in normal form.

Please check this or do you miss something else?

Are you by any chance using Lazy Duchess' Smooth Patch, in particular the latest version with the MC Integration file? This is a known issue with that mod in its current version.