A leaf is a programming term referring to a data structure, computer programming talk. Think of it like a tree. Leafs are at the end. Since programmers write the majority of eBay's error messages that sorta talk gets in there a lot.
In terms of what that means on eBay it means you must pick an end level category. For example you can't just choose "Antiques" for your category. That is not a "leaf."
Antiques->Ethonogrpaphic->African->Masks is an example of a leaf category. eBay does not let you choose either Antiques->Ethnographic or even Antiques->Ethonographic->African as a category.