Unlimit the followers.
Seriously. Make flowers unlimited in number you can HAVE, just limited in number you can use at a time..... You can have THOUSANDS of Brahmin, dogs, slaves, but only USE say two Brahmin or 2 dogs or 2 slaves at any given time, based on party points. the rest have to stay at your tent or mines or base or whatever (using ONLY the "Stay here and wait for me" command).
Except, of course, for the regenerating flowers. They can be limited.