EEA Family permit applies only to spouse, and to dependent childern under the age of 21, and not to other relatives.
If you are already supporting your mother and 27 years old brother outside of the UKI or Ireland, why can't you just continue doing that? The EEA family permit route is not a way to bring other relatives into the UK, as I, who am an Irish citizen, found out when I was hunting for a way to bring my 24 years old Chinese step-daughter here, after she finished university in China, which I funded.
So, the basis on which they can refuse to give EEA faily permits to your mother and brother is simply because neither of them is entitled.

 

I wouldn't agree with EEA Family Permit being only for spouse and children as there is a provision of 'extended family members' in the directive. But yes, it will be an uphill task.

Good luck !