The directive 2004/38C applies to Family members of citizens of those Schengen countries who make use of their right to freely travel and stay in the European Union.
As your partner is married to an Irish and you live and work in the UK, he can obtain a residence card of a "family member of EEA national" which would grant you stay for example in France, Croatia or Italy together for vacation without a visa (as you are irish and not British, you make use of the right of freely stay in other countries).
The visa issue is free of charge and you can make use of the simplified application process which means that he does not have to provide proof of employment or of financial status. Initially issued it lasts for 90 days within a six months period; having been issued several Schengen visa for the same Schengen country your partner can also get a longer valid visa, however the number of days of stay are limited to 90 days within every 6 months.
I am talking about Schengen visas only. Just get the Schengen visa for your partner (based on the status of family member) and he will have no problem to travel freely within the EU.
What I am asking is, Does my husband need a Schengen Visa when go on holiday together to Europe, taking into consideration the fact that I have an Irish passport and he is on an EEA Family Permit.
As I said above, under Directive 2004/38C, a family member of an EU/EEA national does not require a visa if he/she qualifies as a family member of a Union citizen to whom the provisions of this directive apply (EEA permit...)
But what did you mean by 'Just get the Schengen visa for your partner (based on the status of family member) and he will have no problem to travel freely within the EU.'
He will need a Shenghen visa but he will have no problem to get it based and EEA status and being married with you and travelling together (there is often even an accelerate procedure for those cases).