The problem is not loosing your Residence Card (which should only happen after a lengthy absence), but qualifying for Permanent Residence and British Citizenship. For PR you are allowed to be out of the UK for 3 months every year. The BC the conditions depend on the circumstances, and they may very well change by the time you qualify. 3 months per year may be ok, or it may just be 6 weeks. Longer absences may be ok if you have a very good reason (usually business or health related).