There are small differences between a British National and a British citizen.

On a British passport the explanation is:

You will get the full information about British citizenship and nationality on the Home Office website. It all comes from the British Nationality Act 1981 (which came into force on 1 January 1983) when Margaret Thatcher was Prime Minister.

The forms of British nationality are:
- British citizenship;
- British overseas citizenship;
- British overseas territories citizenship;
- British national (overseas);
- British protected person; and
- British subject.

As your mother was born in Gibraltar she must lay within the category: British overseas territories citizen and therefore it seems to me that the passport should actually say either British overseas territories citizen (Gibraltar is in the list of the British overseas territories as defined by the Act) or British citizen (see below with the British Overseas Territories Act 2002).
The category British national (overseas) or BN(O) was created by the Hong Kong Act 1985

118 countries and territories grant British National (Overseas) passport holders either visa-free or visa-on-arrival treatment as of 26 March 2010, but I can see why this is a problem, as the United States are not part of the list!
You have 2 options (you might want to do both, just in case!):
- use the Premium service to get a new passport (cost £128). In order to get more information and maybe also ask them to deal urgently with the matter you can call them: Telephone: 0300 222 0000
All information here: https://www.gov.uk/get-a-passport-urgently
- contact the US Embassy on that matter.

In addition, BOTCs by connection with Gibraltar may apply for registration as a British citizen under section 5 of the 1981 Act. This is an entitlement and cannot be refused.
And anyway, the British Overseas Territories Act 2002 (c.8) superseded parts of the British Nationality Act 1981 says that:

 

Re: Urgent advice needed

It is not because they are from the Border Agency that they are right unfortunately and I can understand your frustration.
At the end there is already a confusion as a BN is a term for Hong Kong, not Gibraltar. She should be British overseas territories citizen in any case. And anyway, since the 20012 act, she should become British Citizen.
But as for any big administration (public and private companies they are all the same on that) it can take a long time and stamina. Finger crossed for the answer this afternoon, keep us informed.

That is why I also advised that you contact the US embassy to see if you can get a Visa or some sort of paper so you can still go on holiday.

 

Arch! I am sorry that you have those difficulties sorting out the passport. Does that definitely prevent you to travel to the US? What is your plan now?