If I understood correct, you are in Pakistan and your wife is in Latvia, there is unfortunately no way you can get a family visa to enter UK. She has to be either resident in UK exercising her treaty rights ( working, studying, running some business or having a lot of money ) so you can join her or she has to be in Pakistan ( again with a lot of money ) so you can accompany her to UK but that will be a lot of headache.

The best and easiest way for you would be to get visa to Latvia first and from there you can get EEA Fanily Permit for UK ( not sure about Ireland ).
There are of course many conditions for that but its still easier than the other two options that I mentioned above.

Good luck

 

As Yousafzai said.

Put simply, you have no right of residence in any EU Member State unless your wife is resident there.