The decision has to be yours. The "effort" of fighting it, I find, tends to be mine and other advisors on the forum rather than yours.
If you are asking what would I do in similar circumstances, then that is a different matter and personally, I would always fight these charges. However, it is not my money that I am risking here if you decide at some stage that the pressure is too much and you want to just pay to make it all go away.