From your description you were not technically "using" the phone, which is one of the criteria for a mobile phone offence. You'd need to argue this in court, and be able to cast sufficient doubt on the police officer's testimony, e.g. that they were "mistaken" in thinking the phone was in use. It won't be easy. You could be done for Not in proper control instead, but again they would need to be persuaded to drop the more serious charge.
If you lose, it'll still be only 6 points, but a larger fine than the fixed penalty you'll likely be offered. You won't be banned (unless you're on 6 or more points already).