What did the 'swimming complex' management say when you complained to them about their agent? That is Plan A.
As you have submitted an appeal, you correctly did not identify the driver. If you are happy to pay just £20, then you can respond by refusing to nae the driver as they say they are relying on PoFA to be able to hold the Keeper liable. That ignores the fact that their Notice to Keeper (NtK) is not fully compliant with ALL the requirements of PoFA, specifically para 9(2)(e)(i). However, they would disagree and it would only be a judge who could decide the point.
If you ignore and don't respond, they will simply reject the appeal and you can try your luck at POPLA. Again, unlikely to succeed. However, a POPLA decision is not binding on you and you can fight this all the way to a county court claim, where you are more likely to succeed than fail. If ParkingEye farm any claim out to DCB Legal, you will know that they do not have any confidence that they could win in court. Any claim issued by DCB Legal, as long as it is defended, will eventually be discontinued. They rely on the low-hanging fruit on the gullible tree to pay up out of ignorance and fear.