Perhaps your resident who offered you duff advice will pay
It would be useful to see the signage at the site. One argument you can run is that the signage is prohibitive, insofar as it does not offer parking to those who are not permit holders (subject to us seeing the wording). The inherent downside of this approach is that neither the operator nor the second stage appeals service are likely to uphold an appeal on this basis, so you would need to see if they initiated a court claim and seek to defend it on this basis.