This is a rather odd traffic arrangement. You came up Upper Ground, and passed Stage Door. Then realising your mistake, you turned round and drove back, but the street is open to all traffic in one direction, but traffic from the other direction just after passing under Waterloo Bridge, is restricted to local buses. So you would have been unaware of the traffic restriction until you turned round and drove back to get to Stage Door.
A difficult one to call, so wait a bit until others have commented. However, if you take them to London Tribunals, I don't think you have a powerful appeal, although you could point out there is no warning at all that the street is essentially one-way for all traffic bar local buses.
