The sign on the opposite side of the road is irrelevant to your case, because each side of a road is separate from the other side when restrictions are imposed. Of course restrictions can be identical, but often aren't, like here. The reason the other side has a sign is because the restricted hours are different to those on the CPZ signs you see on entry to the zone. Councils are allowed to create CPZ, where the hours on the entry signs define the single-yellow line times, but also put up signs within the zone if they want to vary those times in certain locations. For me it is bad practice, but it is legal.
At the moment, based on your narrative, you seem bang-to-rights. Did you drive past the CPZ sign in your photograph, or come down the other way and turn left ?