Now with the GSV link, I measured the distance the sign is away from your house, and it's 50 yards, 45.93 metres. This is not an unreasonable distance away. Frankly, I am astonished (1) you didn't understand the significance ofthe yellow line, and (2) that you haven't been penalised before.
So it comes down to 'legitimate expectation' as I indicated previously. Other thing to check up on is whether everybody else is getting PCNs for parking like yourself. I think it is clearly time to make your local councillor aware of this unexpected enforcement. It is possible somebody has complained due to your car occupying rather too much room on the footway. In London, they would have served a PCN for parking off-carriageway, as it has been an offence there since the 70s, but this is not the case where you are, so they served the next best thing, which is for the yellow line.