It's so easy to get bogged down in minutiae that the bigger picture is lost.
So OP, you posted 'I didn't reply and I never revealed who drove the car to them' from which it's possible to infer that something which gave you the option to reply was received.
But what you're saying is that the first letter you received - addressed as per your V5C-was the follow-up dated 5 Dec. which notified you that it was too late to appeal.
In which case the rest is process without any substantive issues being examined.
If you weren't the driver, which is clearly implied, then the only way that you could be held liable is as the keeper, to do which they would need to have served a compliant Notice to Keeper.
But this is not in evidence.