If you ignore Smart you will get letters from debt collectors followed by a legal Letter of Claim.
If you appeal within their deadlines, they will reject your appeal and allow you to appeal to the IAS who will also reject your appeal, followed by debt collectors and a Letter of Claim.
In either case you will eventually need to file an appeal to a court claim, so if you appeal now you have the basis for a subsequent appeal and set out your position.