It's OK not to want to race on the street, but I do not think that the "not worthy" response is always merited. If you want to claim superiority, you have to prove it. You never know what the other guy has, and there are plenty of tame looking cars that are heavily modified under the hood...or tacky ricers with the speed to back up their gratuitous body kits. It's fine to ignore a challenge, but to pretend you would have automatically smoked him is hubris.
To use a Mopar based example: Would you laugh if a Reliant revved on you? You might be tempted. But it might be this guy:
http://www.thedodgegarage.com/