It refers more to the overall state of the vehicle. It is a rough looking car that you don't feel bad about banging around in, for fear of scratching paint, etc.
Some guys will make their bangers very powerful and reliable mechanically, have good steering, electrical and brakes, etc., but have the body and upholstery look like they just pulled the car out of a barn, where it had been sitting for 50 years. Kind of a fun look.