I would suggest you start reading a lot of audiophile magazines and websites. Every element of your system will make a change in how your stereo sounds. I haven’t purchased an audio system myself yet but did go through an experience of a friend who spent more than $100k US. You can get very detailed, technical, and specific about what type of sound you’re looking for.
While I’m far from being able to afford it, and you may think why would such a thing be necessary, just think about how live music would sound–imagine drums, guitars, and someone’s voice; what would it be like if you’re really there. You won’t get anything close until you spend a ton of money.
As an example, I would suggest reading a review of what so many people believe to be the best amplifier ever made, the Halcro dm58 (the dm68 is really the best). I’m sure none of us will be getting it but it’ll give you an idea of what is possible.
halcro.com/
Then back to the real world, I know there’s a budget audiophile review website, goodsound.com I’d read that a lot. There are probably many other sources too.
Myself, I just use my iPod and a pair of Shure E3 earphones (about $200). Now that I’ve started to really pay attention to music accuracy, I can tell the sound is really not that great. There’s noise, distortion, lack of detail, and lack of reality. There are colorations that wouldn’t be there if I had better equipment.
Big deal or not? At the very minimum, bad sound will fatigue and irritate you. Imagine listening to static on TV or a radio. It would really irritate you if you had to listen to it for a long time. Imagine a very minimal amount, small enough that very few will detect. That will still bother you even if you’re unaware.
So I would pick everything very very carefully whenever I get around to building a system. If you ever look in a store for any component, everything else that’s connected to it will affect how it sounds. So it’s really really difficult to test an individual component in isolation without control over everything else.