I wish it didn't. I wish we were all perfectly evolved beings where looks weren't a concern. For me and for a number of my guy friends, the height aspect doesn't matter too much. I remember one friend saying that he was a bit hesitant to go on a date with a girl he met because she was a good five inches taller than him. The advice my sage BFF came up with in that moment? "Climb her like a tree." Us guys will make jokes along those lines, but at the end of the day, we aren't too hung up on whether a woman is a bit taller than us.
To invert the issue, somehow height difference isn't such a big deal when it comes to shorter women. In our society, the dynamic of a shorter woman with a taller man seems expected, so even if a woman is six inches shorter than a guy it's not given much thought. Short girls: you're in the clear.
Now, I'm 5'7" or 5'8" depending on what mood I'm in. The doctor was right: it's never been a hiccup when it comes to dating. But there's an interesting dynamic for women right around my height, where if they put on heels they are taller than me, flats a bit shorter. I rarely notice, and in investigating this fact with my group of men they don't either. However, the women often do.
The short answer is that men don't prefer taller or shorter women. We prefer women who get us, who are sexy, who are funny, and charming, and smart. Height falls pretty low on the list of what makes or breaks a deal for us. That's not to say there are exceptions to this rule.
Based on my panel of men, I've come to a realization: the height dynamic is actually more important to the women in our lives. If you put on heels and all of a sudden feel like you're towering over your man and that makes you uncomfortable? Maybe wear smaller heels. As one friend so aptly put it, it's the women in our lives "who seem to be mostly interested in guys who are tall - not just taller but tall." It is rare to find a man who simply has to go one way or the other on the height debate. With the prevalence of online dating these days we're going to see a lot more people show up to meet each other and say, "Huh. I thought you would be taller." Let's break convention: let's sit down, get to know the person, and only when we've determined the chemistry is right, start worrying about height. Or better yet, let the wedding photographer worry about it and you two just get on enjoying one another.