Brock,
It is all just about perception because hurricanes category 2 and below cause fairly minor damage so they don't count and the damage from a bigger storm is very isolated. I know a broker that lives 25 miles from Punta Gorda and is about 500 yards from the Gulf. He had great pictures of the fury of the storm, but short of a broken window and a messed up yard, sustained no damage and the neighborhood was exactly the same.
In classes we were told tornadoes and major snowstorms cause about 10 times the annual damage that hurricanes do. The perception is different because hurricanes are predicted in advance so people sweat the storm for days and huge swaths of land are warned, yet maybe 20% of those warned have to really worry about the effects. Further once you get as little as a mile inland, there is virtually no water damage so as long as you aren't in a poorly constructed building you have little chance of sustaining major damage.