I noticed someone already posted about a few in Nashville, but here's a little update:
Grimey's and Grimey's Too
1604 8th Ave S Nashville, TN 37203
(615) 254-4801 they recently split into two stores that are right next to each other -- Grimey's carries new records, CDs, and tapes, while Grimey's Too has used records, CDs, books, and a little coffee shop.
The Groove
1103 Calvin Ave Nashville, TN 37206
(615) 227-5760
smaller than Grimey's, but a really good selection - new and used. it's also in East Nashville, which is a hip area - there's a lot of other places around you can walk to.
Phonoluxe Records
2609 Nolensville Pike Nashville, TN 37211
(615) 259-3500
All used, only open Fri-Sun. Primarily a "collector's record store" - so some records are outrageously priced, but they have an amazing bargain bin (it runs on the bottom shelf of almost half the store) and a relatively good selection of newer stuff at a discount. Also have CDs & DVDs.
Also, if you're even half-heartedly interested in country music, I highly recommend Ernest Tubb Record Shop (there's one on lower Broadway, and one outside the Opry complex). They have a very limited amount of LPs, but their CD selection is incredible...lot of rare re-issues and imports at good prices.