Midtown and CP are a tough call...are you looking for vinyl exclusivly or just a store to browse buy all formats?

http://www.vinylhunt.com/stores?utf8=%E2%9C%93&search=10001

is a good start for indies and stuff...

here is a FB app you can put on your phone:
https://www.facebook.com/tvdrecordstorelocator

here is the Record Store Day site...:

http://www.recordstoreday.com/Home

Generation Records in the West Village is great.  It's where I purchased Jersey Devil  as well as the great Zep boot Flying Circus.  Plenty to rummage through, including a full wall of DVDs - where I picked up Still Life in Seattle. Vinyl in the basement and don't forget to lookat what's on display behind the counter.  On the surface it's a Metal shop, but they carry everything and the tattooed staff are helpful.  I stop by every time I'm in NYC.

http://www.generationrecords.com/

For 17 years, Generation Records has been NYC’s best source for new and used vinyl and CDs, 45s, DVDs, posters and t-shirts. While we are known for our extensive metal and punk collection, you’ll find many different genres on our shelves. You’ll find something you want every time you come into our store.
 
We are located at 210 Thompson St. between w. 3rd and Bleecker St. in the heart of the West Village. We are conveniently located near the W.4th subway stop downtown.

Just go downtown, as mentioned above, in the Greenwich Village there are some GREAT stores