I purge my CD collection every so often and donate the ones I don't want to my local library. There's no reason to keep CDs I won't listen to again (this has mostly been cancelled out by having Spotify.)
For records, I never want more than 200-250 (about 4 squares on a typical record shelf.) I no longer buy records just to buy them. I only get albums I love or when there's no other physical medium (sometimes from bands on bandcamp.)