Sand em both. Sand the glass in a circular motion and with the striker, do it like you're lighting a match. I found that after you sand, sometimes putting a little chalk on the glass will make it sound good TOO.
You can sand both, but take it easy on the striker. I use a conditioning stone on the glass or crystal, and use scotch brite on the striker. I also have some very very fine grit sandpaper that I'll lightly sand the striker with if needed. You just want to be careful not to change the tip of the striker too much with a rough sandpaper.