Take the diverter stem out again and look at it again. There is probably a washer on the end of the stem, but hold the whole assembly in one hand and with the handle holding the stem, turn the handle backwards and note where it seats itself when it's all of the way back. Some times it's brass to brass beveled surfaces and sometimes its with a washer of some sort. Or the washer could be missing.
You say the end washer looks good then there has to be a seating surface.
What is the brand of faucet and how old is it?