No, the 2.5" leathers will not go into the 2" pipe. That is why I was saying to go up to 3". I have no idea how a pitless adapter works for a jet pump, we don't even use pitless for 4" wells here so I can't help you much on that.
Since you reduce down to 2" you will have to install all the pipe first with the over-size leathers and then add the 2.5"x2" bell reducer at the end. It is tricky because you have to hold up the pipe with something while you slip it over, then release the clamp while you connect the 2.5" bell reducer, then install the well head. Hard to describe the process and much easier with a pump hoist.
Leathers should be easy to find, look at a site like Dean Bennet or such. Go ahead and order a new goulds jet, probably a FP2B07 for medium water levels. Install the cup leathers over the ones that come with the jet.
Good luck.