Low flow rate

I have converted my house to County water and now want to use my well for irrigation and filling my pond. However, I am not getting enough water from the well.
The details: Well is 15 years old, drilled 65' pump set at 54', 6" galvanized casing, 4" pvc casing inside 6" casing, 1" pvc downpipe to submersible pump inside 4" pvc, original draw 20-25 gpm per well drilling report.
Static level 29', I can draw 5.5 gpm continuously and the level will drop to 49'. when I stop the pump the well will recover to 29' within 10 minutes.
I suspect there is a restriction but I don't know the purpose of the 4" pvc inside the 6" casing. What can I do? Should I pull the 4" pvc to see if that increases the flow rate? Is the correct method of pulling the downpipe and pump, to screw a fitting with a hook into the top of the pitless adaptor and attach a cable to pull the assembly out?
Thanks for all help

is the 6" part the actual well casing or is it a pitless adapter. you may have a pvc with a pitless adapter or twenty feet of steel surface casing.

There is a 6" steel casing with a 1" pitless adaptor 17" from the top of the casing, going through the side of the casing. A 1" pvc down pipe goes from the pitless adaptor through a 4" pvc pipe that begins 35" from the top of the casing.