It could be several things;
1. Your pump cannot provide adequate pressure.
2. You have a leak in the pipe between the pump and the pressure tank allowing the line to drain between your pump and house when the pump is off. This would account for your psi dropping to 20 before recovering and the slow recovery.
3. The foot valve on your pump is failing, allowing the water to drain back out of the line between you house and the pump. This would also account for your psi dropping to 20 before recovering and the slow recovery.
#2 and #3 are easily masked by having a backflow preventer before your pressure tank. If you have one, remove it. You don't need it. If there is a leak because of #2 or #3 the pressure reading at the pressure tank will start dropping even with all the water in the house shut off.
How old and what type of pump? Is this a drilled well? How deep?