The proper way is to take it all apart and reglue.
However, Rectorseal and others make a fibreglass pipe repair wrap. Comes in a little kit for about $14. A professional plumber would probably never use this, but in your application, it will probably outlive the house. Remove as much of the caulk as you possibly can, and make sure to overlap the caulked area back to "clean" pipe with the wrap, because the caulk residue may cause a local bond problem at that spot.