To be fully effective and minimize air-transported moisture the batts need to be snugged up to both the XPS and the gypsum, and it needs to be air-tight. If there's still a gap between the studs and foam you might want to use gap-filler 1-part expanding foam to fill it at each stud (and at the top plate) to make the cavities air tight, and use unfaced R19s to fill up the more-than-3.5" depth. R13 batts have the same amount of material as R19 batts, just lower loft. If you compress an R19 fully into a true-3.5" cavity it'll be R13. If you actually have 4" or 4.5" it'll be higher R. But if you leave a gap without a snug air-barrier air convects randomly in & out of the R13 batt, and it's performance is undercut substantially. Any where that air can move unimpeded is a thermal-bypass, and a likely contributor to the problem.
You may have been experiencing humid summertime outdoor air infiltration into the gap between the batts & foundation wall too, or if you had kraft faced batts with the facer on the conditioned-space side it may have been trapping ground moisture in there, causing condensation on the interior side of the cavity. Foam-sealing the foundation sill & band joist to limit air movement could be a necessary part of the fix.
Only unfaced batts will do- the facers are too vapor retardent, and the cavity can't dry to the air-conditioned Latex paint on air-tight drywall has about 6-10x the drying capacity of a kraft facer. At 2" XPS still has 1.5-2x the drying capacity of a kraft facer, and it's on the humid side of the assembly in summer, when most of the issues are likely to occur. With latex on the finish surface and XPS on the foundation side, moisture passes more readily out of the studwall toward the interior. In winter the moisture drives are the other direction, but the R-value of the XPS is sufficient to keep all of the wood & fiberglass above the dew point of the interior air (unless you keep it down right tropical by adding a lot of humidity with a humidifier- not recommended.)
Going with a mold-resistant drywall is also moving in the right direction.
Getting serious about air-sealing both the basement and the attic will vastly reduce the infiltration drives that are probably contributing to the problem, and lower both the heating and cooling bills to boot. With lower infiltration rates in to the basement the sweating of AC ducts will also be much reduced- the dew point of the basement air will then track that of the fully conditioned upstairs, which is the same as the air INSIDE those ducts. Under those conditions the ducts won't/can't be much be lower in temperature than the dew point of the basement air.
If you don't have a poly vapor barrier under the slab, you can reduce the amount of groundwater vapor diffusion through the slab with masonry sealers. (On a 4 y.o. house it SHOULD have poly under the slab though, but just because it should doesn't mean that it does, always.)