I would start by getting a remote-reading humidity measuring device. They are often combined with a remote-reading digital thermometer. Put it various places. If you find more than 60% RH for more than a brief time, consider ventilating. Heck, get 2 and put them on separate channels if they are cheap enough.
I am thinking if you are describing a house over a cellar. Measure humidity there. Maybe a heavy polyethylene sheet on the floor there would help. Also, adding a plastic drain to drain from behind the bottom of your retaining wall to the downhill part of the yard could be worthwhile.