You said the dehumidifier 'vents' to the outside. If this isn't just moisture, then it's decreasing the pressure in there, pulling in outside (or interior) air.
I'm assuming that the ground has a cover on it and it's got a vapor barrier either in it or under it.
Relative humidity is not the same as actual humidity per volume...colder air can't hold as much moisture as warmer air, so the crawlspace being cooler could have exactly the same actual moisture in it as the house. If you put a couple foot square sheet of plastic on the floor of the crawlspace and seal the edges as best you can, does it have any trapped moisture underneath it? If so, you need to seal things up better.