No, a pilot light cannot heat a WH to excessive temps. Where I live, they require all WH to have a tempering valve...you may want to consider adding one. This is adjustable, and adds cold water, if required, so the WH output never exceeds that setting. Normally, it is set to 119-120 degrees, or so. This is still hot, but at the upper end of safe. If it can be done easily, sometimes people will run untempered water to the dishwasher. This is especially important if the thing doesn't have a water heater built-in as they just don't work well with cooler water.
It sounds like the thermostat is messed up. Or, it could be as simple as the sensor is loose or not inserted properly into its sensing well.