What kind of volume are you trying to get at the shower/tub? A tankless system will drop its outlet temp if you exceed its ability to produce hot water.
Regardless of the size of the supply pipe, the internal restrictions on the tankless may mean that the (likely higher) volume available on the cold side is disrupting the pressure balance function inside the valve ( if it has one). Some thermostatic valves have both the temp control, and internal pressure balance safety functions. If the available volume is not the same on both the hot and cold sides, it can shut down. It may not be the valve's fault, but it may not be a good match to your supply.
Make sure that you have the hot/cold lines hooked up properly to the valve (not swapped).