All modern shower valves are required to have anti-scald technology. On most of them, they accomplish this with a spool valve that relies on there being equal supply pressure on both the hot and cold. I think what's happening is that because the tankless is restricting the flow, the valve's spool valve is responding and limiting the overall flow. To work well, that type of valve needs equal pressure on both sides, and the tankless system is restricting it.
I ran some quick calculations, and that unit should be able to raise the incoming water temp about 70-degrees at a flow rate of 2.5gpm, the volume for one typical showerhead. Depending on where you live, that may not be enough temperature rise - in the winter, my incoming water is barely above freezing, and to achieve 120-degree output, it would have to throttle back the volume quite a bit - those extra 18-20 degrees of temperature rise just wouldn't cut it and be unachievable at the desired volume, let alone trying to run two at once, or fill the tub or washing machine in a timely manner.
In the wintertime (and maybe all year if you have a deep well and the supply is cold), a tankless of that sort is optimally good for one fixture at a time. Anything more, and it would likely be unsatisfactory. You can run the numbers yourself: 1Kw/hr = 3415BTU/hr and it takes 1BTU to raise one pound of water (a gallon is about 7.6#) one degree.
You might try lowering the set temp to just above the desired shower temperature, this would put less load on the thing and allow it to flow more, keeping the hot/cold supply pressures more equal.
Throw in the minimum flow rate to trigger the thing on, and the pressure variances, and potential variations in incoming cold water, I've not had a great comfort level with the things.