Can in-floor heat mats (low voltage) be safely used under vinyl or lino sheeting. If so, what would be the procedure after mat install to get a smooth surface for flooring? Some sort of self-leveling compound? Directly over the heat mat?
Self-leveling cement (SLC) or gypsum based product (Gypcrete) would work. Both need some minimum depth over a wooden subfloor (slc is typically 1/2" over the highest point), and it's easier to get flat if you go deeper rather than just try to cover things (stronger, too). SLC requires installation of a primer, and may be fussier than a gypsum product, but either should work. Gypcrete or other similar gypsum based products are fairly soft, and removing the vinal down the road (assuming it is glue-down) would likely end up with gouges. SLC is much harder, and would normally allow tile or other flooring if you removed the vinal.
Keep in mind that these products, when installed properly, do actually level the floor as well as flatten it, so you may need more depth than you initally thought to cover everything properly to the minumum depth and transitions to other areas may be askew, if they are not leveled, too.
Important note - I'm not a pro
Retired Defense Industry Engineer; Schluter 2.5-day Workshop Completed 2013