A tankless system is designed for producing potable hot water instead of a conventional water heater. A boiler is designed to provide heat to a dwelling. A side benefit is the ability to heat a tank of water using a zone and an indirect water heater tank.
A tankless system, because it is heating water as it passes through and then dumped down the drain, won't get the water hot enough for many space heating purposes. A boiler, because it can recirculate the water is able to produce much hotter water, typically required for radiators.