Some adhesives will stain stone. Some of the lighter stones will show an uneven coverage you often get with application of mastic type stuff by looking mottled or even show the trowel mark ridges or beads of adhesive. It's a good idea to backbutter stone to make sure you get full coverage so it doesn't show the notch marks from the trowel. Thinset would be cheaper if you have some left. I'd burn in a coat on the back then apply more and notch it, then install without puting any on the wall first.
You might need to lightly sand and then wipe off the wall to remove any crud and get a little tooth to allow the thing to bond.
If you have a scrap piece and want to make the experiment, try a piece on a scrap piece of drywall and let it set for a couple of days.
Personally, I don't think the thinset would be any slower than the other stuff.
