XPS is rated for ground contact, but must be covered with an interior-side thermal-barrier such as half-inch wallboard or plywood to meet fire codes.
The other issue will be ground moisture. Burying a a sheet of 6-mil poly in the dirt under the gravel works (it must be protected from puncture to STAY working. Sloping it away from the foundation is also a good idea, should it ever get bulk-incursions of water. If you don't put in ground vapor retarders when you seal & insulate the walls, in Westchester the mold & mildew season in that space would be 8-9 months out of the year!
To do it at least half-right would involve a poly vapor retarder under a rat-slab (non-structural thin concrete slab that keeps varmints from burrowing in to the now-cozy & dry shed-room), with at least 2" of XPS on the exterior behind the lattice with at least 1/2" ply on the interior through-screwed to the structural members. Use duct-mastic to seal the vapor retarder to the XPS before putting up the plywood,and at where it meets the exterior if the house. Air-seal with spray foam or cobble'n'cut XPS foam-sealed at the edges & seams any of the gaps at the top of the wall to block thermal bypasses of air through the joists, and use a reasonably weatherstripped door or hatch as access. Rat slabs don't take a lot of concrete- an inch or stwo will do, and it'll protect the ground vapor retarder from puncture. For small rooms this is an afternoon of hand-mixing the concrete in a wheelbarrow, slopping it in there, and floating it roughly level. If it's over a hundred square feet do the math, and after you've already placed & sealed the XPS & ground vapor retarder order up the necessary cubic yardage for a 2" slab and have it delivered.