Insulating the walls still makes a difference in both the heat load and comfort levels, especially the top half. But unlike the above grade portion the heat loss doesn't track the hourly outside temperature due to the thermal mass and insulating effects of the soil. The heat loss roughly tracks the weekly or sometimes monthly average outdoor temp, but there are many variables (soil type and moisture levels, etc.), and the U-factor is also less than 1/3 that of exposed above grade foundation. From a financial point of view there is still a long-term return on putting R12-R15 on the foundation walls in your climate. See the table 2 p10 of
this document, and read the first chapter.
Maryland is in US climate zone 4, where they're recommending R15 for basement walls. To hit that in a way that won't create mold conditions in either winter or summer the portion of the wall stackup in contact with the wall must be foam, but the R-value can be fattened up with a batt-insulated non-structural studwall trapping the foam to the foundation. R4 of foam is sufficient to control wintertime condensation within a 2x4 studwall in your climate, which is cheaply done with 1" of unfaced EPS (compressed bead board- like cheap coolers, etc), at about 40 cents per square foot, sometimes less. Unfaced EPS is sufficiently permeable to water vapor that it still allows the foundation to dry toward the interior, which is important in some cases to keep the foundation sill from rotting, especially when there is minimal above grade exposure that would otherwise let the ground moisture dry to the exterior (which sounds like your case.) You could also use 3/4" or 1" XPS (pink, blue, green) but it's usually more expensive and has less drying capacity. Using foil faced iso may be risky when you have very little above grade drying capacity unless you have a metal or EPDM capillary break between the foundation sill and concrete. (That was my prior recommendation, but I'm pulling back on that since you're saying "99% below grade".)
If you already finished it out with an uninsulated studwall butted up against the concrete you could still insulate it with non-expanding injection foam with relatively low mold risk. But just the gypsum and trapped air-films adds another R0.5 to the wall, reducing the U-factor a bit. Without some real insulation you'll still probably have to heat it to stay comfortable though.
A cheap double-pane basement window has a U-factor of about 0.5-0.6, which is half that of an exposed concrete wall. A low-E version typically runs U0.32-0.36. But the total square footage is usually SO minimal that it hardly matters. Most 1000' basements would have less than 15 square feet of window, and at a delta-T of 50 F and U-factor of 0.5 that's (50 x 0.5 x 15= ) 225 BTU/hr of window losses. (A sleeping human emits more heat than that!) Even 65 watts of lighting (say, one 2-tube T8 fluorescent fixture) is enough to offset the window loss.
Inverter drive mini-splits usually come with remote controls, and NOT a wall mounted thermostat. The interior air temp is sensed by the wall-cassette (and the incoming air when the blower is running.) Since they are fully modulating systems that run at highest efficiency at the low end of their blower & compressor speed, you're much better off with a "set and forget" strategy than using programmed setbacks, since it'll run at full blast (and lowest efficiency) on the recovery ramps, using more power than you saved. Turning them off (or way down) when you're going away for days still makes sense though.
There are probably at least a dozen very good 3/4 ton mini-splits out there, and a handful of 2-head multi-splits under 20KBTU. I like Daikin Quaternity series for the ability to dehumidify in both heating & cooling modes, with programmable dehumidistat settings but they're not cheap. The 3/4 ton model is the RXG09HVJU using the FTXG09HVJU indoor cassette. It's rated for 12KBTU/hr max at 47F outdoor temp, but still puts out well over 9000BTU @ 17F under any outdoor humidity conditions (which affects defrost duty-cycle). It's cooling efficiency is 26 SEER, heating efficiency HSPF 11 (COP= 4.5@ 47F), which is way up there compared to bottom-end units.
Others in that efficiency class (albeit with simpler-dumber dehumidify controls) are:
Daikin RXS09LVJU/FTXG09HVJU rated 24 SEER/ 12.5 HSPF
Fujitsu Halcyon 9RLFW/AOU9RLS2 , rated 26 SEER/12 HSPF
Mitsubishi H2i series MSZFE-09NA / MUZFE-09NA rated 26 SEER/10 HSPF
There are some good LGs and Sanyos and others of similar ratings, but the above three probably control a combined 90% of the US market (Mitsubishi is the biggest), so from a support point of view they're probably the best bets. IIRC under the hood many of the others use Fujitsu compressors and variable volume valving controls, etc, but with their own power, coils and control algorithms. Dehumidification mode is cooling only, and not under dehumidistat control the way the Daikin Quaternities are, but they're still pretty good.