My question is how would you go about insulating this? Closed cell spray foam along the exterior wall and floor? That’s probably the easiest solution but is it safe with the dryer duct and also the air register ductwork in those two or three bays?
Closed cell foam is great at air sealing, but at more than 1" becomes a vapor barrier, which you would want to
avoid on the exterior (meaning next to outdoor air, whether up, down or sideways) side of the assembly if you can, given that you're in climate zone 5. Sealing up the seams of the band joist and bottom sheathing with closed cell foam is fine, and won't take very much (a 12 board foot FrothPak kit would be more than enough). A serious issue with DIY spray foam this time of year is staying the required substrate temperatures. It might be safer to air seal those seams with polyurethane caulk instead of foam. If the surfaces are too cold the foam won't adhere well, and could result in incomplete curing of the foam with a long term outgassing problem. Giving it a full coverage continuous 1" for air sealing would be fine as long it's warm enough to cure, at which point you don't care too much about adhesion- closed foam is self-supporting. As long as it sticks well enough to the trussed floor joists to seal it'll be fine.
Installing air dams in the joist bays crossing over the cantilever wall is important. Cut up foam board sealed in place with FrothPak or can-foam works fine. That allows you to completely fill the overhangs with cheap blown fiber insulation. Sealing the truss joists is a PITA to be sure, and it may take a bigger DIY foam kit to deal with sealing around the truss elements. Half-inch foil faced foam board or even corrugated cardboard stapled to the truss elements can serve as an air barrier between & through the truss-joists, as long as the seams and edges are adequately sealed to the truss-joists, subfloor, and top plate of the supporting wall with either housewrap tape or foam (either a 2-part kit, or 1 part can foam) or combinations thereof where the complicated geometry warrants it.
With a decently tight interior air barrier in place, inexpensive blown fiber insulation can fully fill in the spaces in the truss webbing and completely fill the cantilevered cavity. If you're not crazy about drilling holes through the subfloor layer (which would have to be repaired later) it's pretty easy to blow the insulation in mesh. Blowing mesh is usually sold in large rolls, much bigger than you would ever need, but landscaping fabric works about as well, and available in smaller rolls at box stores. Use a brayer to spread wood glue evenly on the tops of the trusses and staple the landscaping fabric on the top of the trusses rolling it smooth into the glue for flatness. Give the wood glue a day to set up to avoid getting insulation between the mesh and wood when installing the insulation, which makes it harder to get a flat floor.
Starting at one end, cut a slit in the mesh between trusses just big enough to fit the hose into, and blow until the insulation stops flowing, ensuring a reasonably dense compression fit when it's time to put the subfloor down. The center of the mesh will pillow up an inch or so, but that's fine. Even though most of the other cavities will be filled-mostly by the initial blow, the density won't be uniform, so move over to the next bay, cut a slit, insert the hose the way to the bottom sheathing, then back off about 8", and blow until it packs out, stopping the flow of insulation, repeat.
With a wimpy single stage rental blower and a 2" - 3" hose you'll usually be able to get ~2.5lbs per cubic foot density out of cellulose using this method. That is about twice the density of an open blow in an attic, and sufficient for preventing settling over time. It is also dense enough to be quite air-retardent (more air retardent than fiberglass using the same method), so there won't be much convection or infiltration from any remaining leaks in the air barriers, subfloor, or bottom sheathing.
I was thinking this: remove the beat up foil as best as possible, close the damage to the sheathing with new plywood, continue sealing all the joints in the cantilever bay sheathing and floor with Dap mouse repellant spray foam (there was a ton of droppings and acorns), spray professional closed cell spray foam on the wall and floor of the cantilever. I only have access from the top of the bay, and not to all of them either so what I can do is somewhat limited if I were to do open cell and vapor barrier. I’m really unsure of what to do around the 2x4s under the balcony door too, just shoot the closed cell around it or remove it and spray? Thank you!
Leave the 2x4 under the door- it's structural, enough to support a threshold, and preserves the door geometry better. Encapsulating it in an inch of closed cell foam is sufficient.
Open cell foam can work for both air sealing and insulation, but has to be installed in ~6" lifts with a cooling/curing period between lifts (for both short term fire hazard and quality reasons.) That's only about R20, and the IRC code minimum is R30 in zone 5, so you're looking at 2 lifts. Time is money, and this is a small job, so it would probably be ~2x as expensive as a bigger job. With the open cell foam there would still need to be an air barrier and vapor retarder between the trusses where they cross the supporting wall. Unlike cellulose, open cell foam has effectively zero hygric buffering capacity, which would mean any water vapor getting into the cantilevered bays would be stored in the band joist & bottom sheathing. A full fill of cellulose would share and redistribute the moisture burden, protecting the structural wood.
With any of these solutions, make sure the ducts and dryer vents are well sealed with duct mastic (not tape) on all the seams and joints to prevent injecting moisture into that space. If you tend to be a worrier, you may want to replace the single wall dryer vent with B-vent as fire mitigation, though I don't believe that is required anywhere by code.
