First, there is NO location in NJ with a
99% design temp of 0F, and it's silly to design to 0F. Coastal areas run about +15F, interior and higher elevation about +10F. Even if you sized it EXACTLY for +10F on an I=B=R heat load you would have margin, due to the internal heat sources and the other factors like curtains & shades, the non-linear improvement in performance of windows and fiber insulation with falling temps. Yes it does get to 0F or below sometimes, but the thermal mass of the house and the internal heat gains of hot-bodies & plug loads mean you don't really lose ground.
Using an outdoor design temp of +10F and an interior design temp of 70F, that's a 60F delta. Assuming your I=B=R calc at 851BTU/degree-hour is correct, that's ~ 51K of load BEFORE you start subtracting of internal heat sources. Every sleeping human is worth ~250BTU/hr, each refrigerator is worth about 200BTU/hr, then there's all the parasitic plug loads (got a Tivo?) it adds up. I'm guessing you're probably between 45-50K at +10F.
Adding 400' of addition does not necessarily increase the heat load very much, since it doesn't add exactly proportionally to the exterior surface area, and even at
IRC 2009 code min for U-factors and R-values, if it's displacing R11 batt-insulated walls with R13+5c.i. and U0.6 -U0.1 windows with U0.35 the heat load can even go DOWN(!).
Do NOT use the IBR rated output unless your boiler is out in the garage, on the other side of an insulated wall. The IBR output rating is a fudge-factor that presumes the boiler's standby loss and distribution losses go somewhere else. If the boiler room and distribution plumbing is is located in a semi-conditioned basement or anywhere else inside the thermal & pressure boundary of the house, the DOE output is the most appropriate number.
The Ulrtra-80 is still a bit overkill for your true heat loads, but since it can modulate down to 16K-in/15K-out, it'll work out OK, since it's min-fire output going to be less than half your design condition heat load. As long as your smallest heating zone can deliver the 15K at condensing temps (average water temp <<125F) it's performance will be AWESOME!
If the 400' addition is going to be it's own zone, you have to keep that in mind- it may take something 2x the amount of radiation that it takes to heat the place to be able to shed 15KBTU/hr at low water temps without short-cycling the boiler. Most fin-tube baseboard delivers ~250BTU/ft @ 125F AWT, so you'd be looking at 60' for it to balance at min-fire, but the heat load of the addition zone itself might only be 5000-6000BTU/hr @ +10F, which could be readily heated with 25' of baseboard, but that would be a too-small micro-zone likely to cause short-cycling, robbing the boiler of what would otherwise be excellent efficiency.
Single pane windows are typically U0.8 (wood framed with muntin bars), to U-1.1 (aluminum framed single-light sashes). It's well worth either replacing them with a code-min window or (for far less money) adding low-E storm windows (which will sometimes outperform a code-min replacement window.) Harvey makes the tightest storm windows in the biz, and has a hard-coat low-E option, but the better grade Larsen's sold through the big box store chains are pretty good too. It's on the order of a couple-hundred per window, but the installation labor is dramatically less than a replacement window. Low-E storms may be the most cost-effective load reduction you can make, but there are surely more.
A heat load of 50K @ +10F on a 2000 house is on the high side (25BTU per foot of conditioned space), and most can be brought under 40K cost effectively with well targeted envelope upgrades. Air-sealing is the critical first step, followed by spot insulation where there are deficits or where it's easy to add more. The single-largest air leak in most timber-framed homes that haven't already been through a round of blower-door verified air sealing is often the band joist and foundation sill, typically more leakage than all of the windows & doors combined, and being at the bottom of the house, a more important leak due to the stack-effect draw. Air sealing all attic penetrations also looms large on the list- plumbing stack & flue chases, electrical penetrations, etc all add up to significant stack effect losses. At 0.18BTU/per cubic foot degree-F every 10 cubic feet per minute count. (10cfm=600 cubic feet per hour, and at a design delta-T of 60F that's 3600 BTU/hr. If you have 20 cfm of stack-effect leak (which isn't out of the question, even common for 2 & 3 story houses) it's more than 10% of the total design condition heat load. Don't worry about making the place "too tight" for human health- it's actually pretty difficult to achieve that level of tightness in retrofit without a full-gut rehab and a concerted air sealing effort. Tight is always right- if you end up with 40%+ indoor relative humidity even during January cold snaps, then it's time to think seriously about mechanical ventilation (or reducing the size and number of tropical house plants you keep.
)