Estimate #2 is a $600-700 unit, but has another ~$200-$500 or more of parts-cost for stainless liner. The liner is necessary to avoid "orphaned water heater" degradation of the chimney, and under no circumstances would you be able to use the chimney without a stainless liner- the water heater doesn't put enough heat into the wider flue to avoid corrosive condensation, and the chimney WOULD begin to fall apart within a decade or so. It's not a cheap tank, not a cheap liner, call it about a grand in component cost, give or take. But going with an atmospheric drafted appliance of any type in new, tight construction comes with a backdrafting risk if the house is as tight as you'd WANT it to be for space-heating energy efficiency. But figure they're charging ~$1500 to install it.
Estimate #1 is for a much higher efficiency condensing water heater that costs about a grand more than the other, and delivers more first-hour gallons to boot. It's power drafted, with no risk of backdrafting and isn't sucking in outdoor air 24/365 just sitting there. There may be state/local efficiency rebate type money available to help offset the cost too. The quote seems a bit on the high side, but not necessarily an outrageous gouge for a heater that runs ~$1600-1700 f.o.b. the distributor's warehouse. PVC venting is dirt-cheap compared to stainless steel, but it too takes time to install. The total parts costs here would be running a bit shy of 2 grand when you add it all up, with a bit more than $1500 of installation cost.
In either case the contractor is risking some money up front unless you're paying for it up front. I'm not sure what plumbers charge per hour on L.I., but neither of those quotes are ridiculously "over the top", if a bit more expensive than what it would be in central MA. It's going to tie up 1-2 guys and a truck for the better part of a day, but it's probably not more than 10 man-hours of labor, maybe not even 8, depending.
By "cap off" I'd presume to mean they're sealing up the port to the chimney the previous water heater had used so that it isn't sucking air 24/365 and providing a flyway for birds, bats, squirrels etc. into your house.