But a prime suspect would be an issue with the wiring between the thermostat and the boiler's controls( rather than the thermostat itself). A short circuit between the wires would be sensed by the boiler as a call for heat. If it sorta worked for awhile after the T-stat was replaced, it could very well be that wiggling the wires during installation disconnected the short, but vibration has re-connected it.
Can we assume you're turning the boiler on/off with the main power to the system, not with the T-stat, not with the aquastat?
BTW: 120F is too cold a temp for an old oil boiler. At 120F the combustion exhaust will condense suphuric acid & water onto the flue & heat exchangers damaging them. 140F is the minimum most oil equipment should be run to protect the heat exchangers, and even then flue-condensation can still be an issue. As a tenant, set it for 150F and switch it on/off with the power switch while you're figuring this out to avoid potential liablilty when the flues begin to crumble or boiler begins to leak. Keeping it at 150F all summer WILL use a considerable amount of oil, and turning it on/off will have it running cold & condensing for a significant fraction of the time, but setting it at 140F & cold-starting every day means it's running MOST of the burn time in cold & condensing mode. Set to 150F it'll be condensing maybe half the the time on a cold start, and not at all during subsequent cycles.
If there's a gas service to your apartment, you might check to see if there are incentives offered by the utilities for retrofitting a gas burner. Gas in most of MA is currently much cheaper than oil per BTU delivered, and it's exhaust condensate while still sligthtly acidic, is much more benign. You can run cast iron boilers on gas down to ~130F without condensing on the heat exchangers, and the condensate is only about as acidic as red wine, not the near-battery-acid you get with oil exhaust condensate.