Unbalanced &/or leaking duct systems in less than air-tight houses can drive fairly high rates of infiltration. Most houses that haven't undergone some level methodical air-sealing measures will leak well over 5 air exchanges per hour (ACH) with an unbalanced duct system, and 2-3 when it isn't running. If air is coming down the flue it doen't necessarily mean that the whole house is under negative pressure (though it might be), only that the great outdoors was the lowest impedance return path for pressure differnences between rooms. It also means that the flue damper isn't a tightly sealed type.
If some rooms have only supply ducts & no returns, without door grilles or jump ducts for equalizing the pressure, that would cause these types of pressure differences. Any rooms with supply-only need to be retrofitted with jump duct. sometimes a grille in the bottom of one side of a partition wall, and the top side of the other can perform this function, or ceiling (or floor) mounted grilles with flex ducts connecting the two rooms or room/hall will work. In some rooms where privacy is less of a concern a door-mounted grill would do it, but getting all rooms as close to the same pressure as possible is key.
It's probably worth contacting two different types of contractors- an home air-sealing contractor (usually a service offered by an insulation installer- often foam-insulation installers), and an outfit that does duct sealing. If the ducts are all inside a reasonably maintained pressure boundary of the building the amount of infiltration even from leaky or unbalanced ducts can be made fairly small. But sealing the ducts & air handler puts the conditioned air where it was designed for rather than random places. In many homes in FL the air handlers & ducts are in a ventilated attic outside the pressure boundary of the house, where duct leaks are all but guaranteed to use the great outdoors as the return. Sealing the attic itself is often easier & preferable to attemping to fully air-seal the attic floor boundary, including all duct, lighting, & plumbing penetrations. (Vented attics in FL are mostly a mis-application of a solution to problems found only in cold climates, and end up introducting more moisture to the building than it ever purges, with only the slightest effect on shingle temps or cooling loads.)