Depending on how they setup the floats there can be a few issues. It Might be a stuck relay, or a faulty off float.
What I mean by faulty off float is, if it is wired like most three float systems. The first float that floats up is the off float, the second float that floats up is the on float and the third float that would float up if the first pump did not turn on would sound an alarm and turn on the other pump.
Now let me explain a little about your system, or at least the way your system should be wired. When the first float floats up nothing happens yet only half the circuit has been completed. When the second float is up the pump will turn on and pump the water down. Now as the water goes down the second float will dangle down but the system will not turn off till the first float dangles at the off level. The next time the system fires up the other pump will run, which is known as an alternator system. The third float is in case one of the other two floats did not give the on signal or if the pump that was to be activated at that time did not run it will sound an alarm and apply power to both pumps.
What sounds like is happening in your case is the off (first or longest float) is not disconnecting from the circuit or the relay that runs the pump is sticky.
I hope this clears things up a little. Do you know the make of you pumps and perhaps the model #'s of them?