The toilet bolts should not be overtightened, but they should be tight. If you hear the sound of metal on porcelain, stop tightening. But if you still feel the bolts tightening against the wax ring, keep going. If the bolts come out of the flange when you are tightening them, then you either messed up when you put them in the flange or your flange needs to be repaired. The trick is to use a very small wrench when you're tightening the bolts, so you can get a better feel for how tight the bolts are getting.
If you are satisfied that the bolts did not pull thru the flange and they are reasonably tight, you can keep the toilet from rocking by caulking the base, if it's a minor rocking, or shimming and then caulking the base, if the rocking is more severe. But make sure the toilet bolts are tightened down first.