flange
EJ,
What type of pipe do you recommend caulking in there? What will fit? A piece of iron pipe or PVC? I think if I reduced the opening down to an inside diameter of 4", the wax ring would give me a better seal in the long run so I don't have to repair the floor again. The plumbing guy at local Home Depot said this might be a commercial fitting that was put on my toilet.
I just talked to a commercial plumbing supply house. Apparently my situation is common plumbing technique in the Chicago area. The flange is 5" diameter with a 1 1/2" drop connecting to a 4" waste pipe. A regular wax ring should do the trick. Apparently, the original installation had a plumber's putty like consistency that reduced this 5" opening to about 4" (this material was very stiff and flaking) when I originally removed it.