4" is overkill for your situation. Back in the day 4" was the norm for homes all the way up to the roof. In KY you must increase to 4" before passing through the building wall.
The only time you need 4" to a toilet in my area is when there are 4 toilets in the home, which is dictated by the DFU load on 3". (36 DFU's)