First point, the ground rods are NOT "grounding" anything. This is not where your ground (as you know it) comes from.
Second, you are fine with the 3-wire feeder between the outside panel and the house. The house panel is treated as a main and the neutral is bonded to the ground. THIS is where your "ground" comes from.