Is this a fiberglass or composite panel framed shower? It sounds like the whole thing was not installed well. There should not be big gaps between the seams. This can be caused by walls that aren't plumb and square, or sloppy installation, or both. You might want to consider taking the panels off, shim the walls so they fit right, then reinstall, or tear it out and put in something else. Maybe it's time for a nice custom tiled shower. Check out www.johnbridge.com for some ideas and help.
You may never get this satisfactorily sealed. But, if the hole is deep, buy some foam backer rods to stuff in there, then caulk. The rods come in various diameters. The caulk works better filling in if it isn't really deep.