That black stuff is mildew. It's a fungus that feeds on the vegetable oils found in bar soaps, like the palm and olive oils from which the Palmolive company got it's name.
That's why you never see mildew growing on ceramic tiling above a bathtub in a house where people have baths instead of showers. It's the soap that gets deposited onto the walls in a shower that provides the food supply mildew needs to survive and multiply. When someone has a bath, neither soap nor soap scum will ever get more than an inch or two above the tub, and so you never see mildew on the grout in a house where people have baths instead of showers ('cept for an inch or two above the tub).
When I clean mildew off silicone caulk, I leave the Borax/Bleach slurry on for several days. However, I've found that most of the cleaning occurs within 24 hours. If you don't have another bathroom you can use to shower or take a bath, just cover the slurry with 2 inch wide painter's masking tape, and then cover that painter's masking tape with packaging tape to make for a moisture-proof seal.
Then shower normally and take the tape off in a few days. Then, remove the dried up borax/bleach slurry with a putty knife (being careful not to harm the underlying silicone caulk) and you should find that your silicone caulk is spotless. (That slurry won't have actually "dried"; it'll have solidified. The water in it gets bound up in the solidified stuff just like the H2O in the gypsum core of drywall.)
PS: Boron is a natural fungicide just like zinc and copper. In both zinc naphthalate and copper naphthalate wood preservatives, it's the zinc and copper that are the active ingredients. Boron is the active ingredient in borate wood preservatives like Tim-Bor and Borocol liquid wood preservatives as well as Impel and Cobra Rods which are also boron based. So, by mixing Borax into the bleach to make the slurry, you've got a natural fungicide in your slurry, thereby making it more effective at killing fungii than if you'd made the slurry using any other powder.
PS #2: I expect you could probably eliminate the possibility of mildew growing on your silicone (or anywhere) by using a skin cleanser that didn't have any vegetable oils in it, like either Cetaphil or Aquanil, both of which are available at most drug stores. Both Cetaphil and Aquanil advertise themselves as being "lipid free", and that's just means they don't contain any "fatty acids" which is what vegetable oils and bar soaps are made of. And, it's those lipids that mildew eats to survive and multiply. So, by switching from using a bar soap to a "lipid free" skin cleanser in your shower, I'll bet dollars to donuts that you won't ever have any mildew anywhere in your shower. That is, by using a lipid free skin cleanser, you deprive the mildew in your shower of food, and whatever mildew is inside your shower slowly starves to death.
So, you could clean the mildew off your silicone caulk with a borax/bleach slurry or switch to a lipid-free skin cleanser and wait for nature to take it's course. Or, do both and prove to yourself that the mildew doesn't return as long as you're not feeding the mildew in your shower with fatty acids from the bar soap you're using.
Ya gotta know this stuff to be worth your salt as a janitor.