The size of the neutral must be the maximum unbalanced load. In order the figure this you must figure the load for the feeders and find the maximum load between the ungrounded conductors and the difference between these and the neutral.
It is much easier to just size the neutral the same as the ungrounded conductors in your case. The cost savings between the sizes would only be a few cents. To answer your question I would need the numbers of the load and each branch circuit.