Ground and neutral at subpanels must be separate. The only place they are combined is at the main panel. If you don't have a proper neutral, then 120vac circuits are trying to return (complete) their circuit through ground, which is never allowed. If you don't have a proper neutral, you won't have a proper 120vac circuit, and anything hooked up ther eis unlikely to work well, if at all.
You should have ground rods at the outbuilding, properly installed. If there isn't one at the pole by the meter, you probably need them there, too (I'm not sure). It's my understanding that you're required to have two rods, interconnected everywhere there's a panel at a building.