Some things that would affect cost are if there is to be any electrical service, water, sewer, or if there is to be any add-on rooms, an attic, maybe a bathroom or maybe a bedroom or workshop in the building. Unless one bid was for say brick or stone facings and extra rooms, my feeling is that they hi-balled you because they don't want to do the work, but if you chose them, the high profit margin made it worthwhile. If all the bids were for the same design and features, somethings wrong with the way outside average bid unless it also included a bunch of extras. If there needs to be a lot of fill, or the soil or location needs a lot of foundation work that is not common, it's not viable. Does one include a new driveway? Something other than asphalt? A poured concrete, maybe stamped, or pavers or brick can add up quickly.
Review the details of the bids to understand what they include, or give them a set of plans on exactly what you want rather than say I want a new garage...then, you can compare apples to apples.