It depends upon your local jurisdiction. In my area many permits can be handled over the Internet with payment of the fees via credit card.
Your area may require you to visit city hall, some county office or perhaps even a state office.
Try calling the information number for your local government and asking about building, plumbing, mechanical and electrical permits. You may be able to get all permits from one source or you may need to visit several different offices.
Some jurisdictions may allow a plumber to disconnect an old electric water heater and reconnect the new one and some may require an electrician do the electrical work. If you are replacing a gas water heater with an electric one then an electrician would be required unless the plumber also was an electrician.
Generally (not always) a homeowner may "pull" permits IF he/she is doing the work. If the work is being done by a contractor then the contractor needs to pull the permit.