During the game the Town Center will be the center of your colony. It is where you train settlers and minutemen, ring the town bell (which tells settlers to garrison inside buildings), you, unless you choose another building, will recieve your shipments from your home city, and ransom your Explorer (which means to pay to get him back if he is killed). They also can support 10 population.
If you choose to start in the nomad period you will have a covered wagon. A covered wagon transforms into a Town Center. Or your Explorer can build one in the Fortress Age for 600 wood.
Town Centers have an anti-ship attack. So it is quite handy to build one by a shore. But in Age IV: Industrial Age monitors become avaliable. Monitors are huge ships, second most powerful, and are used for bombarding buildings. When your Town Center is destroyed, your colony starts to fall apart. Then settlers don't know to garrison in buildings, you can't train new settlers.