The Carter House was built by Frances Newman Perkins, a Loyalist from Connecticut, for his son, Robert Perkins. The house was eventually passed down to Perkins’ daughter Phoebe and her husband William Pywell, a substantial landholder, innkeeper, and lumberman who immigrated to New Brunswick in 1817 from England. In 1855, Pywell sold the house to William A. Carter, a shoemaker. Carter and his wife, Mary Jones, raised a family of ten children in this home, one of whom, William S. Carter, would later become a prominent figure in his community, and the Chief Superintendent of Education for the province. It is for him that the Carter House is named.

In 1964, Peninsula Heritage Inc. purchased the Carter House with the intention of preserving and maintaining it for future generations. In the years that followed, extensive renovations were undertaken. In 1998, the Carter House Tea Room was opened for business. Since that time, it has served not only as an interesting and leisurely stop for travelers and a gathering place for locals, but also a source of summer employment, allowing local students to work close to home and gain valuable experience and knowledge about their community.