The Carter House was built by Frances Newman Perkins, a Loyalist from Connecticut, for his son Robert Perkins. The house was eventually passed down to Perkins’ daughter Phoebe and her husband William Pywell, a substantial landholder, innkeeper and lumberman who immigrated to New Brunswick in 1817 from England. In 1855 Pywell sold the house to William A. Carter, a shoemaker. Carter and his wife Mary Jones raised a family of ten children in this home, one of whom, William S. Carter, would later become a prominent figure in his community, and the Chief Superintendent of Education for the province. It is for him that the Carter House is named.
In 1964 Peninsula Heritage Inc. purchased Carter House with the intention of preserving and maintaining it for future generations. In the years that followed extensive renovations were undertaken and in 1998 the Carter House Tea Room was opened for business. Since that time it has served not only as an interesting and leisurely stop for travelers and a gathering place for locals, but also a source of summer employment, allowing local students to work close to home and gain valuable experience and knowledge about their community.