Robert Matthews was born in 1788 to a Scottish immigrant family in Cambridge, New York. They lived in an isolated community known as Coila and practiced a rigid brand of Scots Calvinism in a traditional Anti-Burgher Secession Church. They were strict fundamentalists, and they preferred simple, plain worship as opposed to the more elaborate Catholic rituals and prayer ceremonies. The town's spiritual leader was a tireless evangelical known as Reverend Beveridge, who took a special interest in the young Robert Matthews. Patriarchy was at the core of church and community life, and men controlled their households and the church as well as all community activities.

Matthews’s parents died in 1795, when he was seven years old, and he and his seven siblings were taken in by family and neighbors. A farmer in the church community agreed to board Matthews in exchange for labor. Matthews was frequently ill and suffered from chronic nervousness attributed to the trauma of losing his parents at a young age. Though Matthews was unable to work, he stayed with the farmer until 1806; at this time, at the age of seventeen, Matthews moved in with a local carpenter and learned his trade. A couple of years later, Matthews moved to Manhattan and found work as an apprentice-journeyman carpenter.