The Tucker's home was located in Clarksville, an early Texas Freedom Colony that was founded by former slave Charles Clark. Its architecture is typical of the simple homes that residents of the Clarksville community built during its early years.

The Tuckers later sold their home to Edwin and Mary Smith, who raised their family there. Mary was the sister of Charles Clark and one of the founders of the Sweet Home Missionary Baptist Church, which became the focal point of the social and religious lives of Clarksville residents.

Like many early Clarksville residents, the Smiths took in boarders to help make ends meet. One of them was Hezikiah Haskell, a former Union soldier and a Buffalo Soldier. He and the Smith's daughter Catherine fell in love and married after which the Smiths deeded their home to the couple and built a new home for themselves. The Haskell's son Hezikiah Junior died in the home in the late 1970s after which the House became the site of a senior lunch program and later a museum about the history of Clarksville as well as a community and event space.