Henry "Buster" Smith was born and raised in Alsdorf, Texas, a small township near Telico in the outskirts of Dallas, where he attended school as a child. Smith earned the name "Buster" from his parents as a baby, as he was born as an overweight child. Buster was the third of five boys and had no sisters, though both of his older brothers died in childhood of measles.
Smith's early musical influences were his mother, and his father, who played guitar. At four years old, Buster was playing the organ with his brother, pianist Boston Smith; Buster played the keys and Boston stepped on the pedals. Soon thereafter, his grandfather gave away the family organ because he believed it would only direct Buster to a life of sin.
In 1941, Smith decided to return to Dallas and to cease touring, though he remained active in the local music scene. In the following years, he wrote for jazz and blues bands, played often, and taught many young Texan musicians, including Aaron "T-Bone" Walker and Red Garland among others. He also performed session work with artists such as Pete Johnson's Boogie-Woogie Boys, Eddie Durham, Leo "Snub" Mosley, Bon and His Buddies, and the Don Redman Orchestra.