William Henry Joseph Bonaparte Bertholoff Smith was born in Goshen, New York. His mother and grandmother chose the names to reflect the different parts of his heritage: Joseph after Saint Joseph (Bible), Bonaparte (French), Bertholoff (biological father's last name), Smith (added when he was three, his stepfather's name), and William and Henry which were added for "spiritual balance". In his memoir he reports that his father, Frank Bertholoff, was Jewish. Willie was at least somewhat conversant in Yiddish, as he demonstrated in a television interview late in his life. Willie's mother, Ida Oliver, had "Spanish, Negro, and Mohawk Indian blood". Her mother, Ann Oliver, was a banjo player and had been in Primrose and Westminstrel shows (Smith also had two cousins who were dancers in the shows, Etta and John Bloom). According to Ida, "Frank Bertholoff was a light-skinned playboy who loved his liquor, girls, and gambling." His mother threw Frank out of the house when "The Lion" was two years old. When his father died in 1901, his mother married John Smith, a master mechanic from Paterson, NJ. The surname Smith was added to that of "The Lion" at age 3. He grew up living at 76 Academy Street in Newark.
John Smith worked for C.M. Bailey, Pork Packers, and he would leave the house around midnight to pick up the freshly killed pigs and bring them to the packing house. He was supposed to be home by 4 A.M., but would usually go to bars. Eventually, Willie's mother wanted him to accompany his stepfather to work to hopefully ensure that John Smith would come straight home and not go drinking. Willie said he actually enjoyed his job, but most of the time he would have to drive the horses home. He also could only work on Fridays and Saturdays, as his mother did not want him to miss school. He wrote about the experience of being at the slaughterhouse with his stepfather:
I couldn't stand to see what I saw at the slaughterhouse. I would watch wide-eyed as the squealing pigs slid down the iron rails to the cutter where they were slashed through the middle, with the two halves falling into a tank of hot water. The kill sometimes went to as many as four hundred pigs a night. It was a sickening sight to watch. But the cries from the pigs brought forth an emotional excitement. It was another weird but musical sound that I can still hear in my head. The squeaks, the squeals, the dipping them in hot water, they put them on a hook, take off the head, the legs, going down an aisle—I hear it on an oboe. That's what you hear in a symphony: destruction, war, peace, beauty, all mixed.
In 1907, the family moved to 90 Bloome Street in Newark, but moved again around 1912. His stepfather got a new job at Crucible Steel Company, across the Passaic River in Harrison, New Jersey. The job paid more, and Willie would have to get him before his bosses got him drunk on his own money.
He attended the Baxter School, rumored to be a school for bad children. The school was notorious for brawls between Irish, Italian, and African-American children. Willie was in Mrs. Black's fruit store and was caught with his hand in her register. According to Smith's memoirs, he had wanted to borrow a dime to see S.H. Dudley's traveling road show at Blaney's Theater. The thing that shocked Willie the most was the fact that she turned him over to the police. Mrs. Black's son-in-law was the number three tough guy in Newark, and their whole family hated policemen and wouldn't allow them into their store. Willie later wrote, "But they sure didn't mind turning over a 10 year old boy to the police." He went to children's court and was sentenced to a ten dollar fine and probation.
After that incident, he was transferred to Morton School, and began sixth grade at his new school (which had a lot less brawling). He would go onto attend Barringer High School (then known as Newark High School). In an effort to get the attention of the ladies, he attempted sports including swimming, skating, track, basketball, sledding, cycling, and boxing. He learned to swim in theMorris Canal.
Prizefighting was the sport he was most interested in. Willie says that "maybe that because I've known most of the great fighters from way back. They liked to visit the night clubs...". He got to kid around with Jack Johnson, Jack Dempsey, Battling Siki, Kid Chocolate, Sam Langford, Joe Gans, Bob Fitzsimmons, Harry Greb, Joe Louis, and Gene Tunney. Fitzsimmons owned a saloon on Market Street in Newark, and that is where While learned about Stanley Ketchel, Kid McCoy, Benny Leonard, Jimmy Britt, and Charlie Warner.
Willie also belonged to a gang, and the gang had a club called The Ramblers (two famous members were Abner Zwillman and Niggy Rutman). Willie was one of two colored men in the gang, the other being Louis Moss, who Willie referred to as a "sweet talker, who could take his foes apart". Moss later became known as "Big Sue" and owned a saloon in Tenderloin, Manhattan. Moss was his own bouncer at his club (according to Willie, Moss was 6'4" and about 240 pounds). Willie says he used to help him out by playing piano in his back room.