Born in Clinton, Massachusetts, to a cotton manufacturer, Brown moved to the South when he was 11. He attended Knoxville High School and theUniversity of Tennessee, both in Knoxville, Tennessee, graduating from the university at the age of 19 with two degrees in engineering. An early fascination in automobiles led Brown to a job with the Stevens-Duryea Company, then to his own Brown Motor Car Company in Alabama. He later abandoned the car dealership after developing an interest in motion pictures around 1913. He was hired by the Peerless Studio at Fort Lee, New Jersey, and became an assistant to the French-born director Maurice Tourneur.