Brown began his career with Charlie Echols and Paul Howard. In 1932 he joined Duke Ellington. His great technical command of the instrument, with its "creamy tone, neurotic vibrato and range" was featured with Ellington's band every year in compositions such as "Blue Cellophane" and "Golden Cress."
Brown left Ellington's band in 1951 to join a band led by former Ellington sideman Johnny Hodges, where he stayed until 1955. After leaving Hodges, Brown took a position for five years with CBS as a session player. In 1960, Brown rejoined Ellington and stayed with him until 1970. After leaving Ellington's band the second time at the age of 63, Brown quit performing completely.
Brown fulfilled many roles in the Ellington Orchestra—as a balladeer, technical soloist, and section leader. His highly melodic ballad playing as well as his fast technical style inspired trombonists from Tommy Dorsey to Bill Harris.
Brown was married to actress Fredi Washington. He died in Los Angeles, California.