John Mercer Langston was born on the plantation of his father, Captain Ralph Quarles in Louisa County, Virginia. His mother, Lucy Langston, was a former slave emancipated by Quarles.

At the age of fourteen, Langston enrolled in Oberlin College, where he earned a Bachelor's degree and a Master's degree in theology.  After graduating from Oberlin,     Langston applied to several law schools but was denied entry because of his race. Langston found a position working for Judge Philemon Bliss, who trained him until Langston passed the bar exam in 1854. Langston became a successful and prominent attorney.

Langston later served on the City Council of Brownhelm, Ohio for about five years starting in 1855. When he was elected to the position of Town Clerk, he became the first African American to be elected to a public office. Called upon to recruit African American volunteers to fight in the Civil War, Langston organized the first Black regiment in the history of the United States, the Massachusetts 54th, and two other regiments made up of African American soldiers.

Langston held many offices, including president of the National Equal Rights League and member of the Board of Education in Oberlin. In 1868 and 1869, he was Education Inspector for the Freedmen's Bureau, working to provide educational opportunities for newly freed slaves. From 1869 until 1876, he was the dean of Howard University's law school. In 1872, Langston was appointed vice president of Howard University; then, served as Acting President for two years before resigning in 1875.  From 1877 to 1885, Langston was appointed to the diplomatic corps and served as U.S. Consul General to Haiti. In 1885, he became president of Virginia Normal College Institute.

In 1889, Langston was elected to the United States Congress, representing the State of Virginia. There was a legal dispute concerning rigging of the polls on Election Day. After an eighteen-month battle, Langston took his congressional position.  He spent his retirement at his Washington, DC home, Hillside.  

In the District, a former DC public school named John Mercer Langston School was built in 1902 on P Street between North Capitol and First Streets NW.  The town of Langston, Oklahoma, and Langston University, in that town, were also named for him.