Founded in 1856 by Henry Johnson as a railroad station called "Johnson's Depot," Johnson City became a major rail hub for the southeast, as three railway lines crossed in the downtown area. In the late 19th and early 20th centuries, Johnson City served as headquarters for the narrow gauge East Tennessee and Western North Carolina Railroad (the ET&WNC, nicknamed "Tweetsie") and the standard gauge Clinchfield Railroad. Both rail systems featured excursion trips through scenic portions of the Blue Ridge Mountains and were engineering marvels of railway construction. The Southern Railway (now Norfolk Southern) also passes through the city.
During the American Civil War, before it was formally incorporated in 1869, the name of the town was briefly changed to Haynesville in honor of Confederate Senator Landon Carter Haynes. Henry Johnson's name was quickly restored following the war, with Johnson elected as the city's first Mayor on January 3, 1870. The town grew rapidly from 1870 until 1890 as railroad and mining interests flourished. However, the national depression of 1893, which caused many railway failures and a resulting financial panic, halted Johnson City's boom town momentum in its tracks.
In 1901, the Mountain Branch of the National Home for Disabled Volunteer Soldiers (now the U.S. Veterans Affairs Medical Center and National Cemetery), Mountain Home, Tennesse was created by an Act of the US Congress introduced by Walter P. Brownlow. Construction on this 450-acre (1.8 km2) campus, designed to serve disabled Civil War veterans, was completed in 1903 at a cost of $3 million. Prior to completion of the facility, the assessed value of the entire town was listed at $750,000. The East Tennessee State Normal School was authorized in 1911 and the new college campus located directly across from the National Soldiers Home. Johnson City began rapidly growing and became the fifth-largest city in Tennessee by 1930.
Together with neighboring Bristol, Tennessee, Johnson City was noted as a hotbed for old-time music; it hosted noteworthy Columbia Records recording sessions in 1928 known as the Johnson City Sessions. Native son "Fiddlin' Charlie" Bowman became a national recording star via these sessions. The Fountain Square area in downtown featured a host of local and traveling street entertainers including Blind Lemon Jefferson.
During the 1920s and the Prohibition era, Johnson City's ties to the bootlegging activity of the Appalachian Mountains earned the city the nickname of "Little Chicago". Stories persist that the town was one of several distribution centers for Chicago gang boss Al Capone during Prohibition. Capone had a well-organized distribution network within the southern United States for alcohol smuggling; it shipped his products from the mountain distillers to northern cities. Capone was, according to local lore, a part-time resident of Montrose Court, a luxury apartment complex now listed on the National Register of Historic Places.
The city is featured in a song and video by Travis Tritt called "Modern Day Bonnie and Clyde," although the line "rollin' north on 95" is fictionalized, as Interstate 81 and Interstate 26 intersect near Johnson City. The city is mentioned in a song by Old Crow Medicine Show called "Wagon Wheel", in the lyric "Walkin' due south out of Roanoke, I caught a trucker out of Philly had a nice long toke. But he’s a heading west from the Cumberland Gap, to Johnson City, Tennessee."] The song gets the geography wrong, as Johnson City is southeast of the Cumberland Gap.
For many years, the city had a municipal 'privilege tax' on carnival shows, in an attempt to dissuade traveling circuses and other transient entertainment businesses from doing business in town. The use of drums by merchants to draw attention to their goods is prohibited. Title Six, Section 106 of the city's municipal code, the so-called Barney Fife ordinance, empowers the city's police force to draft into involuntary service as many of the town's citizens as necessary to aid police in making arrests and in preventing or quelling any riot, unlawful assembly or breach of peace. Since 1985, Johnson City has prohibited the sale of air guns or air-propelled guns, though firearms sales are allowed.