Although many ancient Chinese-lantern-like balloons were flown unmanned, since 280AD, the first manned tethered flight took place in France in June 1783 by the Montgolfier brothers. Before their attempt at humans on a balloon flight, they had previously put animals in a hot air balloon - a sheep, rooster, and duck, and the balloon successfully obtained an altitude of 1,500 feet and last around 10 minutes.
The very first untethered 'free' flight with humans happened in September 1783 with Jean-Francois Pilatre de Rozier and Marquis d'Arlandes flying a Montgolfier balloon. Although the flight was a success, Pilatre de Rozier died a couple of years after when his balloon crashed during an attempt to cross the English Channel. He and Pierre Romain, his companion in the flight, became the first known fatalities in an air crash.
Back in the 1800's the balloons were just cloth bags with a smoky fire built beneath them which was attached to a grill. Because of this many balloons caught fire when they landed. The first manned Hydrogen balloon flight occurred in December 1873 with Jacques Charles and the Robert Brothers in Paris. They carried a barometer and thermometer with them, and ascended to around 1,800 feet, making it the first measured balloon flight.
Although Poilatre de Rozier and Romain crashed the balloon when attempting the cross the English Channel, it was successfully done in 1785 by Jean-Pierre Blanchard. 1785 was a particularly bad year for ballooning, as not only was this the year of the balloon crash and death of Poiltatre de Rozier and Romain, but another balloon also crashed, this time in a town in Ireland. The crash seriously damaged the town, burning down 100 houses in a huge fire.
In 1793 the first manned hot air balloon was flown in North America, with George Washington present to see the balloon launch. Over a hundred years later in 1935 a Swiss scientist, Auguste Piccard, was the first person to achieve a manned balloon flight to the stratosphere, ascending over 52,000 feet. Hot air balloons were also used for military reasons, especially during the early 20th century wars.
In 2012 Felix Baumgartner broke records by jumping from a hot air balloon that had risen to over 125,000 feet. He made history for highest hot air balloon ascent, and he became the first person to go faster than the speed of sound when freefalling from the balloon and before opening his parachute.