The palace, which houses several important museums, was built in the second half of the 15th century probably on a project of Filippo Brunelleschi for Luca Pitti, but was unfinished at his death in 1472. The original building, formed by two floors and the ground floors, with only five windows on each tloor, was purchased in 1550 by Eleonora da Toledo, the wife of the Grand Duke Cosimo I de'Medici, thus becoming the official residence of the family. For this reason it was widened and changed, in 1560 by Bartolomeo Ammannati and at the beginning of the 17th century by Giulio and Alfonso Parigi.
The lather two architects gave the facade its present day aspect, with the only exception of the two lateral projecting pavilions that were built in the age of the Lorraine family and completed during the first half of the 19th century by Paoletti and Poccianti, who also built the Palazzina della Meridiana, added on to the rear section of the palace Downlooking the garden.

Most of the internal decoration was also executed during the 17th century by Giovanni da San Giovanni, Pietro da Cortona, il Volterrano, Antonio Domenico Gabbiani and Sebastiano Ricci.

As regards the domestic life inside the palace, we know that it was the home of several components of the family who were distributed in different private apartments. The rooms on the left wing belonged to the Grand Duke, while those on the right side were used by the heir. The lateral wings housed the apartments of their wives. The rooms on the second floor contained the large library, while the side rooms were used for the children. The left side on the ground floor housed the apartment that the Grand Duke used in summer.