The hill has been inhabited since the 5th millennium BC. The inhabitants of ancient Petovione built a fortress and shrines on the hill, and in the early Middle Ages there was a Slavic burial ground.
The medieval fortress dates back to the 11th century, when the castle and the town were owned by the archbishopric of Salzburg. The archbishops gave the castle to the Lords of Ptuj, who founded Dominican and Minorite monasteries and built the Ptujska Gora sanctuary during their three hundred year stay in Ptuj. The tombstone of the last lord of Ptuj, Frederick IX, is built on the ground floor of the castle.
Between 1656 and 1802, the Counts of Leslie lived in the castle.
The last owners, the Counts of Herberstein, owned the castle between 1873 and 1945, when it was nationalised and annexed to the Ptuj Museum with its rich furnishings.
Today, Ptuj Castle houses the most visited and representative collections of the Ptuj-Ormož Regional Museum: weapons, musical instruments, traditional carnival masks, a collection of feudal residential culture, paintings on glass and the castle gallery.