The first certain news about a fortress built in Castrovillari date back to the last period of the Angevins’ domination and are found in two mandates of 1432 of King Ludwig III of Anjou for the construction of the tower and fortress of Castrovillari. This tower could be identified with one of the four of the castle of dodecaedra shape. The same was conquered in 1440 by Alfonso of Aragon who, after expelling the pro-Angevin faction, ordered to build a fort near the Porta della Catena and the surrounding walls. The construction works were resumed in 1461, after the end of the anti-Aragonese revolt led by Antonio Centelles, who found the support of Francesco Musitano in Castrovillari, but were systematically started only after the conspiracy of the barons in 1485, when the Aragonese provided to build a safer castle than the small Angevin manor, which aroused fear in the population always hostile towards them. King Ferdinand I of Aragon commissioned the construction of the castle Loisetto de Summa, captain of the land of Castrovillari, who was to use the proceeds from the auction of the property seized from the rebels. The work, which began in 1487, was completed in 1490, as stated in the inscription on the tombstone on the portal, surmounted by the coat of arms of the Aragonese royal arms, flanked by two cherubs, the same as the castles of Belvedere Marittimo, Corigliano Calabro and Pizzo Calabro. The castle was designed according to the criteria of the famous military architect Francesco di Giorgio Martini, working at the court of Aragon. The imposing building, with a drawbridge surmounted by two slits, was surrounded by a deep moat, now completely underground. Its body was rectangular with four cylindrical towers. The largest tower, the so-called "mastio", decorated with hanging arches called beccatelli, is sadly known as the "infamous tower" because of the punishments inflicted on prisoners who were locked up in it. From 1521, after Emperor Charles V enfeoffed the city of Castrovillari to the noble Spinelli family of Cariati, the Aragonese castle became a prison. Even more atrocious was the 19th century, when, during the fight against brigandage, the "infamous tower" became the scene of horrendous crimes, also because of a ferocious and cruel jailer, Francesco Minervini da Cassano. It was also during this period that the ancient factories were erected. Since 1995 it has been partially restored and opened to the public.