Per Elst castle ruins in Duffel is one of the oldest buildings in the province of Antwerp.Locally known as Kasteel Ter Elst,it lies on an islet in a public park in the village of Duffel.The oldest reference dates from the 12th century. At that time castle was owned by the brothers Hildincshusen. From 1356 until the French Revolution in 1789 the castle was owned by the Tongerlo Abbey and was inhabited by the steward. It was also used as a residence for the nobility. In 1584 the castle burned down and was then rebuilt. In the late 16th century it was a vicarage.
According to 17th- and 18th-century drawings Elst castle was a large building surrounded by moat and gardens. In 1799 the castle was sold to Louis Hermans showed demolish three of the four main buildings of the complex and restored the wing part with the towers. During the First World War the castle was badly damaged.