In the Middle Ages, the site belonged to the lords of Cabaret, mentioned for the first time in 1067. Their wealth comes from the exploitation of iron mines. Only three castles were probably built in the eleventh century and their location evolved in time following the successive destructions and reconstructions. At that time, the lords of Cabaret were at least 22.

The castles have experienced the events of the crusade against the Albigensians. Indeed, the lords of Cabaret are very connected to the adepts of Catharism. The villages around the castles have welcomed many Cathars. The fortress belonged at the time to the faithful Pierre Roger de Cabaret of Raymond-Roger Trencavel who fought beside him during the defense of Carcassonne. The site was attacked by Simon de Montfort in 1209. The site victoriously resists attacks. But the crusader Bouchard de Marly, then lord of the castle of Saissac, was taken prisoner by Pierre-Roger. His release was negotiated against the surrender of Cabaret in 1211.

In 1223, the lords of Cabaret resumed their lands and Cabaret became the seat of the Cathar bishopric of Carcassès. Lord Pierre-Roger resisted for many years the attacks of Simon de Montfort. But in 1227, the castles were again besieged by Humbert de Beaujeu. In 1229, Cabaret capitulates.

Villages and castles are looted and rebuilt to become royal fortresses. The Regine Tower is built by order of the king to assert its supremacy. They become the administrative and military center of six communities forming the châtellenie of Cabardès. In the sixteenth century, the castles were occupied by the Protestants. They were dislodged by Marshal de Joyeuse in 1591.

The four castles are classified as historical monuments in 1905