The medieval castle was built by Hugh Fitz Osbern to counter the loss of his fort at the Warden in the nearby Presteigne. Roger Port of Kington took control of the fort in 1143, and Hugh Fitz Osbern decided that he needed another defensive position from which he could withstand the Welsh and use as his headquarters.

This new site proved perfect for defence because the South and West (from which the Welsh were most likely to attack) were defended by steep, rocky banks and the East and North both had tall banks from which all enemy forces could be spotted. The two parts of the castle were separated by a shallow ditch between the Bailey on the lower, Northern mound and the rest of the castle on the taller, Southern mound.

Eventually, by the late 12th Century, the de Say family who were the Lords of Richard's Castle, became the owners of the castle and it's estate. The right to hold a market in Stapleton was granted to by the King, to Margaret Say, the owner of the castle, in 1223. These proved important to the local farmers because it gave them a place to sell their stock, closer to their homes, rather than Presteigne or one of the Herefordshire markets.

A marriage between the de Says and the Mortimers of Essex secured the new domination of the region in 1207 by the Mortimer family who took control of Ludlow castle as well. The Mortimer family became popular with the King and made Ludlow castle wealthy and the duel-centre of the nation along with London. In 1304, the last descendant of the Mortimer family died and the castle became property of the Sir Geoffrey de Cornewall. The Cornewall family had possession of the castle for several hundred years. Owain Glyndwr led the Welsh rebellion in 1403 and the castle was garrisoned to defend against him, although no major assault took place. During the 1500's, the castle was modified and upgraded to a fortified manor house. When civil war broke out, the Cornewall family sided with the King and semi-demolished and abandoned the fortified manor to prevent the Parliamentarian force from using it.