Although the Motte & Bailey type of castle was still used in dire emergencies, most castles being erected by the late 12th century were built of stone. A licence had to be obtained from the king before work could start on the project. The mid-Norman castles still used the Motte & Bailey plan, having a large square keep and an enclosed bailey. The keep was the strong-hold for the family who lived there and for the soldiers who had to defend the castle. Surrounding the keep was the bailey, which was protected by a strong stone high wall instead of the earlier timber fence. It also had a walk-way, with access by steps, going the whole distance of the wall, to allow the defenders to fight off attackers trying the breach the walls or gates.

The entrance would have a wooden stair or bridge which could be removed in time of danger. It would also have a portcullis, a great iron bound grill which was lowered blocking the entrance. This allowed the defending archers to fire through it at the attackers. When things got serious, the solid oak gates would be closed behind it to make the entrance doubly safe.

The castle was not only a strong house for the family, but it was also used to accomodate a garrison or act as a prison. The keep might be 27.7 m high and and 30.8 m square. The walls were up to 4.6 m thick and faced with great blocks of dressed stone. The lower part of the building was splayed outwards for stability as well as for defence. Some keeps, such as Orford were octagonal or even circular.

The projecting battlements had 'machicolations' which were holes in the floor from which to drop stones or pour boiling pitch. The windows were narrow to stop arrows being aimed at them, and the most important thing in the castle was the well, in case of a long siege.