The Norman barons were hated and could not risk living undefended in the countryside where their estates lay, so they built castles and fortified homes. Originally these were made of timber, often as a motte and bailey, but were later rebuilt in stone. Around twenty castles were built in Suffolk, with the principle ones at Walton, Clare, Felixstowe, Haughley, Framlingham, Bungay, Mettingham and Orford. The best preserved is the Keep at Orford Castle, which was built between 1165-1173 at a cost of £1,413. Bungay is now almost gone, it was built by Hugh Bigod in 1165, but destroyed by Henry ll 's men nine years later. The castle was rebuilt by Roger Bigod in 1294 with a keep wall 5.5m thick and about 27.7m high. An old Suffolk nursery rhyme links Bungay Castle with Hugh Bigod :-

When the Bailey had ridden to Bramfield Oak,
Sir Hugh was at Ilketshall Bower,
When the Bailey had ridden to Halesworth Cross,
He was singing in Bungay Tower.
Now that I am in my castle of Bungay
Upon the river of Waveney,
I will ne care for the King of Cockney.

The Market Cross stood in the Market Place in Halesworth and was very similar to the Butter Cross at Bungay. It became very delapidated over the years and was taken down about 1812. The King of Cockney would refer to the Kinq in London.

Richard de Clare obtained a licence to build a castle at Southwold, but died in 1262 and his son did not continue with the work. It was possibly at Skilman's Hill.

In most cases, the early castles built by the Normans were not expensive stone structures. They were often erected in a hurry to meet an immediate foe, so they put up cheaper structures known as the motte and bailey. After choosing the site they would round up the English peasants from the surrounding villagers to work in gangs on digging out the moats and building up the mounds. They were made with a high earth mound which was the 'motte' which was surrounded by ditch with a palisade or fence of stakes around the inner side of the ditch which was the 'bailey'. The lord lived in a wooden tower on top of the motte, and the bailey was used for the soldiers' living quarters and for weapons and equipment.