By the 11th century, the settlement was known as Halesuworde and had moved to the top of the ridge, east of the Parish Church. It is also possible that Halesworth became a strategic crossing place over the river and its northern tributary, where the people crossing to the hinterland could be controlled. Perhaps Halesworth was also a collecting place for the produce from the surrounding countryside, with goods moving up and down the river between Halesworth, the royal estates and market of Blythburgh and the coastal port of Dunwich. It may also have been a dependent settlement of the Royal Manor of Blythburgh.

At the time of the Norman Conquest, Halesworth consisted of 3 manors, which include a rural estate held by Aelfric and 2 smaller manors held by Ulf the Priest and by Gunner. These were under the protection of Ralph the Constable and Edric of Laxfield, two of the more important landowners in Suffolk. Ralph the Constable also held lands in Wissett and Spexhall.

According to the Domesday Book, the recorded population of Halesworth in the time of Edward the Confessor, just before the Conquest consisted of six freemen, four villagers, eleven small-holders and two slaves. If each is considered to be the head or representative of a family, we find that using an average family group of five, which was usual at that time, an estimate of Halesworth population was about 115 persons in all.