Dame Margery's Manor is named after Margaret, wife of Sir John Argentein who died in 1382. It probably consisted of land (40 acres) which was 'delivered' out of the estates of an earlier Sir John Argentein (who died in 1318) to his second wife Agnes, and their son, another Sir John, who married the Margery mentioned above.

By the 16th century, Dame Margery's Manor was in the hands of Robert Norton of Gothic House, and by 1561 it passed to his son WaIter. He was succeeded by his son Henry in 1593 and in 1628 the Manor was held by Sir Edward Waldegrave of Morton, Anthony Bedingfield of Peasenhall and Robed Mason of Lincolns' Inn (London) in trust for his son WaIter Norton. This was to take this property out of his own control, and so avoid it being subject to fines for religious purposes.

The Rectory Manor was attached to the Rectory, and listed as 'a small manor belonging to the Rectory, to which several free and copyhold tenants owe suit and service'.

The principal Halesworth Manor consisted of about 200 acres, with some outside the Halesworth Parish. Dame Margery's Manor was smaller (about 40 acres) and this later merged with the Halesworth Manor. Rectory Manor was 40 acres and was always held by the Rector of Halesworth.