During the 13th and 14th centuries, the Manor of Halesworth had been held by the Argentein Family, but in the 16th and 17th centuries, this was the privilege of the Allington Family. When John Argentein died in 1423 without a male heir, the Lordship passed to his sister Joan, who had married Robert Allington (firstly) and then Grey Corbett. She died in 1428 and her 18 year old sister Elizabeth was her next heir, who was married to William Allington, her brother-in-law. Elizabeth died in 1460 and the Manor of Halesworth passed to her son John Allington who was succeeded in 1480 by his son William Allington, then by Sir Giles Allington in 1485. However Sir Giles outlived both his son Robert, and his grandson Giles, and so the Manor passed to his great grandson Giles Allington in 1586.

Giles was knighted by James I at Charterhouse, London, in 1603 and married Dorothy, the daughter of Thomas Cecil, the Earl of Exeter, and when Sir Giles died in 1638, the Manor of Halesworth passed to his 2nd, but eldest surviving son, the next Sir Giles Allington.

Sir Giles married unwisely and it seems that his wife was a half-sister of one of his nieces, the daughter of his sister, who was a Mrs. Dalton. For this breach of the forbidden decree of 'consanguinity' (blood-relationship which sets out who can or cannot marry whom) he was in real trouble. The Star Chamber fined him £12,000 and he also had to provide a further bond of £20,000 which was held to ensure he never came into the private company of his wife again. He was condemned to do penance at St Paul's Cross, London, and in St. Mary's Church, Cambridge in 1631, and his wife had to do the same. She died of smallpox in 1644. The bride's father and brother were fined £2,000 for having 'procured' the licence.

Because of this scandalous event, the estates of Sir Giles, on his death passed, not to his children, for they had been declared illegitimate, but to his only surviving brother, William Allington. William was made a peer in 1642 as Baron Allington of Killard, and married Elizabeth, daughter of Sir Lionel Tollemache, 2nd Baronet of Helmingham. On his death the Manor of Halesworth was willed to his widow, who died in 1671 when it passed to her son William Allington, 3rd Baron Allington who was Constable of the Tower (0f London) in 1679, and made Baron Allington of Wymondley in 1682. When he died in 1684, his eldest son, who became the 4th baron, died at the age of 10yrs, predeceased by his two younger brothers.

At this stage the Manor and title passed to his surviving brother Hildebrand, who became the 5th Baron Allington. He sold the Manor of Halesworth and the advowson (the right to recommend a post or position) of the Church to William Betts, although the date of the sale is uncertain, for the Davy's Suffolk manuscripts states it took place in 1696, while the first Court held by William Betts as Lord of the Manor, did not take place until 1706. The Allington Family, for some of the later period were absentee landlords, living at Horseheath in Cambridgeshire, and Richard Soane, an associate of WaIter Norton of Gothic House, held lands in Halesworth in 1577 which included the 'site of the manor'. By 1602 the publication 'Chorography of Suffolk' rather sadly commented that 'in this town (Halesworth) was a parke and in it is a goodly house, the one ruinated and other disparked'.