The lands where the tower stands belonged to the Cheyne family until 1354 when it passed by marriage to the Keiths, the Earls of Marshall who probably built the castle in the late 14th or early 15th century. The tower appears to have been in the possession of the Sinclair Earls of Caithness in the early 17th century, and to have been garrisoned by Cromwellian troops in 1651. By 1726 it had come into the possession of the Dunbars of Hempriggs by whom it is still owned. The property is currently a hotel.
A legend from the 15th century relates the tale of a young woman by the name of Helen Gunn, who was abducted by John Keith for her beauty. She flung herself, or fell, from the highest tower of Ackergill to escape her abductor’s advances. Supposedly her ghost is still seen, wearing a long red gown and a tall head of black hair. This is said to have been the true beginning for all feuding between the Gunns and the Keiths. It led to the Battle of Champions in either 1478 or 1464, a judicial combat which led to a massacre of the Gunns by the Keiths at the chapel of St Tear (or Tayre) just east of the village.