Through the 13th & 14th centuries the lodge was inhabited as a Medieval Hall House according to William Morris. In 1582 alterations were carried out that characterize the main part of the house today. Some of the timbers from the Armada fleet were incorporated into the design and can be seen in the dining room panelling.
In 1880 Wycham Flower bought the property and commissioned Phillip Webb to further extend the property and design the moat and gardens. Early in the 19th century, after all the works had been completed, the Royal family, including King George V and Queen Mary, visited the property and signed the window of the dining room with a diamond ring. Signatures of King George VI can also be seen on the dressing room windows of the master suite.