The first record of Withdean, 'Whita's Valley', dates from the early 12th century when it was referred to as Wictene. The Manor of Withdean was the property of St. Pancras' Priory at Lewes until 1537 when it was surrendered to King Henry VIII. The following year it was granted to Thomas Cromwell, his Chief Minister. When he was executed in 1540 it passed on to Anne of Cleves as part of a grant on her divorce and since then it has been owned by many people. The hamlet by the London Road formed part of the parish of Patcham but by the 18th century enclosures and the concentration of land had resulted in its depopulation and the establishment of a single farm.
In 1794 the manorial estates of Withdean Cayliffe and Withdean Court were purchased from the Western family by William Roe who was a distinguished civil servant in the reign of George III. The whole estates of Withdean Cayliffe seem to have extended from Blatchington and Hove to Hollingbury or even Moulsecombe. Over the next 50 years or so Roe planted many of the trees in the area, including those around Varndean, which existed in 1750 as a plantation. From 1853 William Roe's heiress, Elizabeth Ogle, began to sell land in London Road to the south of Withdean, for development. This was encouraged by the removal in 1854 of the turnpike tollgate from Preston to a hundred yards north of Withdean. In the middle of the 19th century the hamlet consisted of Withdean Farm, Withdean Court and a number of farm houses and outbuildings grouped around the bottom of Peacock and Tongdean Lane. The site of Withdean Farm, demolished in 1963, is now occupied by Bourne Court.
Withdean Park, covering 38 acres of farmland, was acquired by the Corporation in July 1933, chiefly through the efforts of Sir Herbert Carden, to prevent its development for housing. The buildings at the bottom of Peacock Lane, including the Old Manor House, were demolished in 1936 and the new park was kept in an informal state until the Second World War when it was given over to food production.