Richardson was born in Edinburgh and worked with the BBC as a pianist before moving to London to study piano and composition with Harold Craxton at the Royal Academy of Music from 1929 to 1930. He went on a concert tour of Australia and New Zealand in 1931. From 1936 to 1939, he was the violinist Carl Flesch's accompanist. In 1961, Richardson married Janet Craxton, a famed oboist, and his instructor Harold Craxton. In 1960, he was elected Professor of Piano at the Royal Academy of Music, where he remained until he died in 1978. Richardson composed a lot of piano music and a little chamber music, including some pieces for the oboe that he wrote for his wife. With the viola player Watson Forbes, he captured his Sussex Lullaby.