Varney was born in Canning Town, then in Essex but now part of the London Borough of Newham. His father worked in a rubber factory in Silvertown and he was one of five children who grew up in Addington Road, Canning Town. Varney was educated at the nearby Star Lane Primary School in West Ham and after leaving school at 14, he worked as a messenger boy and a page boy at the Regent Palace Hotel. He took piano lessons as a child and was good enough to find employment as a part-time piano player. His first paid engagement was at Plumstead Radical Club in Woolwich, for which he was paid eight shillings and sixpence (42½p). He also played in working men's clubs, pubs and ABC cinemas, and later sang with big bands of the time. He and his mother decided that show business was the career for him, and he gave up his day jobs.