Yarmy was born in New York in 1933, son of William Yarmy and his wife, Consuelo (née Deiter), who were Jewish and Catholic, respectively. Yarmy was the younger brother of actor and Get Smart star, Don Adams. He also had an elder sister, Gloria Ella Yarmy (later Gloria Burton), a writer who wrote an episode of Get Smart. Dick had served as a corporal, like his brother had in World War II, during the Korean War.