Fifi D'Orsay was born Marie-Rose Angelina Yvonne Lussier in Montreal, Canada, to a father who was a postal clerk. The couple had a large family, with Fifi having 11 siblings. She was educated at the Academy of the Sacred Heart in Montreal before graduating and finding work as a secretary. As a young typist she wished to become an actress, and moved to New York City. Once there she found work with the Greenwich Village Follies, after an audition in which she sang "Yes! We Have No Bananas" in French. When asked where she was from, she told the director she was from Paris, France, and that she had worked in the Folies Bergère. The impressed director hired her, billing her as "Mademoiselle Fifi".