Frankel was born at Chelsea and Westminster Hospital in West London, to English actor Mark Frankel and actress/composer Caroline Besson. His father's family is Jewish, including Jewish ancestors from Iraq and Mumbai, India through his paternal grandmother. Frankel lost his father to a road accident when he was two, while his mother was pregnant with his younger brother Max. The two brothers were raised in London by their mother and spoke French at home. She introduced them to film by taking them to the cinema once a week.