You should definitely consider just using a <video> element, especially if mobile devices are a target. Nobody at Apple or Google sweat themselves over fixing the iframe experience on mobile, they’d rather you just not use them, even if you do. It’s not really about what is or is not recommended or documented.
To reduce download time, if they’re small videos, you can use a CDN, which in many people’s experience is sufficiently faster than S3 alone. S3 isn’t really about serving things quickly. A natural choice is CloudFront.
As you can see, people report issues in “random beta version of Chrome for Android.” Supporting Android browsers horribleness has persisted for the entire decade the platform has existed, so you may consider paying for a licensed multiplatform in-browser video player.