I have been investigating this one for a while and sadly it is complicated.
The issue is that browsers don't load video the same way and there's no standard either, at least not adopted. Support for video files seems to vary in several factors, every vendor pours their own magic. For example iOS uses heif/heic which doesn't work on web so when you pick a file it converts it on the fly to JPEG/MOV.
There's also the issue on how browsers understand HDR video, and several other nasty issues that doesn't worth mentioning.
Will keep digging.