Just a suggestion:
Instead of user manually assign passthrough colors, use the first pixel in a video frame, or something in the metadata to set the transparent color.
Fisheye alpha is quite a genius solution, but equi-retangular format is much more popular and easier to implement. In video editing is quite easy to set all transparent pixels to one specified color only, but it's quite hard to convert a equi-rectangular video to fisheye format with good results.
