In an earlier video they made a couple years back about Disney's sodium vapor technique Paul Debevec suggested he was considering creating a dataset using a similar premise: filming enough perfectly masked references to be able to train models to achieve better keying. So it was interesting seeing Corridor tackle this by instead using synthetic data.
With regards to the sodium vapor process, an idea has been percolating in the back of my head ever since I saw that video. But I don't really have the budget to try it out.
theory: make the mask out of non-visable light
illuminate the backing screen in near Infra-Red light.
point two cameras at a splitting prism with a near IR pass filter(I have confirmed that such thing exists and is commercially available)
Leave the 90 degree(unaltered path) camera untouched, this is the visible camera.
Remove the IR filter from the 180 degree(filter path) camera, this is the mask camera.
Now you get a perfect non-color shifting mask(in theory), The splitting prism would hurt light intake. It might be worth it to try putting the cameras really close together , pointed same direction, no prism, and see if that is close enough.
Don't humans and other warm objects also radiate IR?
That is far-IR, thermal stuff, Near-IR, 700 nanometer-ish is right below red in human vision.
Camera sensors can pick up a little near-IR so they have have a filter to block it. If that filter was removed and a filter to block visable light was used in place you would have a camera that can only see non-visable light. Poorly, the camera was not engineered to operate in this bandwidth, but it might be good enough for a mask. A mask that does not interfere with any visible colors.
I'll do you one better, which requires no special cameras (most have IR filters) nor double cameras or prisms.
Shoot the scene in 48 or 96 fps. Sync the set lighting to odd frames. Every odd frame, the set lights are on. Every even frame, set lights are off.
For the backing screen, do the reverse. Even frames, the backing screen is on. Odd frames, backing screen is off.
There you go. Mask / normal shot / Mask normal shot / Mask ... you get the idea.
Of course, motion will cause normal image and mask go out of sync, but I bet that can be remedied by interpolating a new frame between every mask frame. Plus, when you mix it down to 24fps you can introduce as much motion blur and shutter angle "emulation" as you want.
This is called “ghost frame” and already exists in Red cameras and virtual production wall tools like Disguise.
From ~04:10 till 05:00 they talk about sodium-vapor lights and how Disney has the exclusive rights to use it. From what I read the knowledge on how to make them is a trade secret, so it's not patented. Seems weird that it would be hard to recreate something from the 1950's.
I also wonder how many hours were wasted by people who had to use inferior technology because Disney kept it secret. Cutting out animals and objects from the background 1 frame at a time seems so mindnumbingly boring.
Yeah, that's just nonsense. We used sodium vapor monochromatic bulbs in my high school physics class to duplicate the double slit experiment.
I suspect the real reason is that digital green screen in the hands of experienced people is "good enough" vs the complication of needing a double camera and beam splitting prism rig and such.
The community has managed to drastically lower hardware requirements, but so far I think only Nvidia cards are supported, so as an AMD owner I'm still missing out :(
See also this video comparing Corridor Key to traditional keyers:
Watched this a few days ago. The video is light on technical details, except maybe that they used CGI to generate training data.
I'm a software engineer that, like the vast majority of you, uses AI/agents in my workflow every day. That being said, I have to admit that it feels a little weird to hear someone who does not write code say that they built something, without even mentioning that they had an agent build it (unless I missed that).
I mean, the heading of the video says "he solved the problem," which I think is wise to pay a lot of attention to.
In an earlier video they made a couple years back about Disney's sodium vapor technique Paul Debevec suggested he was considering creating a dataset using a similar premise: filming enough perfectly masked references to be able to train models to achieve better keying. So it was interesting seeing Corridor tackle this by instead using synthetic data.
With regards to the sodium vapor process, an idea has been percolating in the back of my head ever since I saw that video. But I don't really have the budget to try it out.
theory: make the mask out of non-visable light
illuminate the backing screen in near Infra-Red light.
point two cameras at a splitting prism with a near IR pass filter(I have confirmed that such thing exists and is commercially available)
Leave the 90 degree(unaltered path) camera untouched, this is the visible camera.
Remove the IR filter from the 180 degree(filter path) camera, this is the mask camera.
Now you get a perfect non-color shifting mask(in theory), The splitting prism would hurt light intake. It might be worth it to try putting the cameras really close together , pointed same direction, no prism, and see if that is close enough.
Debevec tried a version of this: https://arxiv.org/abs/2306.13702
Don't humans and other warm objects also radiate IR?
That is far-IR, thermal stuff, Near-IR, 700 nanometer-ish is right below red in human vision.
Camera sensors can pick up a little near-IR so they have have a filter to block it. If that filter was removed and a filter to block visable light was used in place you would have a camera that can only see non-visable light. Poorly, the camera was not engineered to operate in this bandwidth, but it might be good enough for a mask. A mask that does not interfere with any visible colors.
I'll do you one better, which requires no special cameras (most have IR filters) nor double cameras or prisms.
Shoot the scene in 48 or 96 fps. Sync the set lighting to odd frames. Every odd frame, the set lights are on. Every even frame, set lights are off.
For the backing screen, do the reverse. Even frames, the backing screen is on. Odd frames, backing screen is off.
There you go. Mask / normal shot / Mask normal shot / Mask ... you get the idea.
Of course, motion will cause normal image and mask go out of sync, but I bet that can be remedied by interpolating a new frame between every mask frame. Plus, when you mix it down to 24fps you can introduce as much motion blur and shutter angle "emulation" as you want.
This is called “ghost frame” and already exists in Red cameras and virtual production wall tools like Disguise.
From ~04:10 till 05:00 they talk about sodium-vapor lights and how Disney has the exclusive rights to use it. From what I read the knowledge on how to make them is a trade secret, so it's not patented. Seems weird that it would be hard to recreate something from the 1950's.
I also wonder how many hours were wasted by people who had to use inferior technology because Disney kept it secret. Cutting out animals and objects from the background 1 frame at a time seems so mindnumbingly boring.
Yeah, that's just nonsense. We used sodium vapor monochromatic bulbs in my high school physics class to duplicate the double slit experiment.
I suspect the real reason is that digital green screen in the hands of experienced people is "good enough" vs the complication of needing a double camera and beam splitting prism rig and such.
The community has managed to drastically lower hardware requirements, but so far I think only Nvidia cards are supported, so as an AMD owner I'm still missing out :(
See also this video comparing Corridor Key to traditional keyers:
https://www.youtube.com/watch?v=abNygtFqYR8
Watched this a few days ago. The video is light on technical details, except maybe that they used CGI to generate training data.
I'm a software engineer that, like the vast majority of you, uses AI/agents in my workflow every day. That being said, I have to admit that it feels a little weird to hear someone who does not write code say that they built something, without even mentioning that they had an agent build it (unless I missed that).
I mean, the heading of the video says "he solved the problem," which I think is wise to pay a lot of attention to.
Pretty impressive results! Seems like someone has even made a GUI for it: https://github.com/edenaion/EZ-CorridorKey
Still Python unfortunately.
Looking forward to trying it out, 8gb of vram or unified memory required!