Imagine looking into another room through a glass window and as you walked around in your room the perspective of everything in the room through the window changed just as in real life. In other words, when you move, the video follows, adjusting itself in real-time to give the effect that it was a real window! Everything displayed of course is life size!
The Kinect makes this possible by having a depth detecting feature, allowing 3d video capture. 10 Kinect cameras are used for capture and 1 for tracking.
I am imaging that the screen/window was a cylinder in the middle of the room that you could walk around. Is that possible?
There's also a whitepaper from MIT