The lens system in a camera limits the bandwidth of the image projected onto the detector array. The detector array can then sample this image. If the sampling satisfies the Nyquist criterion, the image projected onto the detector can be exactly reconstructed from the samples. Detector elements need to be large to collect light efficiently. In infrared imaging, these elements also need to be isolated from each other, which makes the fill factor small and the pixel pitch large. This causes the array not to be sufficiently dense to sample according to Nyquist, and the sampled images are aliased.
Some IR camera manufacturers solve this undersampling problem by mounting the detector array on a piezoelectric element that allows scanning of the image projected onto the detector plane (‘microscanning’). The previous discussion is also valid for other types of camera; for example, Carl-Zeiss incorporates this technique is the ProgRes digital camera.
The approach we explored is to induce a random motion by vibration of the camera, and combine a series of frames to create a single image with a higher spatial resolution. Note that this can only work if there is a sub-pixel translation between those frames. This translation causes the image to be sampled at more points than provided by the detector array. Also, note that resolution cannot be improved upon if the frames are sampled correctly (according to Nyquist). The requirements that the algorithms must meet are:
We will assume that the scene does not change during the acquisition of the image sequence and the motion is small and random, as if caused by vibration of the camera.
Superresolution restoration requires knowledge about the relative shifts between the frames. This is extracted from the frames themselves using a registration algorithm. The second step is to fuse the data present in all the frames. We have developed and tested some methods that accomplish this.
Here you can see results acquired on one of the image sequences we recorded at TNO-FEL.
Fig 1. Scene as recorded using infrared camera.
Fig 2. Left: region of interest, contrast-stretched. Right: bicubic interpolation.
Fig 3. Left: result of applying MTS and NC to sequence of 10 images. Right: same on sequence of 25 images.
These are the publications produced by this project:
This project was continued by Tuan Pham (2002-2006), who wrote a fabulous PhD thesis, “Spatiotonal Adaptivity in Super-Resolution of Under-sampled Image Sequences,” and by Adam W.M. van Eekeren (2004-2008), with the very interesting PhD thesis “Super-Resolution of Moving Objects in Under-Sampled Image Sequences.”
© 2001-2010 Cris Luengo
Last modified June 2, 2010.