We propose exploiting hallucination technique to remove occlusion in a face image. This is essentially a learning-based approach to obtain a de-occluded face image from a single occluded one. Hallucination works well because it applies domain knowledge, learned from faces in general, to a given novel face. The key idea is to predict the appearance of the occluded region by using the best (most correlated) un-occluded region, which can be learned using a set of training images.

Given an occluded input image, we should ignore the occluded region, and infer the “true” contents (or equivalently, its PCA coefficients) from the corresponding most correlated region. The prior knowledge for recovery includes the auto-covariance matrix of the most correlated region and the cross-covariance matrix of this occluded region with its most correlated region, which are both learned from the training dataset.

By applying de-occlusion as a pre-processing step, we can handle face recognition of partially occluded images more efficiently. Our technique provides the first reasonable solution to the face occlusion problem. Some face de-occlusion results are shown in the following figures.