FTM1636 Super-Interpolation With Edge-Orientation-Based Mapping Kernels for Low Complex 2× Upscaling – IEEE MATLAB Project 2016 – 2017
With the advent of Ultra High Definition (UHD) video services, super-resolution (SR) techniques are often required to generate high-resolution (HR) images from low-resolution (LR) images such as High Definition (HD) images. To generate such HR images and video of UHD resolutions in limited computing devices with hardware and software, low complex but excellent SR methods are particularly required. In this paper, we present a novel and fast SR method, called Super-Interpolation (SI), by unifying an interpolation step and a quality-enhancement step. The proposed SI method utilizes edge-orientation-based pre-learned kernels, which inherits the simplicity of interpolation and the quality enhancement of SR. It performs SR directly from the initial resolution of an input image to the target resolution of an up-scaled output image without requiring any intermediate interpolated image. The proposed SI method involves off-line training and on-line up-scaling phases: In the off-line training phase, training LR image patches are clustered based on their edge orientations into different edge-orientation (EO) classes for which class-dependent linear mapping functions are learned between training LR and HR image patches; In up-scaling phase, an HR output image patch for each LR input image patch is generated by applying an appropriate linear mapping function selected based on the EO of LR input image patch. Our proposed SI method is intensively compared with ten state-of-the-art SR methods for common image sets and many HD/UHD images. The experimental results show that the SI method yields the smallest running time and requires relatively small hardware resources. It outperforms six state-of-the-art methods in average PSNR/SSIM, and exhibits competitive or somewhat lower PSNR/SSIM performance compared to the others.
UHDTVS (Ultra High Definition Televisions) are now slowly emerging in the consumer markets, thereby growing the needs for UHD video contents. Most of smartphones in near future will have a capability to support UHD video. However, due to costly acquisition devices and large bandwidth occupancy, there are not enough UHD contents available to suffice the UHD services. Also, many legacy contents of low spatial-resolution video are under services via IPTV and Internet. This has attracted the use of up-scaling techniques, super-resolution, which are able to convert small-resolution video to large-resolution ones, especially for video up-scaling by a magnification factor of 2 as in Full High Definition (FHD)-to-UHD or 4K-to-8K conversions. It is widely known that the reconstruction of high-resolution (HR) images from low-resolution (LR) inputs is an ill-posed inverse problem.
The proposed SI method comprises two phases: off-line training and on-line up-scaling: In the off-line training phase, training LR image patches are clustered based on their edge orientations into different edge-orientation (EO) classes. For each EO class, a class-dependent linear mapping function is learned between training LR and HR image patch pairs. Once learned, these multiple pre-trained linear mapping functions (kernels in matrix forms) are stored and can be globally used in any LR inputs; In the on-line up-scaling phase, when an LR input image patch is given, its EO is calculated and an appropriate linear mapping kernel is simply selected based on its EO class index from a look-up table that contains the pre-learned mapping kernels for different EO classes.
- Low-Complexity Up-Scaling Methods
- High-Quality-Oriented Up-Scaling Methods
- Edge-Orientation Analysis Stage
- Performance Analysis for the proposed SI method
- Implementation and Computational Complexity Issues
- Report Generation
Low-Complexity Up-Scaling Methods:
The low-complexity-oriented up-scaling, interpolation-based methods can be considered first. Yang proposed a fine edge-preserving image interpolation algorithm based on the local gradient features where the missing HR pixels are interpolated using the dominant edge direction. Giachetti introduced an interpolation algorithm that iteratively corrects interpolated pixels by minimizing an objective function of the second-order directional derivatives. Kang presented an edge-orientation based interpolation method where steerable filters are used to estimate edge orientations, and to generate HR images by applying the directionally adaptive truncated constrained least-squares filter.
High-Quality-Oriented Up-Scaling Methods:
Contrary to the methods in the high-quality-oriented up-scaling methods prefer reconstructed HR quality to reconstruction speed. These methods mostly exploit the strength of machine learning techniques, and try to enhance the quality of reconstructed HR images as high as possible. In a deep convolutional neural network is employed for SR. Peleg addressed single image SR using a statistical prediction model based on sparse representations. Wang proposed a sharpness-preserving interpolation technique based on displacement fields.
In particular, there are machine learning-based SR methods where the LR-HR correspondences are extracted from certain examples in dictionaries, which are called example-based SR methods. The example-based SR methods divide an LR input image into smaller image patches, and the examples similar to each patch are searched in pre-built dictionaries of image patches. The found examples are presumed to hold appropriate LR-HR correspondences between the input LR and target HR image patches. The found LR-HR correspondences are utilized for the current input LR image patch to reconstruct its HR version.
Edge-Orientation Analysis Stage:
SI method, we propose a simple but very effective image patch clustering scheme based on edge orientations. The previous interpolation methods have used edge-orientation information to selectively apply specific interpolation filters, which motivated our edge-orientation based clustering of image patches. Our image patch clustering scheme utilizes edge-orientations on LR image patches to classify them into appropriate EO classes, each of which constitutes one cluster of LR-HR image patch pairs with similar edge orientations. For each cluster, a very effective linear mapping function from LR to HR image patches can be learned for later use in the up-scaling phase.
Performance Analysis for the proposed SI method:
The objective quality performance of our proposed SI method is compared with bicubic interpolation and the five SR methods in terms of PSNR and structural similarity metric (SSIM). SCSR, JNB-SCSR and MLM were implemented with IBP. Compares the performances of the SR methods for TestSet-1 dataset. NEDI and ICBI have used down-sampled LR images by taking even pixels from HR images. The reconstructed HR images by their SR methods inherently turn out to be half-pixel shifted compared to the original HR images. Since the original half-pixel shifted HR images are not available in our experiments, we aligned them to their original HR images by bi-linear interpolation which is built-in function in MATLABTM. So, it is noted that due to such a bilinear interpolation for the alignment, NEDI and ICBI are compared to other methods with somewhat inevitable quality degradation. The best PSNR performance is achieved by SRCNN2015. However, SRCNN2015 and SRCNN2014 methods require to store 64 internal output buffers after applying 64 convolution filters for each interpolated LR input image in the first convolution layer, and 32 internal output buffers for serial processing after applying 32 convolution filters for each output of the first convolution layer, which is definitely impractical in low-complexity hardware implementation.
Implementation and Computational Complexity Issues:
The performance comparison for the proposed SI method and the other SR methods in terms of PSNR, SSIM and computation time for TestSet-2 which consists of 4K-UHD test images. ANR, APLUS, SRCNN2014, SRCNN2015 and JOR methods all outperform the proposed SI method. Since the proposed SI method categorizes input LR patches into homogeneous or non-homogeneous patch types by a predefined threshold value, more LR patches in images of higher spatial resolutions tend to be classified into the homogeneous patch type. From this, HR output patches are reconstructed by roughly learned linear mapping for the homogeneous patch type. The cropped regions of the reconstructed HR output images for the Children image of 4K-UHD resolution.
we presented a novel super-interpolation (SI) method, which unifies both simplicity of interpolation and quality enhancement of SR. Using pre-trained edge-orientation based linear mappings and interpolation-like scheme, our SI method can generate HR images of competitive quality in terms of PSNR and SSIM with respect to computation complexity and required hardware resources, and yield comparable subjective visual qualities with fine texture details and sharp edges, compared to the other state-of-the-art SR methods. Moreover, SI is a hardware-friendly scheme with one-step up-scaling that does not require i) a frame buffer for intermediate interpolated images, ii) iterative computations (IBP), iii) full search for patch examples and linear mapping functions, and iv) overlapping/averaging.
- System : Intel Core i3 2.8 GHz.
- Hard Disk : 250 GB.
- Monitor : 15” VGA Colour.
- Mouse : Logitech.
- Ram : 1 GB.
- Operating system : Windows 7.
- Software : Matlab with Simulink.
 E. Perez-Pellitero, J. Salvador, J. Ruiz-Hidalgo and B. Rosenhahn, “Accelerating super-resolution for 4K up scaling,” IEEE Trans. Cons. Elec., Las Vegas, USA, Jan. 2015, pp. 317-320.
 M. Sakurai, Y. Sakuta, M. Watanabe, T. Goto and S. Hirano, “Super-resolution through non-linear enhancement filters,” IEEE Int. Conf. on Image Proc., Melbourne, Australia, Sept. 2013, pp. 854-858.
 K. I. Kim and Y. Kwon, “Single-image super-resolution using sparse regression and natural image prior,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 32, no. 6, pp. 1127–1133, Jun. 2010.
 J. Yang, J. Wright, T. S. Huang, and Y. Ma, “Image super-resolution via sparse representation,” IEEE Trans. Image Process., vol. 19, no. 11, pp. 2861–2873, Nov. 2010.
 L. Zhang and X. Wu, “An edge-guided image interpolation algorithm via directional filtering and data fusion,” IEEE Trans. Image Process., vol. 15, no. 8, pp. 2226–2238, Aug. 2006.