Home

NRF (2024~2027)

National Research Institute of Cultural Heritage (2024~2025)

This work presents the restoration of drawings of wooden built heritage. Hand-drawn drawings contain the most important original information but are often severely degraded over time. A novel restoration method based on the vector quantized variational autoencoders is presented. Latent space representations of drawings and noise are learned, which are used to map noisy drawings to clean drawings for restoration and to generate authentic noisy drawings for data augmentation. The proposed method is applied to the drawings archived in the Cultural Heritage Administration. Restored drawings show significant quality improvement and allow more accurate interpretations of information.

AI Innovation Park, 2023~2024

Traditional wooden buildings deform over time and are vulnerable to fire or earthquakes. Therefore, traditional wooden buildings require continuous management and repair, and securing architectural drawings is essential for repair and restoration. Unlike modernized CAD drawings, traditional wooden building drawings scan and store hand-drawn drawings, and in this process, many noise is included due to damage to the drawing itself. These drawings are digitized, but their utilization is poor due to noise. Difficulties in systematic management of traditional wooden buildings are increasing. Noise removal by existing algorithms has limited drawings that can be applied according to noise characteristics and the performance is not uniform. This study presents deep artificial neural network based noised reduction for architectural drawings. Front/side elevation drawings, floor plans, detail drawings of Korean wooden treasure buildings were considered. First, the noise properties of the architectural drawings were learned with both a cycle generative model and heuristic image fusion methods. Consequently, a noise reduction network was trained through supervised learning using training sets prepared using the noise models. The proposed method provided effective removal of noise without deteriorating fine lines in the architectural drawings and it showed good performance for various noise types.

This study presents a deep neural network (DNN)-based safety monitoring method. Nonstationary objects such as moving workers, heavy equipment, and pallets were detected, and their trajectories were tracked. Time-varying safety zones (SZs) of moving objects were estimated based on their trajectories, velocities, proceeding directions, and formations. SZ violations are defined by set operations with sets of points in the estimated SZs and the object trajectories. The proposed methods were tested using images acquired by CCTV cameras and virtual cameras in 3D simulations in plants and on loading docks. DNN-based detection and tracking provided accurate online estimation of time-varying SZs that were adequate for safety monitoring in the workplace. The set operation-based SZ violation definitions were flexible enough to monitor various violation scenarios that are currently monitored in workplaces. The proposed methods can be incorporated into existing site monitoring systems with single-view CCTV cameras at vantage points.

Selfies are a popular form of photography. However, due to physical constraints, the compositions of selfies are limited. We present algorithms for creating virtual portraits with interesting compositions from a set of selfies. The selfies are taken at the same location while the user spins around. The scene is analyzed using multiple selfies to determine the locations of the camera, subject, and background. Then, a view from a virtual camera is synthesized. We present two use cases. After rearranging the distances between the camera, subject, and background, we render a virtual view from a camera with a longer focal length. Following that, changes in perspective and lens characteristics caused by new compositions and focal lengths are simulated. Second, a virtual panoramic view with a larger field of view is rendered, with the user’s image placed in a preferred location. In our experiments, virtual portraits with a wide range of focal lengths were obtained using a device equipped with a lens that has only one focal length. The rendered portraits included compositions that would be photographed with actual lenses. Our proposed algorithms can provide new use cases in which selfie compositions are not limited by a camera’s focal length or distance from the camera.

Innopolis Foundation 2022

Korea Shipbuilding & Offshore Engineering Co., Ltd. KSOE 2022

Korea Shipbuilding & Offshore Engineering Co., Ltd. KSOE 2022

Related Projects: AI-based traditional architectural hand drawing to CAD conversion Nonstationary and asymmetric lens blur restoration 8K 120Hz AR/VR display SoC HDR color image processing Embedded image processing for low-power LED public displays Smart pico projector parts and engine for 3D HD images Layerwise super resolution Generalized minimal residual based parallel image restoration Contrast enhancement for UHD displays

Related Project: Audio/video coding and light field media fundamental technologies for ultra realistic tera-media Object detection and distortion correction for 360 view camera system Computational photography with rotating selfies Multi-view image acquisition system for face recognition Dynamic view generation for augmented reality OpenGL based SMMD Simulator

Related Project: UniBrain – Ultimate neuromorphic intelligent brain system engineering

Related Project: Self-elevating crane system for installation and maintenance of onshore wind power systems AI-based simultaneous product classification and quality evaluation Abnormal behavior detection in loading dock Inteligent image masking Pedestrian behavior recognition Back seat passenger detection Personal protective equipment detection and safety distance monitoring Deep network optimization for pedestrian detection Infra-red camera based vehicular part quality evaluation Deep belief network for face recognition Vision-based command recognition for cleaning robot

Abstract – This work presents a neural network that consists of nodes with heterogeneous sensitivity. Each node in a network is assigned a variable that determines the sensitivity with which it learns to perform a given task. The network is trained by a constrained optimization that maximizes the sparsity of the sensitivity variables while ensuring the network’s performance. As a result, the network learns to perform a given task using only a small number of sensitive nodes. Insensitive nodes, the nodes with zero sensitivity, can be removed from a trained network to obtain a computationally efficient network. Removing zerosensitivity nodes has no effect on the network’s performance because the network has already been trained to perform the task without them. The regularization parameter used to solve the optimization problem is found simultaneously during the training of networks. To validate our approach, we design networks with computationally efficient architectures for various tasks such as autoregression, object recognition, facial expression recognition, and object detection using various datasets. In our experiments, the networks designed by the proposed method provide the same or higher performance but with far less computational complexity.

Korea Energy Technology Evaluation and Planning 2021-2024

ETRI 2021~2023

ETRI 2021-2022

Hanguk mold Co., Ltd./UNIST 2021

Abstract – Images acquired through a lens show nonstationary blur due to defocus and optical aberrations. This paper presents a method for accurately modeling nonstationary lens blur using eigen blur kernels obtained from samples of blur kernels through principal component analysis. Pixelwise variant nonstationary lens blur is expressed as a linear combination of stationary blur by eigen blur kernels. Operations that represent nonstationary blur can be implemented efficiently using the discrete Fourier transform. The proposed method provides a more accurate and efficient approach to modeling nonstationary blur compared with a widely used method called the efficient filter flow, which assumes stationarity within image regions. The proposed eigen blur kernel-based modeling is applied to total variation restoration of nonstationary lens blur. Accurate and efficient modeling of blur leads to improved restoration performance. The proposed method can be applied to model various nonstationary degradations of image acquisition processes, where degradation information is available only at some sparse pixel locations.

Abstract – In this study, we present deep neural networks with a set of node-wise varying activation functions. The feature-learning abilities of the nodes are affected by the selected activation functions, where the nodes with smaller indices become increasingly more sensitive during training. As a result, the features learned by the nodes are sorted by the node indices in order of their importance such that more sensitive nodes are related to more important features. The proposed networks learn input features but also the importance of the features. Nodes with lower importance in the proposed networks can be pruned to reduce the complexity of the networks, and the pruned networks can be retrained without incurring performance losses. We validated the feature-sorting property of the proposed method using both shallow and deep networks as well as deep networks transferred from existing networks.

BK21 2020~2027

POSCO 2020

Korea Shipbuilding & Offshore Engineering Co., Ltd. KSOE 2020

Korea Evaluation Institute of Industrial Technology, KEIT 2020-2022

SL Corporation 2020

SL Corporation 2020

Abstract –  In many applications, images acquired from multiple cameras are stitched to form an image with a wide viewing angle. We propose a method of estimating a seam line using motion information to stitch multiple images without distortion of the moving object. Existing seam estimation techniques usually utilize an energy function based on image gradient information and parallax. In this paper, we propose a seam estimation technique that prevents distortion of moving object by adding temporal motion information, which is calculated from the gradient information of each frame. We also propose a measure to quantify the distortion level of stitched images and to verify the performance differences between the existing and proposed methods.

UNIST 2019

POSCO 2019

ETRI 2019

SL Corporation 2018

Abstract –  A novel saliency detection algorithm for videos is proposed in this paper. We adaptively determine the weights of color and motion features to extract combined global feature contrast by adopting the compactness prior of salient object. We localize a saliency searching area in a current frame using the saliency distribution computed at the previous frame. We estimate the saliency by computing a relative feature distance with respect to the salient object and local background, which is weighted by global feature contrast. Experimental results show that the proposed algorithm captures salient objects faithfully on various videos, and outperforms the state-of-the art video saliency detection methods qualitatively and quantitatively.

Abstract – How we perceive property of surfaces with distinct geometry and reflectance under various illumination conditions is not fully understood. One widely studied approach to understanding perceptual surface property is to derive statistics from images of surfaces with the goal of constructing models that can estimate surface property attributes. This paper presents machine-learning-based methods to estimate the lightness and glossiness of surfaces. Instead of deriving image statistics and building estimation models on top of them, we use deep networks to estimate the perceptual surface property directly from surface images. We adopt the attention models in our networks to allow the networks to estimate the surface property based on features in certain parts of images. This approach can rule out image variations due to geometry, reflectance, and illumination when making the estimations. The networks are trained with perceptual lightness and glossiness data obtained from psychophysical experiments. The trained deep networks provide accurate estimations of the surface property that correlate well with human perception. The network performances are compared with various image statistics derived for the estimation of perceptual surface property.

Abstract – Optical aberrations of a lens introduce lens blur to photographed images. Lens blur is non-stationary with the amount and characteristics of blur varying depending on spatial pixel locations in an image. This work presents non-stationary deep networks for the restoration of non-stationary lens blur. Deep networks have relatively larger receptive fields. However, the receptive fields of stationary deep networks are not wide enough for the networks to cope with the non-stationarity of lens blur that span the entire image. We use spatial pixel locations as an additional input to networks to let the network utilize location dependent features to handle the non-stationarity. Experimental results show that even shallower non-stationary networks provide better performance than deeper stationary networks. The non-stationary networks are trained from pairs of images photographed at different aperture settings, eliminating the necessity of estimation or measurement of pixel-wise variant non-stationary lens blur.

Abstract – This paper presents a recurrent learning-based facial attribute recognition method that mimics human observers’ visual fixation. The concentrated views of a human observer while focusing and exploring parts of a facial image over time are generated and fed into a recurrent network. The network makes a decision concerning facial attributes based on the features gleaned from the observer’s visual fixations. Experiments on facial expression, gender, and age datasets show that applying visual fixation to recurrent networks improves recognition rates significantly. The proposed method not only outperforms state-of-the-art recognition methods based on static facial features, but also those based on dynamic facial features.

Korea Evaluation Institute of Industrial Technology, KEIT 2018

Noa LED/UNIST 2017

Abstract – Applying alpha matting to large images is a challenging task because of its computational complexity. This paper provides a divide and conquer strategy for performing closed-form matting. The matting problem, defined for an entire image, is broken down into systems of linear equations defined for very small blocks of the image. The sizes of the small systems are small enough for us to find solutions efficiently using a direct sparse linear equation system solver. The small systems are solved following a sequential order such that the alpha matte grows from a user scribble. With the block sequential application, matting is performed on fan-shaped partitions in parallel on multiple processing cores. Experiments on large test images as well as on standard benchmark test images show that the proposed parallel block sequential matting provides high quality alpha mattes with good scalability.

Abstract – In this paper we propose an efficient method for a convex optimization problem which involves a large nonsymmetric and non-Toeplitz matrix. The proposed method is an instantiation of the alternating direction method of multipliers applied in Krylov subspace. Our method offers significant advantages in computational speed for the convex optimization problems involved with general matrices of large size. We apply the proposed method to the restoration of spatially variant blur. The matrix representing spatially variant blur is not block circulant with circulant blocks (BCCB). Efficient implementation based on diagonalization of BCCB matrices by the discrete Fourier transform is not applicable for spatially variant blur. Since the proposed method can efficiently work with general matrices, the restoration of spatially variant blur is a good application of our method. Experimental results for total variation restoration of spatially variant blur show that the proposed method provides meaningful solutions in a short time.

National Research Foundation of Korea, NRF 2016~2022

Abstract – Fingerprint recognition systems are vulnerable to impersonation by fake or spoof fingerprints. Fingerprint liveness detection is a step to ensure whether a scanned fingerprint is live or fake prior to a recognition step. This paper presents a fingerprint liveness detection method based on a deep belief network (DBN). A DBN with multiple layers of restricted Boltzmann machine is used to learn features from a set of live and fake fingerprints and also to detect the liveness. The proposed method is a systematic application of a deep learning technique, and does not require specific domain expertise regarding fake fingerprints or recognition systems. The proposed method provides accurate detection of the liveness with various sensor datasets collected for the international fingerprint liveness detection competition.

Abstract – Images acquired by a camera show lens blur due to imperfection in the optical system even when images are properly focused. Lens blur is non-stationary in a sense that the amount of blur depends on pixel locations in a sensor. Lens blur is also asymmetric in a sense that the amount of blur is different in the radial and tangential directions, and also in the inward and outward radial directions. This paper presents parametric blur kernel models based on the normal sinh-arcsinh distribution function. The proposed models can provide flexible shapes of blur kernels with a different symmetry and skewness to model complicated lens blur due to optical aberration in a properly focused images accurately. Blur of single focal length lenses is estimated, and the accuracy of the models is compared with the existing parametric blur models. An advantage of the proposed models is demonstrated through deblurring experiments.

Abstract – This chapter presents an interpolation method of low-computation for a Region Of Interest (ROI) using multiple low-resolution images of the same scene. Interpolation methods using multiple images require the accurate motion information between the reference image of interpolation and the other images. Sometimes complex local motions applied to the entire images are estimated incorrectly, yielding seriously degraded interpolation results. The authors apply the proposed Superresolution (SR) method, which employs a simple global motion model, only to the ROI that contains important information of the scene. The ROIs extracted from multiple images are assumed to have simple global motions. At first, using a mean absolute difference measure, they extract the regions from the multiple images that are similar to the selected ROI in the reference image of interpolation and use feature points to estimate the affine motion parameters. The authors apply the Projection Onto Convex Sets (POCS)-based method to the ROI using the estimated motion, simplify the iterative computation of the whole system, and use an edge-preserving smoothing filter to reduce the distortion caused by additive noise. In experiments, they acquire test image sets with a hand-held digital camera and use a Gaussian noise model. Experimental results show that the feature-based Motion Estimation (ME) is accurate and reducing the computational load of the ME step is efficient in terms of the computational complexity. It is also shown that the SR results using the proposed method are remarkable even when input images contain complex motions and a large amount of noise. The proposed POCS-based SR algorithm can be applied to digital cameras, portable camcorders, and so on.

Suprema 2014~2015

Abstract – In this paper, we propose a novel contrast enhancement algorithm for low light level images, which preserves image details and color constancy based on Retinex. We decompose an input low contrast image into luminance and chrominance components in Lab color space, which reflects the perception characteristics of human visual system well, and enhance the luminance component only. We first estimate illumination using adaptive bilateral filtering, which guarantees the available range of reflectance by considering proper neighboring pixels according to their luminance and color values. Then we enhance the contrast of the estimated illumination image using parabola-based tone mapping function. Finally, the enhanced luminance and the original chrominance are combined together to yield an enhanced color image. Experiment results show that the proposed algorithm enhances image details and edge structures by alleviating halo artifacts, and also preserves naturalness faithfully by avoiding color shifting artifacts.

Abstract – Environmental illumination can degrade performance of face recognition systems. This paper presents a method to reduce the effect of environmental illumination for face recognition with near infrared (NIR) imaging systems. The camera response function that maps the image irradiance to image brightness is estimated via principal component analysis with pairs of facial images taken under different lighting conditions. The inverse of camera response function is used to map two facial images of a subject taken with and without frontal NIR lighting under ambient light. The irradiance of facial image with frontal NIR lighting in the absence of ambient light is estimated using the inverse camera response function. Then the brightness image with only the frontal NIR lighting is obtained from the irradiance to be used for face recognition. Since the brightness images are used by face recognition system when there is no ambient light, the same face recognition algorithm and database can be used without any modification. The proposed method is a pre-processing method and does not require additional hardware. Performance is evaluated and compared to existing methods at various false matching rates. The proposed method can enhance the performance of face recognition in the presence of ambient light at false matching rates at which practical biometric authentication systems operate.

Abstract – The color, gloss, and texture (i.e., pearliness) of 15 glossy samples containing pearl flakes were investigated. Psychophysical experimental data from 21 observers were compared with measurement data. Color measurement data obtained using the CIE D/0 and ASTM E2539-08 multiangle geometry did not predict the overall color appearance variation of pearly samples. Pearly samples have a lower perceived glossiness than non-pearly surfaces with the same level of gloss treatment, but a much higher measured gloss. Pearliness describes the texture of pearly samples well and can be predicted as a function of the pearl flakes’ average size and area coverage measured from magnified surface images. These results suggest that an image statistics approach is required to properly describe the visual appearance of pearly surfaces.

Abstract – Scanned laser pico projectors utilising laser light sources and microelectromechanical systems mirror devices introduce geometric distortion to the displayed images because of nonlinear mirror driving methods. A pre-processing algorithm is proposed that corrects the nonlinear geometric distortion of scanned laser pico projectors. The distortion is modelled as blockwise homography transforms. The geometry of the images being displayed is pre-distorted using the pixel indices obtained by the backward blockwise homography transforms. The pre-distorted images are displayed by the pico projector whose geometrical distortion undoes the pre-distortion such that distortion-free images can be displayed. The proposed algorithm requires a small amount of information stored in the memory and three bilinear interpolations per pixel, thus providing an efficient way to correct nonlinear distortion without additional optical components, such as aspherical lenses.

LG Electronics 2015

Korea Evaluation Institute of Industrial Technology, KEIT 2013~2016

Samsung Electronics 2014~2105

Abstract – Perceptual resolution and crosstalk of active and passive stereoscopic 3D displays are evaluated by subjective tests. Test patterns of gratings with various contrast and spatial frequencies are used to determine the contrast thresholds, from which the contrast sensitivity of human visual systems is obtained. Perceptual resolution tests show that viewers can perceive the highest vertical frequency on active displays, but not on passive displays. However, the contrast sensitivity is higher for passive displays than active displays for all other frequencies. Viewers can perceive frequency components with lower contrast, except for the highest frequency, more easily on passive displays than on active displays. Perceptual crosstalk tests show that interocular crosstalk is a function of spatial frequency. Crosstalk is lower on passive displays than on active displays.

Abstract – A crosstalk characterization model is proposed that shows the relationship between colors with and without crosstalk by using CIE tristimulus values. The proposed model is based on the 3D color measurement results of two different stereoscopic 3D televisions with shutter glasses, namely, a PDP and an LCD. The PDP exhibits a significant dependence of crosstalk on channel and tristimulus values, meaning that different channels and wavelengths result in different degrees of crosstalk, while the LCD exhibits little channel or wavelength dependence. Moreover, for PDP, the proportion of leaked light increases as the luminance of the opposite view decreases. In spite of different crosstalk characteristics, both PDP and LCD show similar performance resulting in 1.5ΔE* ab and 1.9ΔE* ab average color difference between the measured and the predicted. The proposed model could be used to develop a better crosstalk metric or a crosstalk cancellation algorithm by more accurately predicting colors including crosstalk.

Abstract – This paper provides a vision based HCI method for a user to command a cleaning robot to move to a specific location in home environment. Six hand poses are detected from a video sequence taken from a camera on the cleaning robot. AdaBoost based hand-pose detectors are trained with a reduced Haar-like feature set to make the detectors robust to the influence of the complex background. The first three stages of the cascade in the six detectors are used as pose estimation to reduce the computational complexity. The cleaning area is determined from the detected pose. The performances of the proposed detectors are validated with a set of test images with cluttered background. The cleaning area control is simulated with real-world video sequences. The proposed method can effectively control a cleaning robot without the need for a user to wear or employ any input devices.

Abstract – The perceived glossiness of 12 flat samples and 18 bumpy samples with various colors and gloss levels is estimated by 13 observers using magnitude estimation technique. Each sample is measured with the gloss-meter as well. It is found that bumpy surface shows lower measured gloss level than flat surface treated with the same level of UV coating. The perceived glossiness of bumpy surface is higher than that of flat surface with low level UV coating treatment while perceived glossiness of bumpy surface is lower than that of flat surface with high level UV coating treatment.

National Research Foundation of Korea, NRF 2012~2015

Suprema 2012~2013

Abstract – A particle filtering based block-wise estimation method for estimation of motion in a video sequence and joint estimation of disparity and motion in a stereo video sequence is proposed. Parameters of motion and disparity of a block in a sequence are defined as a state, and evolution of the state with respect to the block index is tracked with particle filtering. The state is assumed to be dependent on the states of neighboring blocks. Estimated motion and disparity fields are consistent and suitable for intermediate frame or view generation. The particle filter provides a method to effectively sample the search space. The particles are concentrated in regions where the probability density function for the state has large values. Hence, the locations of the particles are good candidates for search points of a fast search method. The proposed method can estimate motion and disparity with a fraction of search points necessary for conventional estimation methods.

Abstract – A reduced reference picture quality measure is proposed for MPEG-2 compressed videos. The measure can be calculated from the ratio of three N tap one-dimensional discrete cosine transform (DCT) coefficients. The proposed measures are highly correlated to subjective test scores and can provide accurate assessment of picture quality for a certain type of sequences.

LG Electronics 2011

National Research Foundation of Korea, NRF 2009~2012

Samsung Electronics 2011~2012

LG Electronics 2011

Suprema 2010~2011

Lim, B.R., Lee, H.S., Park, R.H. and Yang, S., 2009. A wavelet packet-based noise reduction algorithm of NTSC images using CVBS characteristics. IEEE Transactions on Consumer Electronics, 55(4), pp.2407-2415.

Abstract – According to advances in digital imaging technology, interest in high-resolution (HR) images has been increased. Various methods that convert low-resolution (LR) images to HR ones have been presented. In this paper, to reduce the computational load we propose a vector quantization (VQ) based algorithm that reconstructs an interpolation image by adding to an initially interpolated image high-frequency components predicted from training with a number of example image sets. The proposed interpolative classified VQ (ICVQ) algorithm combines interpolative VQ with classified VQ. With a number of (LR and HR) example image sets, we construct two types of (LR and HR) codebooks. Comparative experiments with three conventional image interpolation algorithms show that the proposed interpolation algorithms using ICVQ effectively preserve edges to which the human visual system is sensitive. The proposed algorithm can be applicable to various image- and video-based applications such as digital camera and digital television.

Abstract – This paper presents a low-computational superresolution method of a region of interest (ROI) using multiple low-resolution input images of the same scene. We extract the regions from the multiple input images that are similar to the selected ROI in the reference image and use feature points to estimate the affine motion parameters. We apply a projection onto convex sets based method to interpolate the ROI using the estimated motion and simplify the iterative computation of the whole system, in which an edge-preserving smoothing filter is utilized to reduce the motion compensation error caused by additive noise. Experiments with several test image sets show the effectiveness of the proposed method.