Dynamic Near-regular Texture Tracking and Manipulation

 

Overview

 

A near-regular texture (NRT) is a geometric and photometric deformation from its regular origin -- a congruent wallpaper pattern formed by 2D translations of a single tile. A dynamic NRT is an NRT under motion. Correspondingly, the basic unit of a dynamic NRT is a well-defined texton, as a geometrically and photometrically deformed tile, moving through a 3D spatiotemporal space. Although NRTs are pervasive in man-made and natural environments, effective computational algorithms for NRTs are few. Through a systematic and quantitative comparison study of multiple texture synthesis algorithms, we are able to show that faithful NRT synthesis has challenged most of the state of the art texture synthesis algorithms. Our recent work on static NRTs analysis and manipulation (SIGGRAPH 2004) is the first algorithmic treatment aimed specifically to preserve the regularity and randomness in real world near regular textures.

 

The theme of this project is to address computational issues in modeling, tracking and manipulating dynamic NRTs. One basic observation on dynamic NRT is its topology invariance property: the lattice structure of a dynamic NRT remains invariant despite its drastic geometry or appearance variations. We propose a lattice-based Markov-Random-Field (MRF) model for dynamic NRT in a 3D spatiotemporal space. Our dynamic NRT model consists of a global lattice structure that characterizes the topological constraint among multiple textons and an image observation model that handles local geometry and appearance variations. Our model behaves like a network of statistically varied springs. Based on our dynamic NRT model, we develop a tracking algorithm that can effectively handle the special challenges of dynamic NRT tracking, including: ambiguous correspondences, occlusions, illumination variations, and appearance variations. Furthermore, we implement a dynamic NRT manipulation system that can replace and superimpose augmented images on a dynamic NRT from an unknown environment.

 

Publications

 

        Wen-Chieh Lin and Yanxi Liu, Tracking Dynamic Near-regular Textures under Occlusions and Rapid Movements, 9th European Conference on Computer Vision. Graz, Austria. May, 2006.

        Wen-Chieh Lin, A Lattice-based MRF Model for Dynamic Near-regular Texture Tracking and Manipulation, Technical Report CMU-RI-TR-05-58, Ph.D. Thesis, Robotics Institute, Carnegie Mellon University, Dec, 2005.

        Wen-Chieh Lin and Yanxi Liu, A Lattice-based MRF Model for Dynamic Near-regular Texture Tracking, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 29, No. 5, 2007, pp. 777-792.

 

Links to results

 

To play the following videos, a quicktime player and a media player are required.

The avi format is DivX 5.2 (http://www.divx.com/divx or http://www.divx-digest.com/software/divxcodec5.html)

 

If you use the Microsoft IE browser, please temporarily allow pop-ups and active controls.

Please click on images to play videos.

 

Texton Detection Results

 

 

 

Tracking Dynamic NRTs without Occlusion

 

Slowly waving cloth

 

Underwater texture tracking and replacement

(A texture placed under a water tank was seen through disturbed water)

Playing at normal speed, 30 fps

Playing 10 times slower, 3 fps

 

Playing at normal speed, 30 fps

Playing 4 times slower, 7.5fps

 

 

Tracking Dynamic NRTs with Occlusion

 

Crowd motion

(Special thanks to Yu and Wu for running their algorithm on this video)

 

Fabric texture

(Special thanks to Guskov for providing the input video)

top-left: visibility map of aligned textons

results with visibility map

 

 

Validation and Comparison

 

Different neighborhood systems

 

Multiple texton templates vs. PCA texton template

The tracking result using multiple texton templates is better (see frames 16, 68, 88)

 

Different initial texton positions

 

 

Video Editing Applications

Texture replacement of a fabric texture

tracked lattices and lighting deformation field

texture replacement results

Texture replacement of an underwater texture

 

Video superimposing:

USA is superimposed on the dress