Cylindrical Panoramic Transfer

and Appearance Prediction

for the Match of Real Observations

Salvatore Livatino

Email: sl@cvmt.dk

The summation of natural phenomena which can be synthesized in realistically generated virtual views may provide the user with a strong sense of presence (based on the visual realism). This can be the case, even if not all the original physical effects are "correctly" transferred to the newly generated view. A physically-based image mapping is consequently not always necessary in order to provide visual realism. However, when the goal of virtual-view synthesis is not a realistic visualization but, as in case of (Livatino [14]), the match of real observations, a physically based image mapping represents an important factor to take into account, and it has a great influence on the way the image-matching process should be designed, (e.g. if it should be feature- or correlation- based).

The class of image-based rendering techniques characterized as "geometric-valid pixel reconstruction", [11], typically uses relatively small number of images because of the application of geometric constraints, (either recovered at some stage or known a priori), to reproject image pixels appropriately at a given camera viewpoint. The geometric constraints can be of the form of known depths or correspondence values, epipolar constraints between pairs of images, or trilinear tensors that link correspondences between triplets of images. The geometric constraints can also be exploited to solve the visibility problem, (i.e. when an object or scene surface appears in front of another object or surface, even if it should lie behind).

In the literature different possibilities can be found within geometrically-valid pixel reprojection, depending on the chosen method and available information. These are mainly characterized by the use of trilinear tensors, (Shashua [21], Avidan et al. [1], Hartley [10]) and fundamental matrix, (Faugeras [9], Leveau et al. [13]). The image reprojection is very often based on a direct exploitation of known depths, correspondences, (Chen and Williams [6], Chang et al. [5], Seitz and Dyer [20]), epipolar constraints, (McMillan and Bishop [19], Kang and Szeliski [12]), etc.

It is in the exploitation of epipolar constraints, that the author have focused his attention and got inspired concerning the transfer of pixel values between cylindrical reference-views to a new viewing position, where the application context is a realistic visualization of landmark visual predictions for the purpose of mobile robot navigation, (Livatino [14], [17]). In particular, based on recent developments in image-based rendering involving the use of the Plenoptic function, (which describes light rays visible at any point in space), and on cylindrical panoramic images, (McMillan and Bishop [19], Kang [11]), it is proposed the interpolation of cylindrical panoramic images. The exploitation of cylindrical panoramic views for the purpose of mobile robot navigation is receiving increasing attention in the recent years, (Yuen and MacDonald [22]). In addition, some authors have very recently announced as their future research activity in mobile robotics, the synthesis of virtual views based on cylindrical panoramic references (Bunschoten and Kröse [4]).

Cylindrical panoramic images can "naturally" be acquired by a robotic system during its navigation in a learning phase, and they can represent the basis from where landmark reference-views can be extracted, (Livatino [14], [15]). Then, during the self-localization phase, the proposed interpolation of cylindrical reference-views can be used to render realistic visual predictions and to match current observations. Figure 1 visually describes the transfer of texture values from cylindrical reference-views to the visual prediction for an example landmark.

Figure: The figure visually describes the transfer of texture values from cylindrical reference-views to the visual prediction for an example landmark, (the "ideal" landmark). The top-right image represents the "current observation" where the visual prediction represent the view re-mapped from reference cylindrical images according to the current estimate of camera viewpoint. The visual prediction can then used as template in an image correlation process to locate the landmark precisely in the image-plane.

The transfer of pixels to the new (virtual) view by interpolation of cylindrical landmark reference-views is proposed to be performed in the following way.

1.

Cylindircal Pixel Transfer. I.e. mapping geometrical pixel correspondences between reference landmark views and visual prediction. For every pixel in the visual prediction, the corresponding pixels in the reference views are calculated by estimating the angular disparity. This mapping follows the idea of "plenoptic transfer", (McMillan and Bishop [19]).

2.

View-Adapted Texture Mapping. I.e. mapping texture correspondences between reference views and visual prediction. For every pixel in the visual prediction, its texture value (light intensity) is calculated by a weighted average of corresponding texture values in the reference views. This follows the idea of view-dependent texture mapping, (Debevec et al. [7]), where texture in the visual prediction can primarily be based on reference-images taken from viewpoints which are the closer to current observation. Closer images are in fact expected to best approximate landmark visible aspects and local illumination effects present in current camera observation.

It is based on the realism of virtual-views generated by the proposed method the author aim of improving the match between landmark visual prediction and current observation, (the match between landmark visual prediction and current observation can be used for mobile robot self-positioning, Livatino and Madsen [16]). The proposed technique thus represents the system's "answer" to the issue of synthesizing realistic and geometric-valid visual predictions.