Neural Rendering + Displays

2022

1. Opt. Exp.

   

   J4 Neural distortion fields for spatial calibration of wide field-of-view near-eye displays

   Yuichi Hiroi, Kiyosato Someya, and Yuta Itoh

   Opt. Express, Oct 2022

   Award Selected as one of Japan’s leading optics research achievements of 2022, Optical Society of Japan (2023).

   Abs Bib DOI Video

   We propose a spatial calibration method for wide field-of-view (FoV) near-eye displays (NEDs) with complex image distortions. Image distortions in NEDs can destroy the reality of the virtual object and cause sickness. To achieve distortion-free images in NEDs, it is necessary to establish a pixel-by-pixel correspondence between the viewpoint and the displayed image. Designing compact and wide-FoV NEDs requires complex optical designs. In such designs, the displayed images are subject to gaze-contingent, non-linear geometric distortions, which explicit geometric models can be difficult to represent or computationally intensive to optimize. To solve these problems, we propose neural distortion field (NDF), a fully-connected deep neural network that implicitly represents display surfaces complexly distorted in spaces. NDF takes spatial position and gaze direction as input and outputs the display pixel coordinate and its intensity as perceived in the input gaze direction. We synthesize the distortion map from a novel viewpoint by querying points on the ray from the viewpoint and computing a weighted sum to project output display coordinates into an image. Experiments showed that NDF calibrates an augmented reality NED with 90° FoV with about 3.23 pixel (5.8 arcmin) median error using only 8 training viewpoints. Additionally, we confirmed that NDF calibrates more accurately than the non-linear polynomial fitting, especially around the center of the FoV.

   @article{hiroi2022neural,
     author = {Hiroi, Yuichi and Someya, Kiyosato and Itoh, Yuta},
     journal = {Opt. Express},
     keywords = {Image metrics; Lens design; Near eye displays; Optical aberration; Optical systems; Systems design},
     number = {22},
     pages = {40628--40644},
     publisher = {Optica Publishing Group},
     title = {Neural distortion fields for spatial calibration of wide field-of-view near-eye displays},
     volume = {30},
     month = oct,
     year = {2022},
     url = {https://opg.optica.org/oe/abstract.cfm?URI=oe-30-22-40628},
     doi = {10.1364/OE.472288},
   }

2. ACM VRST

   CF5 NeARportation: A Remote Real-time Neural Rendering Framework

   Yuichi Hiroi, Yuta Itoh, and Jun Rekimoto

   In Proceedings of the 28th ACM Symposium on Virtual Reality Software and Technology, Tsukuba, Japan, Nov 2022

   Abs Bib DOI PDF Video

   While presenting a photorealistic appearance plays a major role in immersion in Augmented Virtuality environment, displaying that of real objects remains a challenge. Recent developments in photogrammetry have facilitated the incorporation of real objects into virtual space. However, reproducing complex appearances, such as subsurface scattering and transparency, still requires a dedicated environment for measurement and possesses a trade-off between rendering quality and frame rate. Our NeARportation framework combines server–client bidirectional communication and neural rendering to resolve these trade-offs. Neural rendering on the server receives the client’s head posture and generates a novel-view image with realistic appearance reproduction that is streamed onto the client’s display. By applying our framework to a stereoscopic display, we confirm that it can display a high-fidelity appearance on full-HD stereo videos at 35-40 frames per second (fps) according to the user’s head motion.

   @inproceedings{hiroi2022nearportation,
     pages = {1--5},
     month = nov,
     author = {Hiroi, Yuichi and Itoh, Yuta and Rekimoto, Jun},
     title = {NeARportation: A Remote Real-time Neural Rendering Framework},
     year = {2022},
     isbn = {9781450398893},
     publisher = {Association for Computing Machinery},
     address = {New York, NY, USA},
     url = {https://doi.org/10.1145/3562939.3565616},
     doi = {10.1145/3562939.3565616},
     booktitle = {Proceedings of the 28th ACM Symposium on Virtual Reality Software and Technology},
     articleno = {23},
     numpages = {5},
     keywords = {remote rendering, real-time rendering, neural rendering, augmented virtuality, appearance reproduction},
     location = {Tsukuba, Japan},
     series = {VRST '22},
   }

2019

1. ISMAR Poster

   

   CP4 OSTNet: Calibration Method for Optical See-Through Head-Mounted Displays via Non-Parametric Distortion Map Generation

   Kiyosato Someya, Yuichi Hiroi, Makoto Yamada, and Yuta Itoh

   In 2019 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), Oct 2019

   Abs Bib DOI PDF

   We propose a spatial calibration method for Optical See-Through Head-Mounted Displays (OST-HMDs) having complex optical distortion such as wide field-of-view (FoV) designs. Viewpoint-dependent non-linear optical distortion makes existing spatial calibration methods either impossible to handle or difficult to compensate without intensive computation. To overcome this issue, we propose OSTNet, a non-parametric data-driven calibration method that creates a generative 2D distortion model for a given six-degree-of-freedom viewpoint pose.

   @inproceedings{someya2019ostnet,
     author = {Someya, Kiyosato and Hiroi, Yuichi and Yamada, Makoto and Itoh, Yuta},
     booktitle = {2019 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct)},
     title = {OSTNet: Calibration Method for Optical See-Through Head-Mounted Displays via Non-Parametric Distortion Map Generation},
     year = {2019},
     volume = {},
     number = {},
     pages = {259-260},
     keywords = {Optical distortion;Calibration;Cameras;Nonlinear distortion;Two dimensional displays;Decoding;optical see through head mounted display;calibration;Variational Autoencoder},
     doi = {10.1109/ISMAR-Adjunct.2019.00-34},
     issn = {},
     month = oct,
   }