publications

2025

1. IEEE TVCG

   

   ChromaGazer: Unobtrusive Visual Modulation using Imperceptible Color Vibration for Visual Guidance

   Rinto Tosa, Shingo Hattori, Yuichi Hiroi, Yuta Itoh, and Takefumi Hiraki

   IEEE Transactions on Visualization and Computer Graphics, 2025

   Abs arXiv Bib PDF

   Visual guidance (VG) plays an essential role in directing user attention in virtual reality (VR) and augmented reality (AR) environments. However, traditional approaches rely on explicit visual annotations, which often compromise visual clarity and increase user cognitive load. To address this issue, we propose an unobtrusive VG technique based on color vibration, a phenomenon in which rapidly alternating colors at frequencies above 25 Hz are perceived as a single intermediate color. Our work explores a perceptual state that exists between complete color fusion and visible flicker, where color differences remain detectable without conscious awareness of vibration. Through two experimental studies, we first identified the thresholds separating complete fusion, this intermediate perceptual state, and visible flicker by systematically varying color vibration parameters. Subsequently, we applied color vibrations with derived thresholds to natural image regions and validated their attention-guiding capabilities using eye-tracking measurements. The results demonstrate that controlled color vibration successfully directs user attention while maintaining low cognitive demand, providing an effective method for implementing unobtrusive VG in VR and AR systems.

   @article{tosa2025chromagazer,
     author = {Tosa, Rinto and Hattori, Shingo and Hiroi, Yuichi and Itoh, Yuta and Hiraki, Takefumi},
     journal = {IEEE Transactions on Visualization and Computer Graphics},
     title = {ChromaGazer: Unobtrusive Visual Modulation using Imperceptible Color Vibration for Visual Guidance},
     year = {2025},
     volume = {to appear},
     keywords = {Visual Guidance;Imperceptible Color Vibration;Color Perception;Augmented Reality},
   }

2. IEEE Access

   

   MagicItem: Dynamic Behavior Design of Virtual Objects With Large Language Models in a Commercial Metaverse Platform

   Ryutaro Kurai, Takefumi Hiraki, Yuichi Hiroi, Yutaro Hirao, Monica Perusquia-Hernandez, Hideki Uchiyama, and Kiyoshi Kiyokawa

   IEEE Access, 2025

   Abs DOI arXiv Bib

   To create rich experiences in virtual reality (VR) environments, it is essential to define the behavior of virtual objects through programming. However, programming in 3D spaces requires a wide range of background knowledge and programming skills. Although Large Language Models (LLMs) have provided programming support, they are still primarily aimed at programmers. In metaverse platforms, where many users inhabit VR spaces, most users are unfamiliar with programming, making it difficult for them to modify the behavior of objects in the VR environment easily. Existing LLM-based script generation methods for VR spaces require multiple lengthy iterations to implement the desired behaviors and are difficult to integrate into the operation of metaverse platforms. To address this issue, we propose a tool that generates behaviors for objects in VR spaces from natural language within Cluster, a metaverse platform with a large user base. By integrating LLMs with the Cluster Script provided by this platform, we enable users with limited programming experience to define object behaviors within the platform freely. We have also integrated our tool into a commercial metaverse platform and are conducting online experiments with 63 general users of the platform. The experiments show that even users with no programming background can successfully generate behaviors for objects in VR spaces, resulting in a highly satisfying system. Our research contributes to democratizing VR content creation by enabling non-programmers to design dynamic behaviors for virtual objects in metaverse platforms.

   @article{kurai2025magicitem,
     author = {Kurai, Ryutaro and Hiraki, Takefumi and Hiroi, Yuichi and Hirao, Yutaro and Perusquia-Hernandez, Monica and Uchiyama, Hideki and Kiyokawa, Kiyoshi},
     journal = {IEEE Access},
     title = {MagicItem: Dynamic Behavior Design of Virtual Objects With Large Language Models in a Commercial Metaverse Platform},
     year = {2025},
     volume = {13},
     number = {},
     pages = {19132-19143},
     doi = {10.1109/ACCESS.2025.3530439},
     keywords = {Metaverse;Codes;Three-dimensional displays;Large language models;Natural languages;Aerospace electronics;Web servers;Usability;Programming profession;Hands;Large-language model;low-code programming;metaverse platform;virtual reality},
   }

3. IEEE VR

   

   Slim Diffractive Waveguide Glasses for Beaming Displays with Enhanced Head Orientation Tolerance

   Yuta Itoh, Tomoya Nakamura, Yuichi Hiroi, and Kaan Aksit

   In 2025 IEEE Conference on Virtual Reality and 3D User Interfaces (IEEE VR), Mar 2025

   Abs Bib PDF

   Augmented Reality (AR) glasses must be slim, lightweight, and energy-efficient to achieve widespread adoption. Beaming Displays present a promising solution by offloading active components, such as the power-supplied light engine, into the surrounding environment while leaving only passive elements, like the eyepiece, in the wearable device. However, existing approaches still struggle to achieve both a slim design and a wide tolerance for projection angles relative to the user’s head orientation. In this work, we introduce a design for light-receiving glasses using a diffractive waveguide with in-coupling and out-coupling gratings. Our approach expands the allowable range of incident angles while maintaining a compact, lightweight form factor. We developed a proof-of-concept prototype and demonstrated an incident angle tolerance of approximately 20-30 degrees range, overcoming the previous design of 5 degrees.

   @inproceedings{itoh2025hoebeaming,
     author = {Itoh, Yuta and Nakamura, Tomoya and Hiroi, Yuichi and Aksit, Kaan},
     title = {Slim Diffractive Waveguide Glasses for Beaming Displays with Enhanced Head Orientation Tolerance},
     booktitle = {2025 IEEE Conference on Virtual Reality and 3D User Interfaces (IEEE VR)},
     year = {2025},
     volume = {},
     number = {},
     pages = {to appear},
     keywords = {beaming display; augmented reality; waveguide; near-eye displays; DOEs},
     month = mar,
   }

4. IEEE VR Poster

   

   An implementation of MagicCraft: Generating Interactive 3D Objects and Their Behaviors from Text for Commercial Metaverse Platforms

   Ryutaro Kurai, Takefumi Hiraki, Yuichi Hiroi, Yutaro Hirao, Monica Perusquia-Hernandez, Hideki Uchiyama, and Kiyoshi Kiyokawa

   In 2025 IEEE Conference on Virtual Reality and 3D User Interfaces Adjunct 2025, Mar 2025

   Abs Bib PDF

   Metaverse platforms are rapidly evolving to provide immersive spaces. However, the generation of dynamic and interactive 3D objects remains a challenge due to the need for advanced 3D modeling and programming skills. We present MagicCraft, a system that generates functional 3D objects from natural language prompts. MagicCraft uses generative AI models to manage the entire content creation pipeline: converting user text descriptions into images, transforming images into 3D models, predicting object behavior, and assigning necessary attributes and scripts. It also provides an interactive interface for users to refine generated objects by adjusting features like orientation, scale, seating positions, and grip points.

   @inproceedings{kurai2025magiccraft-poster,
     author = {Kurai, Ryutaro and Hiraki, Takefumi and Hiroi, Yuichi and Hirao, Yutaro and Perusquia-Hernandez, Monica and Uchiyama, Hideki and Kiyokawa, Kiyoshi},
     title = {An implementation of MagicCraft: Generating Interactive 3D Objects and Their Behaviors from Text for Commercial Metaverse Platforms},
     booktitle = {2025 IEEE Conference on Virtual Reality and 3D User Interfaces Adjunct 2025},
     year = {2025},
     volume = {},
     number = {},
     pages = {to appear},
     keywords = {Metaverse;3D Object Generation;Generative AI;AI-Assisted Design;Spatio-temporal Light pattern},
     month = mar,
   }

5. IEEE VR Workshop

   

   BeamStellar: Low-Latency, 6-DoF Head Tracking for Beaming Displays with Spatio-Temporal LED Encoding

   Jonas Weigand, Yuichi Hiroi, and Yuta Itoh

   In 2025 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), Mar 2025

   Abs Bib PDF

   Conventional augmented reality (AR) head-mounted displays (HMDs) require the integration of all components, including optics and processors, which presents significant barriers to practical implementation due to weight and heat generation concerns. While Beaming Displays address these challenges by offloading computing and projection functions to the environment, existing solutions leaving 6-degrees of freedom (6DoF) image projection with low motion-to-photon (M2P) latency. In this position paper, we introduce our trial BeamStellar, a Beaming Display system that integrates 6 DoF head tracking by combining spatio-temporally encoded LED patterns with a position-sensing detector. Through digital signal processing using an FPGA, the system aims a theoretical latency of less than 200 microseconds. This implementation alleviates the hardware burden on users, facilitating seamless AR experiences that seamlessly blend virtual content with the physical world.

   @inproceedings{weigand2025stellar,
     author = {Weigand, Jonas and Hiroi, Yuichi and Itoh, Yuta},
     title = {BeamStellar: Low-Latency, 6-DoF Head Tracking for Beaming Displays with Spatio-Temporal LED Encoding},
     booktitle = {2025 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)},
     year = {2025},
     volume = {},
     number = {},
     pages = {to appear},
     keywords = {Beaming Displays;Optical See-through Display;Low latency head tracking;Spatio-temporal Light pattern},
     month = mar,
   }

6. IEEE VR Workshop

   

   Development of Digital Twin Environment through Integration of Commercial Metaverse Platform and IoT Sensors of Smart Building

   Yusuke Masubuchi, Takefumi Hiraki, Yuichi Hiroi, Masanori Ibara, Kazuki Matsutani, Megumi Zaizen, and Junya Morita

   In 2025 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), Mar 2025

   Abs Bib

   The digital transformation of smart cities and workplaces requires effective integration of physical and cyber spaces, yet existing digital twin solutions remain limited in supporting real-time, multi-user collaboration. While metaverse platforms enable shared virtual experiences, they have not supported comprehensive integration of IoT sensors on physical spaces, especially for large-scale smart architectural environments. This paper presents a digital twin environment that integrates Kajima Corp.’s smart building facility "The GEAR" in Singapore with a commercial metaverse platform Cluster. Our system consists of three key components: a standardized IoT sensor platform, a real-time data relay system, and an environmental data visualization framework. Quantitative end-to-end latency measurements confirm the feasibility of our approach for real-world applications in large architectural spaces. The proposed framework enables new forms of collaboration that transcend spatial constraints, advancing the development of next-generation interactive environments.

   @inproceedings{masubuchi2025digitaltwin,
     author = {Masubuchi, Yusuke and Hiraki, Takefumi and Hiroi, Yuichi and Ibara, Masanori and Matsutani, Kazuki and Zaizen, Megumi and Morita, Junya},
     title = {Development of Digital Twin Environment through Integration of Commercial Metaverse Platform and IoT Sensors of Smart Building},
     booktitle = {2025 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)},
     year = {2025},
     volume = {},
     number = {},
     pages = {to appear},
     keywords = {Digital Twin;Metaverse Integration;Smart Building;IoT Sensors},
     month = mar,
   }

2024

1. IEEE TVCG

   

   HaptoFloater: Visuo-Haptic Augmented Reality by Embedding Imperceptible Color Vibration Signals for Tactile Display Control in a Mid-Air Image

   Rina Nagano, Takahiro Kinoshita, Shingo Hattori, Yuichi Hiroi, Yuta Itoh, and Takefumi Hiraki

   IEEE Transactions on Visualization and Computer Graphics, Nov 2024

   Abs DOI Bib PDF Video

   We propose HaptoFloater, a low-latency mid-air visuo-haptic augmented reality (VHAR) system that utilizes imperceptible color vibrations. When adding tactile stimuli to the visual information of a mid-air image, the user should not perceive the latency between the tactile and visual information. However, conventional tactile presentation methods for mid-air images, based on camera-detected fingertip positioning, introduce latency due to image processing and communication. To mitigate this latency, we use a color vibration technique; humans cannot perceive the vibration when the display alternates between two different color stimuli at a frequency of 25 Hz or higher. In our system, we embed this imperceptible color vibration into the mid-air image formed by a micromirror array plate, and a photodiode on the fingertip device directly detects this color vibration to provide tactile stimulation. Thus, our system allows for the tactile perception of multiple patterns on a mid-air image in 59.5 ms. In addition, we evaluate the visual-haptic delay tolerance on a mid-air display using our VHAR system and a tactile actuator with a single pattern and faster response time. The results of our user study indicate a visual-haptic delay tolerance of 110.6 ms, which is considerably larger than the latency associated with systems using multiple tactile patterns.

   @article{nagano2024haptofloater,
     author = {Nagano, Rina and Kinoshita, Takahiro and Hattori, Shingo and Hiroi, Yuichi and Itoh, Yuta and Hiraki, Takefumi},
     journal = {IEEE Transactions on Visualization and Computer Graphics},
     title = {HaptoFloater: Visuo-Haptic Augmented Reality by Embedding Imperceptible Color Vibration Signals for Tactile Display Control in a Mid-Air Image},
     year = {2024},
     volume = {30},
     number = {11},
     pages = {7463-7472},
     keywords = {Haptic interfaces;Vibrations;Visualization;Image color analysis;Actuators;Delays;Tactile sensors;Visuo-haptic displays;mid-air images;LCD displays;imperceptible color vibration},
     doi = {10.1109/TVCG.2024.3456175},
     issn = {1941-0506},
     month = nov,
   }

2. IEEE TVCG

   

   StainedSweeper: Compact, Variable-Intensity Light-Attenuation Display with Sweeping Tunable Retarders

   Yuichi Hiroi, Takefumi Hiraki, and Yuta Itoh

   IEEE Transactions on Visualization and Computer Graphics, May 2024

   Abs DOI Bib PDF Supp Video

   Light Attenuation Displays (LADs) are a type of Optical See-Through Head-Mounted Display (OST-HMD) that present images by attenuating incoming light with a pixel-wise polarizing color filter. Although LADs can display images in bright environments, there is a trade-off between the number of Spatial Light Modulators (SLMs) and the color gamut and contrast that can be expressed, making it difficult to achieve both high-fidelity image display and a small form factor. To address this problem, we propose StainedSweeper, a LAD that achieves both the wide color gamut and the variable intensity with a single SLM. Our system synchronously controls a pixel-wise Digital Micromirror Device (DMD) and a nonpixel polarizing color filter to pass light when each pixel is the desired color. By sweeping this control at high speed, the human eye perceives images in a time-multiplexed, integrated manner. To achieve this, we develop the OST-HMD design using a reflective Solc filter as a polarized color filter and a color reproduction algorithm based on the optimization of the time-multiplexing matrix for the selected primary color filters. Our proof-of-concept prototype showed that our single SLM design can produce subtractive images with variable contrast and a wider color gamut than conventional LADs.

   @article{hiroi2024stainedsweeper,
     author = {Hiroi, Yuichi and Hiraki, Takefumi and Itoh, Yuta},
     journal = {IEEE Transactions on Visualization and Computer Graphics},
     title = {StainedSweeper: Compact, Variable-Intensity Light-Attenuation Display with Sweeping Tunable Retarders},
     year = {2024},
     volume = {30},
     number = {5},
     pages = {2682-2692},
     keywords = {Image color analysis;Optical filters;Filtering algorithms;Light sources;Optical imaging;Optical reflection;Optical polarization;Light attenuation display;see-through display;augmented reality;time-multiplexing;polarized color filter},
     doi = {10.1109/TVCG.2024.3372058},
     issn = {1941-0506},
     month = may,
   }

3. IEEE TVCG

   

   Towards Co-Operative Beaming Displays: Dual Steering Projectors for Extended Projection Volume and Head Orientation Range

   Hiroto Aoki, Takumi Tochimoto, Yuichi Hiroi, and Yuta Itoh

   IEEE Transactions on Visualization and Computer Graphics, May 2024

   Honorable Mention Abs DOI Bib PDF

   Existing near-eye displays (NEDs) have trade-offs related to size, weight, computational resources, battery life, and body temperature. A recent paradigm, beaming display, addresses these trade-offs by separating the NED into a steering projector (SP) for image presentation and a passive headset worn by the user. However, the beaming display has issues with the projection area of a single SP and has severe limitations on the head orientation and pose that the user can move. In this study, we distribute dual steering projectors in the scene to extend the head orientation and pose of the beaming display by coordinating the dual projections on a passive headset. For cooperative control of each SP, we define a geometric model of the SPs and propose a calibration and projection control method designed for multiple projectors. We present implementations of the system along with evaluations showing that the precision and delay are 1.8 ∼ 5.7 mm and 14.46 ms, respectively, at a distance of about 1 m from the SPs. From this result, our prototype with multiple SPs can project images in the projection area (20 \textmm \times 30 \textmm) of the passive headset while extending the projectable head orientation. Furthermore, as applications of cooperative control by multiple SPs, we show the possibility of multiple users, improving dynamic range and binocular presentation.

   @article{aoki2024dualbeaming,
     author = {Aoki, Hiroto and Tochimoto, Takumi and Hiroi, Yuichi and Itoh, Yuta},
     journal = {IEEE Transactions on Visualization and Computer Graphics},
     title = {Towards Co-Operative Beaming Displays: Dual Steering Projectors for Extended Projection Volume and Head Orientation Range},
     year = {2024},
     volume = {30},
     number = {5},
     pages = {2309-2318},
     keywords = {Mirrors;Cameras;Headphones;Calibration;Optical imaging;High-speed optical techniques;Head;Near-eye display;Augmented reality;Projectors},
     doi = {10.1109/TVCG.2024.3372118},
     issn = {1941-0506},
     month = may,
   }

4. IEEE ISMAR

   

   FactoredSweeper: Optical See-Through Display Integrating Light Attenuation and Addition with a Single Spatial Light Modulator

   Yuichi Hiroi, Takefumi Hiraki, and Yuta Itoh

   In 2024 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), Oct 2024

   Abs PDF Supp Video

   Light Attenuation Displays (LADs), a subset of Optical See-Through Head-Mounted Displays (OST-HMDs), enable image display in bright environments by filtering incident light at the pixel level. Although recent methods have proposed single-DMD light attenuation, they do not consider additive color display and background compensation, limiting their applicability in real-world scenarios. We present FactoredSweeper, a single digital micromirror device (DMD) system that incorporates both light attenuation and addition. By synchronizing the DMD, color filter, and light source, our system generates an additive virtual image, light attenuation, and occlusion through time multiplexing. To produce the target image while compensating for the background, we optimize time-multiplexed binary DMD patterns and LED/color filter schedules using perceptually-driven non-negative matrix factorization. Simulations and prototypes demonstrate that our integrated attenuation-addition single-SLM system achieves superior dynamic range and perceptual image quality compared to conventional occlusion-capable OST-HMDs using grayscale occlusion masks.

5. ISMAR Demo

   

   MetaGadget: IoT Framework for Event-Triggered Integration of User-Developed Devices into Commercial Metaverse Platforms

   Ryutaro Kurai, Yuichi Hiroi, and Takefumi Hiraki

   In 2024 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) , Nov 2024

   Abs Video Code

   This demonstration introduces MetaGadget, an IoT framework designed to integrate user-developed devices into commercial metaverse platforms. Synchronizing virtual reality (VR) environments with physical devices has traditionally required a constant connection to VR clients, limiting flexibility and resource efficiency. MetaGadget overcomes these limitations by configuring user-developed devices as IoT units with server capabilities, supporting communication via HTTP protocols within the commercial metaverse platform, Cluster. This approach enables event-triggered device control without the need for persistent connections from metaverse clients. Through the demonstration, users will experience event-triggered interaction between VR and physical devices, as well as real-world device control through the VR space by multiple people. Our framework is expected to reduce technical barriers to integrating VR spaces and custom devices, contribute to interoperability, and increase resource efficiency through event-triggered connections.

6. ISMAR Poster

   

   Beaming Display Using Thin Holographic Waveguides for Wider Head Orientation Angle Range

   Yuta Itoh, Tomoya Nakamura, Yuichi Hiroi, and Kaan Aksit

   In 2024 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), Oct 2024

   Abs DOI Bib PDF

   Augmented Reality (AR) glasses face fundamental challenges related to technical trade-offs. Emerging Beaming Displays (BDs) offer a compelling solution by separating the active and passive components. However, existing BD-based AR glasses have yet to achieve a thin and lightweight design with wide incident projection angles. This work proposes an eyepiece for BDs, including a holographic waveguide with tailored in- and out-coupling gratings. The proposed design aims to achieve a millimeter-thin form factor with a wide tolerance for incident angles, thus overcoming the limitations of existing designs. We have constructed proof-of-concept passive AR glasses prototypes, all approximately 2mm thick, including one in the form of conventional eyeglasses, and demonstrated an acceptable lateral angle of incidence of up to 90 degrees.

   @inproceedings{itoh2024thinholographic,
     author = {Itoh, Yuta and Nakamura, Tomoya and Hiroi, Yuichi and Aksit, Kaan},
     title = {Beaming Display Using Thin Holographic Waveguides for Wider Head Orientation Angle Range},
     booktitle = {2024 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct)},
     year = {2024},
     volume = {},
     number = {},
     pages = {475-476},
     keywords = {Beaming display; Augmented reality; Near-eye display; waveguide; HOEs},
     doi = {10.1109/ISMAR-Adjunct64951.2024.00138},
     month = oct,
   }

7. arXiv

   

   PanoTree: Autonomous Photo-Spot Explorer in Virtual Reality Scenes

   Tomohiro Hayase, Sacha Braun, Hikari Yanagawa, Itsuki Orito, and Yuichi Hiroi

   Oct 2024

   arXiv Bib Video Code Website

   @misc{hayase2024panotree,
     title = {PanoTree: Autonomous Photo-Spot Explorer in Virtual Reality Scenes},
     author = {Hayase, Tomohiro and Braun, Sacha and Yanagawa, Hikari and Orito, Itsuki and Hiroi, Yuichi},
     year = {2024},
     eprint = {2405.17136},
     archiveprefix = {arXiv},
     primaryclass = {cs.CV},
   }

8. SIGGRAPH Poster

   

   Measurement of the Imperceptible Threshold for Color Vibration Pairs Selected by using MacAdam Ellipse

   Shingo Hattori, Yuichi Hiroi, and Takefumi Hiraki

   In ACM SIGGRAPH 2024 Posters, Denver, CO, USA, Oct 2024

   Awarded Abs DOI arXiv Bib

   1st Place in SIGGRAPH 2024 Student Research Competition for Undergraduate Work

   We propose an efficient method for searching for color vibration pairs that are imperceptible to the human eye based on the MacAdam ellipse, an experimentally determined color-difference range that is indistinguishable to the human eye. We created color pairs by selecting eight colors within the sRGB color space specified by the ellipse, and conducted experiments to confirm the threshold of the amplitude of color vibration amplitude at which flicker becomes imperceptible to the human eye. The experimental results indicate a general guideline for acceptable amplitudes for pair selection.

   @inproceedings{hattori2024macadam,
     author = {Hattori, Shingo and Hiroi, Yuichi and Hiraki, Takefumi},
     title = {Measurement of the Imperceptible Threshold for Color Vibration Pairs Selected by using MacAdam Ellipse},
     year = {2024},
     isbn = {9798400705168},
     publisher = {Association for Computing Machinery},
     address = {New York, NY, USA},
     url = {https://doi.org/10.1145/3641234.3671041},
     doi = {10.1145/3641234.3671041},
     booktitle = {ACM SIGGRAPH 2024 Posters},
     articleno = {68},
     numpages = {2},
     keywords = {MacAdam ellipse, color perception, imperceptible color vibration},
     location = {Denver, CO, USA},
     series = {SIGGRAPH '24},
   }

9. IEEE VR Workshop

   

   Design and Implementation of Agent APIs for Large-Scale Social VR Platforms

   Ryutaro Kurai, Takefumi Hiraki, Yuichi Hiroi, Yutaro Hirao, Monica Perusquia-Hernandez, Hideaki Uchiyama, and Kiyoshi Kiyokawa

   In 2024 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW) , Mar 2024

   Abs DOI Bib

   Implementing an autonomous agent on a social VR platform where many users share space requires diverse information. In particular, it is required to recognize the distance from other users, their orientation toward each other, the avatar’s pose, and text and voice messages, and to behave accordingly. This paper proposes an API to obtain the above information on “Cluster,” a multi-device social VR platform in operation, and an agent that uses the API. We have implemented this API using a network proxy. The agent using this API can connect to ChatGPT [7] and have a conversation in real time. We measured the latency required for the conversation and confirmed that the response time was about 1 second.

   @inproceedings{kurai2024agentapi,
     author = {Kurai, Ryutaro and Hiraki, Takefumi and Hiroi, Yuichi and Hirao, Yutaro and Perusquia-Hernandez, Monica and Uchiyama, Hideaki and Kiyokawa, Kiyoshi},
     booktitle = { 2024 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW) },
     title = {{ Design and Implementation of Agent APIs for Large-Scale Social VR Platforms }},
     year = {2024},
     volume = {},
     issn = {},
     pages = {584-587},
     keywords = {Three-dimensional displays;Text recognition;Oral communication;Virtual reality;Speech recognition;User interfaces;Software},
     doi = {10.1109/VRW62533.2024.00112},
     url = {https://doi.ieeecomputersociety.org/10.1109/VRW62533.2024.00112},
     publisher = {IEEE Computer Society},
     address = {Los Alamitos, CA, USA},
     month = mar,
   }

2023

1. IEEE TVCG

   

   Low-Latency Beaming Display: Implementation of Wearable, 133 μs Motion-to-Photon Latency Near-Eye Display

   Yuichi Hiroi , Akira Watanabe, Yuri Mikawa, and Yuta Itoh

   IEEE Transactions on Visualization and Computer Graphics, Nov 2023

   Abs DOI Bib PDF Video Slides

   This paper presents a low-latency Beaming Display system with a 133 μ\mathrms motion-to-photon (M2P) latency, the delay from head motion to the corresponding image motion. The Beaming Display represents a recent near-eye display paradigm that involves a steerable remote projector and a passive wearable headset. This system aims to overcome typical trade-offs of Optical See-Through Head-Mounted Displays (OST-HMDs), such as weight and computational resources. However, since the Beaming Display projects a small image onto a moving, distant viewpoint, M2P latency significantly affects displacement. To reduce M2P latency, we propose a low-latency Beaming Display system that can be modularized without relying on expensive high-speed devices. In our system, a 2D position sensor, which is placed coaxially on the projector, detects the light from the IR-LED on the headset and generates a differential signal for tracking. An analog closed-loop control of the steering mirror based on this signal continuously projects images onto the headset. We have implemented a proof-of-concept prototype, evaluated the latency and the augmented reality experience through a user-perspective camera, and discussed the limitations and potential improvements of the prototype.

   @article{hiroi2023lowlatency,
     author = {Hiroi, Yuichi and Watanabe, Akira and Mikawa, Yuri and Itoh, Yuta},
     journal = {IEEE Transactions on Visualization and Computer Graphics},
     title = {Low-Latency Beaming Display: Implementation of Wearable, 133 μs Motion-to-Photon Latency Near-Eye Display},
     year = {2023},
     volume = {29},
     number = {11},
     pages = {4761-4771},
     keywords = {Mirrors;Low latency communication;Headphones;Head;Cameras;Tracking;Two dimensional displays;Low-Latency Display;Beaming Display;Motion-to-Photon Latency;Lateral-effect Photodiodes},
     doi = {10.1109/TVCG.2023.3320212},
     issn = {1941-0506},
     month = nov,
   }

2. ACM UIST

   

   Telextiles: End-to-end Remote Transmission of Fabric Tactile Sensation

   Takekazu Kitagishi, Yuichi Hiroi, Yuna Watanabe, Yuta Itoh, and Jun Rekimoto

   In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, San Francisco, CA, USA, Nov 2023

   Abs DOI Bib Video Website

   The tactile sensation of textiles is critical in determining the comfort of clothing. For remote use, such as online shopping, users cannot physically touch the textile of clothes, making it difficult to evaluate its tactile sensation. Tactile sensing and actuation devices are required to transmit the tactile sensation of textiles. The sensing device needs to recognize different garments, even with hand-held sensors. In addition, the existing actuation device can only present a limited number of known patterns and cannot transmit unknown tactile sensations of textiles. To address these issues, we propose Telextiles, an interface that can remotely transmit tactile sensations of textiles by creating a latent space that reflects the proximity of textiles through contrastive self-supervised learning. We confirm that textiles with similar tactile features are located close to each other in the latent space through a two-dimensional plot. We then compress the latent features for known textile samples into the 1D distance and apply the 16 textile samples to the rollers in the order of the distance. The roller is rotated to select the textile with the closest feature if an unknown textile is detected.

   @inproceedings{kitagishi2023telextiles,
     author = {Kitagishi, Takekazu and Hiroi, Yuichi and Watanabe, Yuna and Itoh, Yuta and Rekimoto, Jun},
     title = {Telextiles: End-to-end Remote Transmission of Fabric Tactile Sensation},
     year = {2023},
     isbn = {9798400701320},
     publisher = {Association for Computing Machinery},
     address = {New York, NY, USA},
     url = {https://doi.org/10.1145/3586183.3606764},
     doi = {10.1145/3586183.3606764},
     booktitle = {Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology},
     articleno = {67},
     numpages = {10},
     keywords = {Haptic feedback, Machine learning, Passive haptic feedback, Self supervised learning, Tactile Display, Tactile perception, Texture, Texture perception, Texture recognition},
     location = {San Francisco, CA, USA},
     series = {UIST '23},
   }

3. ACM VRST

   

   Retinal Homing Display: Head-Tracking Auto-stereoscopic Retinal Projection Display

   Hiroto Aoki, Yuichi Hiroi, Yuta Itoh, and Jun Rekimoto

   In Proceedings of the 29th ACM Symposium on Virtual Reality Software and Technology, Christchurch, New Zealand, Nov 2023

   Abs DOI Bib PDF

   This paper introduces Retinal Homing Display, which presents focus-free stereoscopic images via retinal projection, thus eliminating the need for the user to wear additional equipment. Traditional 3D displays, typically classified as either naked-eye stereoscopic or wearable, present inherent challenges: the former involves a compromise between resolution and accurate depth perception, while the latter imposes an additional burden on the user. Our proposed display employs optical and mechanical mechanisms to converge projector light at the user’s pupil center, simultaneously tracking eye movements. This lets the user perceive focus-free, high-resolution stereoscopic images without wearable equipment. We implemented a proof-of-concept system utilizing a robotic arm and a Dihedral Corner Reflector Array (DCRA), subsequently evaluating image quality and its eyebox. Finally, we discuss the limitations of the current prototype and outline potential directions for future research.

   @inproceedings{aoki2023retinal,
     author = {Aoki, Hiroto and Hiroi, Yuichi and Itoh, Yuta and Rekimoto, Jun},
     title = {Retinal Homing Display: Head-Tracking Auto-stereoscopic Retinal Projection Display},
     year = {2023},
     isbn = {9798400703287},
     publisher = {Association for Computing Machinery},
     address = {New York, NY, USA},
     url = {https://doi.org/10.1145/3611659.3615715},
     doi = {10.1145/3611659.3615715},
     booktitle = {Proceedings of the 29th ACM Symposium on Virtual Reality Software and Technology},
     articleno = {13},
     numpages = {10},
     keywords = {autostereoscopic display, motion-following display, retinal projection},
     location = {Christchurch, New Zealand},
     series = {VRST '23},
   }

4. IEEE VR

   

   A Compact Photochromic Occlusion Capable See-through Display with Holographic Lenses

   Chun-Wei Ooi, Yuichi Hiroi, and Yuta Itoh

   In 2023 IEEE Conference Virtual Reality and 3D User Interfaces (VR), Mar 2023

   Abs DOI Bib PDF

   Occlusion is a crucial visual element in optical see-through (OST) augmented reality, however, implementing occlusion in OST displays while addressing various design trade-offs is a difficult problem. In contrast to the traditional method of using spatial light modulators (SLMs) for the occlusion mask, using photochromic materials as occlusion masks can effectively eliminate diffraction artifacts in see-through views due to the lack of electronic pixels, thus providing superior see-through image quality. However, this design requires UV illumination to activate the photochromic mate-rial, which traditionally requires multiple SLMs, resulting in a larger form factor for the system. This paper presents a compact photochromic occlusion-capable OST design using multilayer, wavelength-dependent holographic optical lenses (HOLs). Our approach employs a single digital mi-cromirror display (DMD) to form both the occlusion mask with UV light and a virtual image with visible light in a time-multiplexed man-ner. We demonstrate our proof-of-concept system on a bench-top setup and assess the appearance and contrasts of the displayed image. We also suggest potential improvements for current prototypes to encourage the community to explore this occlusion approach.

   @inproceedings{ooi2023photochromic,
     author = {Ooi, Chun-Wei and Hiroi, Yuichi and Itoh, Yuta},
     booktitle = {2023 IEEE Conference Virtual Reality and 3D User Interfaces (VR)},
     title = {A Compact Photochromic Occlusion Capable See-through Display with Holographic Lenses},
     year = {2023},
     volume = {},
     number = {},
     pages = {237-242},
     keywords = {Visualization;Optical diffraction;Three-dimensional displays;Optical design;Prototypes;Holography;User interfaces;Human-centered computing-Human computer interaction (HCI)-Interaction paradigms-Mixed / augmented reality;Human-centered computing-Communication hardware;interfaces and storage-Displays and imagers},
     doi = {10.1109/VR55154.2023.00039},
     issn = {2642-5254},
     month = mar,
   }

5. ISMAR Poster

   Dual Beaming Display for Extended Head Orientation and Projection Volume

   Takumi Tochimoto, Yuichi Hiroi, and Yuta Itoh

   In 2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), Oct 2023

   Abs DOI Bib

   Existing near-eye displays (NEDs) come with various tradeoffs, including size, weight, and battery life. The new "beaming display" model attempts to address these by splitting the NED into a steering projector (SP) and a passive headset. However, it has limitations in terms of projection area and user head movement. In this research, we utilize two steering projectors to expand the beaming display’s capabilities. We introduce a geometric model and a control method specifically for multiple projectors. Tests indicate an accuracy of 3-19mm and a delay of 14.46 ms from a 1m distance to the SPs, allowing projection onto a 20mm x 30mm area of the headset. This method not only expands head movement possibilities but also shows potential for multiple users and improved dynamic range and binocular presentation.

   @inproceedings{tochimoto2023dualbeaming,
     author = {Tochimoto, Takumi and Hiroi, Yuichi and Itoh, Yuta},
     booktitle = {2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct)},
     title = {Dual Beaming Display for Extended Head Orientation and Projection Volume},
     year = {2023},
     volume = {},
     number = {},
     pages = {377-378},
     keywords = {Headphones;Design methodology;Geometric modeling;Display systems;Focusing;Dynamic range;Delays;Human-centered computing;Visualization;Visualization techniques;Treemaps; Human-centered computing;Visualization design and evaluation methods},
     doi = {10.1109/ISMAR-Adjunct60411.2023.00081},
     issn = {2771-1110},
     month = oct,
   }

6. AHs Poster

   

   Brain-Computer Interface using Directional Auditory Perception

   Yuto Koike, Yuichi Hiroi, Yuta Itoh, and Jun Rekimoto

   In Proceedings of the Augmented Humans International Conference 2023, Glasgow, United Kingdom, Oct 2023

   Awarded Abs DOI Bib

   Best Poster Honorable Mention

   We investigate the potential of brain-computer interface (BCI) using electroencephalogram (EEG) induced by listening (or recalling) auditory stimuli of different directions. In the initial attempt, we apply a time series classification model based on deep learning to the EEG to demonstrate whether each EEG can be classified by recognizing binary (left or right) auditory directions. The results showed high classification accuracy when trained and tested on the same users. Discussion is provided to further explore this topic.

   @inproceedings{koike2023braincomputer,
     author = {Koike, Yuto and Hiroi, Yuichi and Itoh, Yuta and Rekimoto, Jun},
     title = {Brain-Computer Interface using Directional Auditory Perception},
     year = {2023},
     isbn = {9781450399845},
     publisher = {Association for Computing Machinery},
     address = {New York, NY, USA},
     url = {https://doi.org/10.1145/3582700.3583713},
     doi = {10.1145/3582700.3583713},
     booktitle = {Proceedings of the Augmented Humans International Conference 2023},
     pages = {342–345},
     numpages = {4},
     keywords = {BCI, BMI, Brain-computer Interface, Brain-machine Interface, Directional Auditory Sensation, EEG, Electroencephalogram, Electroencephalography},
     location = {Glasgow, United Kingdom},
     series = {AHs '23},
   }

2022

1. Opt. Exp.

   

   Neural distortion fields for spatial calibration of wide field-of-view near-eye displays

   Yuichi Hiroi, Kiyosato Someya, and Yuta Itoh

   Opt. Express, Oct 2022

   Awarded Abs DOI Bib Video

   Japan’s Leading Optics Research in 2022 (Top 30), Optical Society of Japan

   We propose a spatial calibration method for wide field-of-view (FoV) near-eye displays (NEDs) with complex image distortions. Image distortions in NEDs can destroy the reality of the virtual object and cause sickness. To achieve distortion-free images in NEDs, it is necessary to establish a pixel-by-pixel correspondence between the viewpoint and the displayed image. Designing compact and wide-FoV NEDs requires complex optical designs. In such designs, the displayed images are subject to gaze-contingent, non-linear geometric distortions, which explicit geometric models can be difficult to represent or computationally intensive to optimize. To solve these problems, we propose neural distortion field (NDF), a fully-connected deep neural network that implicitly represents display surfaces complexly distorted in spaces. NDF takes spatial position and gaze direction as input and outputs the display pixel coordinate and its intensity as perceived in the input gaze direction. We synthesize the distortion map from a novel viewpoint by querying points on the ray from the viewpoint and computing a weighted sum to project output display coordinates into an image. Experiments showed that NDF calibrates an augmented reality NED with 90° FoV with about 3.23 pixel (5.8 arcmin) median error using only 8 training viewpoints. Additionally, we confirmed that NDF calibrates more accurately than the non-linear polynomial fitting, especially around the center of the FoV.

   @article{hiroi2022neural,
     author = {Hiroi, Yuichi and Someya, Kiyosato and Itoh, Yuta},
     journal = {Opt. Express},
     keywords = {Image metrics; Lens design; Near eye displays; Optical aberration; Optical systems; Systems design},
     number = {22},
     pages = {40628--40644},
     publisher = {Optica Publishing Group},
     title = {Neural distortion fields for spatial calibration of wide field-of-view near-eye displays},
     volume = {30},
     month = oct,
     year = {2022},
     url = {https://opg.optica.org/oe/abstract.cfm?URI=oe-30-22-40628},
     doi = {10.1364/OE.472288},
   }

2. ACM VRST

   

   NeARportation: A Remote Real-time Neural Rendering Framework

   Yuichi Hiroi, Yuta Itoh, and Jun Rekimoto

   In Proceedings of the 28th ACM Symposium on Virtual Reality Software and Technology, Tsukuba, Japan, Oct 2022

   Abs DOI Bib PDF Video

   While presenting a photorealistic appearance plays a major role in immersion in Augmented Virtuality environment, displaying that of real objects remains a challenge. Recent developments in photogrammetry have facilitated the incorporation of real objects into virtual space. However, reproducing complex appearances, such as subsurface scattering and transparency, still requires a dedicated environment for measurement and possesses a trade-off between rendering quality and frame rate. Our NeARportation framework combines server–client bidirectional communication and neural rendering to resolve these trade-offs. Neural rendering on the server receives the client’s head posture and generates a novel-view image with realistic appearance reproduction that is streamed onto the client’s display. By applying our framework to a stereoscopic display, we confirm that it can display a high-fidelity appearance on full-HD stereo videos at 35-40 frames per second (fps) according to the user’s head motion.

   @inproceedings{hiroi2022nearportation,
     author = {Hiroi, Yuichi and Itoh, Yuta and Rekimoto, Jun},
     title = {NeARportation: A Remote Real-time Neural Rendering Framework},
     year = {2022},
     isbn = {9781450398893},
     publisher = {Association for Computing Machinery},
     address = {New York, NY, USA},
     url = {https://doi.org/10.1145/3562939.3565616},
     doi = {10.1145/3562939.3565616},
     booktitle = {Proceedings of the 28th ACM Symposium on Virtual Reality Software and Technology},
     articleno = {23},
     numpages = {5},
     keywords = {remote rendering, real-time rendering, neural rendering, augmented virtuality, appearance reproduction},
     location = {Tsukuba, Japan},
     series = {VRST '22},
   }

3. IEEE ISMAR

   

   Towards Spatial Airflow Interaction: Schlieren Imaging for Augmented Reality

   Zhang Zhibin, Yuichi Hiroi, and Yuta Itoh

   In 2022 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), Oct 2022

   Abs DOI Bib PDF Video

   This work integrates Schlieren Imaging, a unique sensing modality, into Augmented Reality (AR) to explore ways to utilize invisible airflows for AR. Schlieren imaging is an imaging technique that visualizes the flow of fluids, which is normally invisible to the eyes. Theoretically, the technique can calculate the motion, pressure, temperature, and density of the airflow in our physical world. This unique, but less applied modality may expand interaction paradigms in AR and VR. We build a proof-of-concept AR system combined with Schlieren imaging that allows real airflow to affect virtual objects. The results of quantitative analyses show that our system can integrate different types of airflow with pressure values ranging from weak breathing actions to a heat gun up to 10m/s or 0.25m3/min airflow. We also showcase AR use cases including blowing out a virtual candle and a heat gun.

   @inproceedings{zhibin2022airflow,
     author = {Zhibin, Zhang and Hiroi, Yuichi and Itoh, Yuta},
     booktitle = {2022 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)},
     title = {Towards Spatial Airflow Interaction: Schlieren Imaging for Augmented Reality},
     year = {2022},
     volume = {},
     number = {},
     pages = {215-223},
     keywords = {Heating systems;Temperature sensors;Photography;Visualization;Sensitivity;Statistical analysis;User interfaces;Schlieren imaging;augmented reality;spatial airflow interaction},
     doi = {10.1109/ISMAR55827.2022.00036},
     issn = {1554-7868},
     month = oct,
   }

2021

1. Opt. Exp.

   

   Focal surface occlusion

   Yuichi Hiroi, Takumi Kaminokado, Shunsuke Ono, and Yuta Itoh

   Opt. Express, Oct 2021

   Abs DOI Bib

   This paper proposes focal surface occlusion to provide focal cues of occlusion masks for multiple virtual objects at continuous depths in an occlusion-capable optical see-through head-mounted display. A phase-only spatial light modulator (PSLM) that acts as a dynamic free-form lens is used to conform the focal surface of an occlusion mask to the geometry of the virtual scene. To reproduce multiple and continuous focal blurs while reducing the distortion of the see-through view, an optical design based on afocal optics and edge-based optimization to exploit a property of the occlusion mask is established. The prototype with the PSLM and transmissive liquid crystal display can reproduce the focus blur of occluded objects at multiple and continuous depths with a field of view of 14.6°.

   @article{hiroi2021focalsurface,
     author = {Hiroi, Yuichi and Kaminokado, Takumi and Ono, Shunsuke and Itoh, Yuta},
     journal = {Opt. Express},
     keywords = {Head mounted displays; Microlens arrays; Near eye displays; Optical components; Optical systems; Spatial light modulators},
     number = {22},
     pages = {36581--36597},
     publisher = {Optica Publishing Group},
     title = {Focal surface occlusion},
     volume = {29},
     month = oct,
     year = {2021},
     url = {https://opg.optica.org/oe/abstract.cfm?URI=oe-29-22-36581},
     doi = {10.1364/OE.440024},
   }

2. Augmented Humans

   

   CircadianVisor: Image Presentation with an Optical See-Through Display in Consideration of Circadian Illuminance

   Takumi Tochimoto, Yuichi Hiroi, and Yuta Itoh

   In Proceedings of the Augmented Humans International Conference 2021, Rovaniemi, Finland, Oct 2021

   Abs DOI Bib PDF

   In modern society, the impact of nighttime artificial lighting on the human sleep/wake cycle (the circadian rhythm), has long been an important issue. In augmented reality, such health hazards should be prevented if we are to become a society that wears optical see-through head-mounted displays (OST-HMDs) on a daily basis. We present CircadianVisor, an OST display system that controls circadian performance. Our system combines an OST-HMD with a liquid crystal (LC) shutter and a spectrometer to control the circadian illuminance (CIL, biolux) of light incident on the user’s eyes. To prevent the CIL at the eyes from exceeding the threshold, correct for the displayed image based on RGB values, and adjust the transmittance of the LC visor to pass through the environment light based on spectral measurements, we build a proof-of-concept system to evaluate the feasibility of the system’s CIL control and test it with a spectrometer installed at the user’s viewpoint. The evaluation shows that the CIL at the user’s viewpoint can be kept below the threshold.

   @inproceedings{tochimoto2021circadian,
     author = {Tochimoto, Takumi and Hiroi, Yuichi and Itoh, Yuta},
     title = {CircadianVisor: Image Presentation with an Optical See-Through Display in Consideration of Circadian Illuminance},
     year = {2021},
     isbn = {9781450384285},
     publisher = {Association for Computing Machinery},
     address = {New York, NY, USA},
     url = {https://doi.org/10.1145/3458709.3458938},
     doi = {10.1145/3458709.3458938},
     booktitle = {Proceedings of the Augmented Humans International Conference 2021},
     pages = {66–76},
     numpages = {11},
     keywords = {Augmented reality, Blue-light-blocking glasses, Circadian rhythm, Head-mounted displays},
     location = {Rovaniemi, Finland},
     series = {AHs '21},
   }

3. ISMAR Poster

   

   Focus-Aware Retinal Projection-based Near-Eye Display

   Mayu Kaneko, Yuichi Hiroi, and Yuta Itoh

   In 2021 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), Oct 2021

   Abs DOI Bib PDF Video

   The primary challenge in optical see-through near-eye displays lies in providing correct optical focus cues. Established approaches such as varifocal or light field display essentially sacrifice temporal or spatial resolution of the resulting 3D images. This paper explores a new direction to address the trade-off by combining a retinal projection display (RPD) with ocular wavefront sensing (OWS). Our core idea is to display a depth of field-simulated image on an RPD to produce visually consistent optical focus cues while maintaining the spatial and temporal resolution of the image. To obtain the current accommodation of the eye, we integrate OWS. We demonstrate that our proof-of-concept system successfully renders virtual contents with proper depth cues while covering the eye accommodation range from 28.5 cm (3.5 D) to infinity (0.0 D).

   @inproceedings{kaneko2021focusaware,
     author = {Kaneko, Mayu and Hiroi, Yuichi and Itoh, Yuta},
     booktitle = {2021 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct)},
     title = {Focus-Aware Retinal Projection-based Near-Eye Display},
     year = {2021},
     volume = {},
     number = {},
     pages = {207-208},
     keywords = {Three-dimensional displays;Optical variables measurement;Optical imaging;Retina;Rendering (computer graphics);Adaptive optics;Light fields;Augmented reality;accommodation sensing;retinal projection;near-eye display},
     doi = {10.1109/ISMAR-Adjunct54149.2021.00049},
     issn = {},
     month = oct,
   }

2020

1. IEEE TVCG

   

   StainedView: Variable-Intensity Light-Attenuation Display with Cascaded Spatial Color Filtering for Improved Color Fidelity

   Takumi Kaminokado, Yuichi Hiroi, and Yuta Itoh

   IEEE Transactions on Visualization and Computer Graphics, Dec 2020

   Abs DOI Bib PDF

   We present StainedView, an optical see-through display that spatially filters the spectral distribution of light to form an image with improved color fidelity. Existing light-attenuation displays have limited color fidelity and contrast, resulting in a degraded appearance of virtual images. To use these displays to present virtual images that are more consistent with the real world, we require three things: intensity modulation of incoming light, spatial color filtering with narrower bandwidth, and appropriate light modulation for incoming light with an arbitrary spectral distribution. In StainedView, we address the three requirements by cascading two phase-only spatial light modulators (PSLMs), a digital micromirror device, and polarization optics to control both light intensity and spectrum distribution. We show that our design has a 1.8 times wider color gamut fidelity (75.8% fulfillment of sRGB color space) compared to the existing single-PSLM approach (41.4%) under a reference white light. We demonstrated the design with a proof-of-concept display system. We further introduce our optics design and pixel-selection algorithm for the given light input, evaluate the spatial color filter, and discuss the limitation of the current prototype.

   @article{kamino2020stainedview,
     author = {Kaminokado, Takumi and Hiroi, Yuichi and Itoh, Yuta},
     journal = {IEEE Transactions on Visualization and Computer Graphics},
     title = {StainedView: Variable-Intensity Light-Attenuation Display with Cascaded Spatial Color Filtering for Improved Color Fidelity},
     year = {2020},
     volume = {26},
     number = {12},
     pages = {3576-3586},
     keywords = {Image color analysis;Optical imaging;Optical polarization;Adaptive optics;Optical distortion;Optical modulation;Nonlinear optics;Light attenuation display;phase modulation;see-through display;vision augmentation;augmented reality},
     doi = {10.1109/TVCG.2020.3023569},
     issn = {1941-0506},
     month = dec,
   }

2. Augmented Humans

   

   DehazeGlasses: Optical Dehazing with an Occlusion Capable See-Through Display

   Yuichi Hiroi, Takumi Kaminokado, Atsushi Mori, and Yuta Itoh

   In Proceedings of the Augmented Humans International Conference, Kaiserslautern, Germany, Dec 2020

   Abs DOI Bib PDF

   We present DehazeGlasses, a see-through visual haze removal system that optically dehazes the user’s field of vision. Human vision suffers from a degraded view due to aspects of the scene environment, such as haze. Such degradation may interfere with our behavior or judgement in daily tasks. We focus on hazy scenes as one common degradation source, which whitens the view due to certain atmospheric conditions. Unlike typical computer vision systems that process recorded images, we aim to realize a see-through glasses system that can optically manipulate our field of view to dehaze the perceived scene. Our system selectively modulates the intensity of the light entering the eyes via occlusion-capable optical see-through head-mounted displays (OST-HMD). We built a proof-of-concept system to evaluate the feasibility of our haze removal method by combining a digital micromirror device (DMD) and an OST-HMD, and tested it with a user-perspective viewpoint camera. A quantitative evaluation with 80 scenes from a haze removal dataset shows that our system realizes a dehazed view that is significantly closer to the ground truth scene compared to the native view under a perceptual image similarity metric. This evaluation shows that our system achieves perceptually natural haze removal while maintaining the see-through view of actual scenes.

   @inproceedings{hiroi2020dehaze,
     author = {Hiroi, Yuichi and Kaminokado, Takumi and Mori, Atsushi and Itoh, Yuta},
     title = {DehazeGlasses: Optical Dehazing with an Occlusion Capable See-Through Display},
     year = {2020},
     isbn = {9781450376037},
     publisher = {Association for Computing Machinery},
     address = {New York, NY, USA},
     url = {https://doi.org/10.1145/3384657.3384781},
     doi = {10.1145/3384657.3384781},
     booktitle = {Proceedings of the Augmented Humans International Conference},
     articleno = {3},
     numpages = {11},
     keywords = {Vision Augmentation, Occlusion-Capable HMD, Head-Mounted Displays, Haze Removal, Augmented Reality},
     location = {Kaiserslautern, Germany},
     series = {AHs '20},
   }

3. ISMAR Poster

   

   Stencil Marker: Designing Partially Transparent Markers for Stacking Augmented Reality Objects

   Xuan Zhang, Jonathan Lundgren, Yoya Mesaki, Yuichi Hiroi, and Yuta Itoh

   In 2020 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), Nov 2020

   Abs DOI Bib PDF Video

   We propose a transparent colored AR marker that allows 3D objects to be stacked in space. Conventional AR markers make it difficult to display multiple objects in the same position in space, or to manipulate the order or rotation of objects. The proposed transparent colored markers are designed to detect the order and rotation direction of each marker in the stack from the observed image, based on mathematical constraints. We describe these constraints to design markers, the implementation to detect its stacking order and rotation of each marker, and a proof-of-concept application Totem Poles. We also discuss the limitations of the current prototype and possible research directions.

   @inproceedings{zhang2020stencil,
     author = {Zhang, Xuan and Lundgren, Jonathan and Mesaki, Yoya and Hiroi, Yuichi and Itoh, Yuta},
     booktitle = {2020 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct)},
     title = {Stencil Marker: Designing Partially Transparent Markers for Stacking Augmented Reality Objects},
     year = {2020},
     volume = {},
     number = {},
     pages = {255-257},
     keywords = {Human computer interaction;Three-dimensional displays;Image color analysis;Stacking;Prototypes;Augmented reality;Human-centered computing;Visualization;Visualization techniques;Treemaps;Human-centered computing;Visualization;Visualization design and evaluation methods},
     doi = {10.1109/ISMAR-Adjunct51615.2020.00073},
     issn = {},
     month = nov,
   }

2019

1. ISMAR Poster

   

   OSTNet: Calibration Method for Optical See-Through Head-Mounted Displays via Non-Parametric Distortion Map Generation

   Kiyosato Someya, Yuichi Hiroi, Makoto Yamada, and Yuta Itoh

   In 2019 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), Oct 2019

   Abs DOI Bib PDF

   We propose a spatial calibration method for Optical See-Through Head-Mounted Displays (OST-HMDs) having complex optical distortion such as wide field-of-view (FoV) designs. Viewpoint-dependent non-linear optical distortion makes existing spatial calibration methods either impossible to handle or difficult to compensate without intensive computation. To overcome this issue, we propose OSTNet, a non-parametric data-driven calibration method that creates a generative 2D distortion model for a given six-degree-of-freedom viewpoint pose.

   @inproceedings{someya2019ostnet,
     author = {Someya, Kiyosato and Hiroi, Yuichi and Yamada, Makoto and Itoh, Yuta},
     booktitle = {2019 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct)},
     title = {OSTNet: Calibration Method for Optical See-Through Head-Mounted Displays via Non-Parametric Distortion Map Generation},
     year = {2019},
     volume = {},
     number = {},
     pages = {259-260},
     keywords = {Optical distortion;Calibration;Cameras;Nonlinear distortion;Two dimensional displays;Decoding;optical see through head mounted display;calibration;Variational Autoencoder},
     doi = {10.1109/ISMAR-Adjunct.2019.00-34},
     issn = {},
     month = oct,
   }

2018

1. IEEE TVCG

   

   HySAR: Hybrid Material Rendering by an Optical See-Through Head-Mounted Display with Spatial Augmented Reality Projection

   Takumi Hamasaki, Yuta Itoh, Yuichi Hiroi, Daisuke Iwai, and Maki Sugimoto

   IEEE Transactions on Visualization and Computer Graphics, Apr 2018

   Abs DOI Bib PDF

   Spatial augmented reality (SAR) pursues realism in rendering materials and objects. To advance this goal, we propose a hybrid SAR (HySAR) that combines a projector with optical see-through head-mounted displays (OST-HMD). In an ordinary SAR scenario with co-located viewers, the viewers perceive the same virtual material on physical surfaces. In general, the material consists of two components: a view-independent (VI) component such as diffuse reflection, and a view-dependent (VD) component such as specular reflection. The VI component is static over viewpoints, whereas the VD should change for each viewpoint even if a projector can simulate only one viewpoint at one time. In HySAR, a projector only renders the static VI components. In addition, the OST-HMD renders the dynamic VD components according to the viewer’s current viewpoint. Unlike conventional SAR, the HySAR concept theoretically allows an unlimited number of co-located viewers to see the correct material over different viewpoints. Furthermore, the combination enhances the total dynamic range, the maximum intensity, and the resolution of perceived materials. With proof-of-concept systems, we demonstrate HySAR both qualitatively and quantitatively with real objects. First, we demonstrate HySAR by rendering synthetic material properties on a real object from different viewpoints. Our quantitative evaluation shows that our system increases the dynamic range by 2.24 times and the maximum intensity by 2.12 times compared to an ordinary SAR system. Second, we replicate the material properties of a real object by SAR and HySAR, and show that HySAR outperforms SAR in rendering VD specular components.

   @article{hamasaki2018hysar,
     author = {Hamasaki, Takumi and Itoh, Yuta and Hiroi, Yuichi and Iwai, Daisuke and Sugimoto, Maki},
     journal = {IEEE Transactions on Visualization and Computer Graphics},
     title = {HySAR: Hybrid Material Rendering by an Optical See-Through Head-Mounted Display with Spatial Augmented Reality Projection},
     year = {2018},
     volume = {24},
     number = {4},
     pages = {1457-1466},
     keywords = {Augmented realtiy;Head-mounted displays;Rendering (computer graphics);Optical reflection;Adaptive optics;Optical see-through displays;hybrid material rendering;spatial augmented reality},
     doi = {10.1109/TVCG.2018.2793659},
     issn = {1941-0506},
     month = apr,
   }

2017

1. Augmented Human

   

   AdaptiVisor: assisting eye adaptation via occlusive optical see-through head-mounted displays

   Yuichi Hiroi, Yuta Itoh, Takumi Hamasaki, and Maki Sugimoto

   In Proceedings of the 8th Augmented Human International Conference, Silicon Valley, California, USA, Apr 2017

   Best Paper 3rd Place Abs DOI Bib PDF Video

   Brightness adaptation is a fundamental ability in human visual system, and adjusts various levels of darkness and light. While this ability is continuously used, and it can mostly handle sudden lighting changes in the environment, the adaptation could still take several minutes. Moreover, during the adaptation, the color perception changes as well. This slow reactivity and perception change of the eyes could lead to mistakes for tasks performed in dazzling or temporally high-contrast environments such as when driving into the sun or during a welding process.We present AdaptiVisor, a vision augmentation system that assists the brightness adaptation of the eye. Our system selectively modulates the intensity of the light coming into the eyes via occlusion-capable Optical See-Through Head-Mounted Displays (OST-HMD). An integrated camera captures highlights and brightness in the environment via high-dynamic range capture, and our display system selectively dims or enhances part of field of views so that the user would not perceive rapid brightness changes. We build a proof-of-concept system to evaluate the feasibility of the adaptation assistance by combining a transmissive LCD panel and an OST-HMD, and test it with a user-perspective, view-point camera. The evaluation shows that the system decreases the overexposed area in a scene to 1/15th, and enhances the color by reducing majorly underexposed area to half. We also include a preliminary user trial and it indicates that the system also works for real eyes for the HMD part and to some extent for the LCD.

   @inproceedings{hiroi2017adaptivisor,
     author = {Hiroi, Yuichi and Itoh, Yuta and Hamasaki, Takumi and Sugimoto, Maki},
     title = {AdaptiVisor: assisting eye adaptation via occlusive optical see-through head-mounted displays},
     year = {2017},
     isbn = {9781450348355},
     publisher = {Association for Computing Machinery},
     address = {New York, NY, USA},
     url = {https://doi.org/10.1145/3041164.3041178},
     doi = {10.1145/3041164.3041178},
     booktitle = {Proceedings of the 8th Augmented Human International Conference},
     articleno = {9},
     numpages = {9},
     keywords = {AdaptiVisor, augmented reality, brightness adaptation, head-mounted displays, near-eye displays, occlusive HMD, optical see-through, vision augmentation},
     location = {Silicon Valley, California, USA},
     series = {AH '17},
   }

2. IEEE VR Poster

   HySAR: Hybrid material rendering by an optical see-through head-mounted display with spatial augmented reality projection

   Yuichi Hiroi, Yuta Itoh, Takumi Hamasaki, Daisuke Iwai, and Maki Sugimoto

   In 2017 IEEE Virtual Reality (VR), Mar 2017

   Abs DOI Bib PDF

   We propose a hybrid SAR concept combining a projector and Optical See-Through Head-Mounted Displays (OST-HMD). Our proposed hybrid SAR system utilizes OST-HMD as an extra rendering layer to render a view-dependent property in OST-HMDs according to the viewer’s viewpoint. Combined with view-independent components created by a static projector, the viewer can see richer material contents. Unlike conventional SAR systems, our system theoretically allows unlimited number of viewers seeing enhanced contents in the same space while keeping the existing SAR experiences. Furthermore, the system enhances the total dynamic range, the maximum intensity, and the resolution of perceived materials. With a proof-of-concept system that consists of a projector and an OST-HMD, we qualitatively demonstrate that our system successfully creates hybrid rendering on a hemisphere object from five horizontal viewpoints. Our quantitative evaluation also shows that our system increases the dynamic range by 2.1 times and the maximum intensity by 1.9 times compared to an ordinary SAR system.

   @inproceedings{7892251,
     author = {Hiroi, Yuichi and Itoh, Yuta and Hamasaki, Takumi and Iwai, Daisuke and Sugimoto, Maki},
     booktitle = {2017 IEEE Virtual Reality (VR)},
     title = {HySAR: Hybrid material rendering by an optical see-through head-mounted display with spatial augmented reality projection},
     year = {2017},
     volume = {},
     number = {},
     pages = {211-212},
     keywords = {Rendering (computer graphics);Dynamic range;Spatial resolution;Cameras;Optical imaging;Electronic mail;H.5.1 [Information Interfaces and Presentation]: Multimedia Information Systems — Artificial, augmented, and virtual realities},
     doi = {10.1109/VR.2017.7892251},
     issn = {2375-5334},
     month = mar,
   }

2016

1. ACM TEI

   

   MARCut: Marker-based Laser Cutting for Personal Fabrication on Existing Objects

   Takashi Kikuchi, Yuichi Hiroi, Ross T. Smith, Bruce H. Thomas, and Maki Sugimoto

   In Proceedings of the TEI ’16: Tenth International Conference on Tangible, Embedded, and Embodied Interaction, Eindhoven, Netherlands, Mar 2016

   Abs DOI Bib

   Typical personal fabrication using a laser cutter allows objects to be created from raw material and the engraving of existing objects. Current methods to precisely align an object with the laser is a difficult process due to indirect manipulations. In this paper, we propose a marker-based system as a novel paradigm for direct interactive laser cutting on existing objects. Our system, MARCut, performs the laser cutting based on tangible markers that are applied directly onto the object to express the design. Two types of markers are available; hand constructed Shape Markers that represent the desired geometry, and Command Markers that indicate the operational parameters such as cut, engrave or material.

   @inproceedings{kikuchi2016marcut,
     author = {Kikuchi, Takashi and Hiroi, Yuichi and Smith, Ross T. and Thomas, Bruce H. and Sugimoto, Maki},
     title = {MARCut: Marker-based Laser Cutting for Personal Fabrication on Existing Objects},
     year = {2016},
     isbn = {9781450335829},
     publisher = {Association for Computing Machinery},
     address = {New York, NY, USA},
     url = {https://doi.org/10.1145/2839462.2856549},
     doi = {10.1145/2839462.2856549},
     booktitle = {Proceedings of the TEI '16: Tenth International Conference on Tangible, Embedded, and Embodied Interaction},
     pages = {468–474},
     numpages = {7},
     keywords = {Laser Cutting, Marker-based, Personal Fabrication},
     location = {Eindhoven, Netherlands},
     series = {TEI '16},
   }

2. SIGGRAPH E-tech

   

   Laplacian vision: augmenting motion prediction via optical see-through head-mounted displays and projectors

   Yuta Itoh, Yuichi Hiroi, Jiu Otsuka, Maki Sugimoto, Jason Orlosky, Kiyoshi Kiyokawa, and Gudrun Klinker

   In ACM SIGGRAPH 2016 Emerging Technologies, Anaheim, California, Mar 2016

   DOI Bib Video

   @inproceedings{itoh2016laplacian,
     author = {Itoh, Yuta and Hiroi, Yuichi and Otsuka, Jiu and Sugimoto, Maki and Orlosky, Jason and Kiyokawa, Kiyoshi and Klinker, Gudrun},
     title = {Laplacian vision: augmenting motion prediction via optical see-through head-mounted displays and projectors},
     year = {2016},
     isbn = {9781450343725},
     publisher = {Association for Computing Machinery},
     address = {New York, NY, USA},
     url = {https://doi.org/10.1145/2929464.2949028},
     doi = {10.1145/2929464.2949028},
     booktitle = {ACM SIGGRAPH 2016 Emerging Technologies},
     articleno = {13},
     location = {Anaheim, California},
     series = {SIGGRAPH '16},
   }

2015

1. ISMAR Poster

   

   Remote Welding Robot Manipulation Using Multi-view Images

   Yuichi Hiroi, Kei Obata, Katsuhiro Suzuki, Naoto Ienaga, Maki Sugimoto, Hideo Saito, and Tadashi Takamaru

   In 2015 IEEE International Symposium on Mixed and Augmented Reality, Sep 2015

   Abs DOI Bib

   This paper proposes a remote welding robot manipulation system by using multi-view images. After an operator specifies two-dimensional path on images, the system transforms it into three-dimensional path and displays the movement of the robot by overlaying graphics with images. The accuracy of our system is sufficient to weld objects when combining with a sensor in the robot. The system allows the non-expert operator to weld objects remotely and intuitively, without the need to create a 3D model of a processed object beforehand.

   @inproceedings{hiroi2015remote,
     author = {Hiroi, Yuichi and Obata, Kei and Suzuki, Katsuhiro and Ienaga, Naoto and Sugimoto, Maki and Saito, Hideo and Takamaru, Tadashi},
     booktitle = {2015 IEEE International Symposium on Mixed and Augmented Reality},
     title = {Remote Welding Robot Manipulation Using Multi-view Images},
     year = {2015},
     volume = {},
     number = {},
     pages = {128-131},
     keywords = {Cameras;Welding;Robot vision systems;Three-dimensional displays;Robot kinematics;Remote robot control;multi-view images;industrial robot;augmented reality},
     doi = {10.1109/ISMAR.2015.38},
     issn = {},
     month = sep,
   }