publications

2026

1. IEEE TVCG

   

   J18 DELUGE: Decomposed Entropy-coded Live Unstructured Geometry Exchange for Real-time Particle Streaming

   Hikari Yanagawa, Yuichi Hiroi, and Takefumi Hiraki

   IEEE Transactions on Visualization and Computer Graphics, Oct 2026

   Abs Bib arXiv

   Particle-based physics simulations, including fluids, smoke, and granular media, are fundamental to visual realism in immersive VR and AR. With the growing adoption of social VR and digital twins, demand is increasing for shared experiences in which multiple users interact with the same simulation in real time. Realizing such experiences requires low-latency streaming of large-scale particle data from a server to each client, yet existing point cloud compression methods such as G-PCC (TMC13) and Draco assume static geometric structures; when applied to dynamic particle streaming, their encoding latency exceeds the frame period, failing to meet real-time delivery requirements. We propose DELUGE, a streaming compression architecture that exploits the temporal coherence and velocity predictability inherent in physics simulation particles, achieving sub-frame-latency encoding and decoding through three complementary techniques. Evaluation on dynamic point cloud datasets demonstrates that DELUGE achieves approximately 20× faster decoding than G-PCC (TMC13) and 6× faster encoding than Draco. We further build end-to-end client implementations for both web browsers and Apple Vision Pro, and confirm through a within-participants perceptual quality evaluation and a two-person collaborative task study on Vision Pro that the proposed method supports real-time collaborative experiences with hand-tracked fluid interaction.

   @article{yanagawa2026deluge,
     author = {Yanagawa, Hikari and Hiroi, Yuichi and Hiraki, Takefumi},
     title = {DELUGE: Decomposed Entropy-coded Live Unstructured Geometry Exchange for Real-time Particle Streaming},
     journal = {IEEE Transactions on Visualization and Computer Graphics},
     year = {2026},
     volume = {},
     number = {},
     pages = {to appear},
     month = oct,
   }

2. Front. Virtual Real.

   

   J17 LUIDA: Large-scale Unified Infrastructure for Digital Assessments based on commercial metaverse platform

   Yong-Hao Hu, Sotaro Yokoi, Yuji Hatada, Yuichi Hiroi, Takuji Narumi, and Takefumi Hiraki

   Frontiers in Virtual Reality, Sep 2026

   Award Best Presentation Award, APMAR 2026 (related LUIDA presentation).

   Abs Bib DOI arXiv

   Introduction: Online experiments using metaverse platforms have gained significant traction in Human-Computer Interaction (HCI) and Virtual Reality (VR) research. However, current research workflows are highly fragmented, as researchers must use separate tools for system implementation, participant recruitment, experiment execution, and data collection, reducing consistency and increasing workload. Moreover, the lack of a structured methodology for these metaverse-based approaches forces researchers to either acquire specialized expertise needed for each different platform, or resort to inefficient, manual methods for their experiments. We present LUIDA (Large-scale Unified Infrastructure for Digital Assessments), a metaverse-based framework that integrates these fragmented processes. Methods: LUIDA automatically allocates interconnected virtual environments for parallel experiment execution and provides implementation templates adaptable to various VR research domains, requiring minimal metaverse development expertise. Our evaluation included two studies using a prototype built on Cluster, the commercial metaverse platform. First, VR researchers used LUIDA to develop and run experiments. Second, we conducted three replicated experiments with public Cluster users, each recruiting approximately 200 participants within 1 week. Results: The VR researchers reported good usability (SUS: 73.75) and moderate workload (NASA-TLX: 24.11) for overall usage, while the lower-level experiment-authoring template scored lower on usability and remains a target for improvement; interviews confirmed streamlined workflows compared to traditional laboratory experiments. The replicated experiments largely reproduced the original findings. Discussion: These results support LUIDA’s experimental validity while also revealing where large-scale reproduction diverges from laboratory results. LUIDA has been released publicly and is undergoing technical refinements, eventually providing a unified and structured protocol to improve research efficiency and experimental reproducibility in VR studies.

   @article{hu2026luida,
     author = {Hu, Yong-Hao and Yokoi, Sotaro and Hatada, Yuji and Hiroi, Yuichi and Narumi, Takuji and Hiraki, Takefumi},
     title = {LUIDA: Large-scale Unified Infrastructure for Digital Assessments based on commercial metaverse platform},
     journal = {Frontiers in Virtual Reality},
     year = {2026},
     volume = {7},
     pages = {1847904},
     month = sep,
     doi = {10.3389/frvir.2026.1847904},
     url = {https://www.frontiersin.org/journals/virtual-reality/articles/10.3389/frvir.2026.1847904},
     issn = {2673-4192},
   }

3. Sci. Data

   

   J16 Cluster Haptic Texture Dataset: Haptic Texture Dataset with Varied Velocity-Direction Sliding Contacts

   Michikuni Eguchi, Tomohiro Hayase, Yuichi Hiroi, and Takefumi Hiraki

   Nature Scientific Data, Mar 2026

   Abs Bib DOI arXiv Code Website

   Haptic sciences and technologies benefit greatly from comprehensive datasets that capture tactile stimuli under controlled, systematic conditions. However, existing haptic datasets collect data through uncontrolled exploration, which hinders the systematic analysis of how motion parameters influence tactile perception. This paper introduces the Cluster Haptic Texture Dataset, a multimodal dataset recorded using a 3-axis machine with a rubber tip as an artificial finger to precisely control sliding velocity and direction. The dataset encompasses 118 textured surfaces across nine material categories, with recordings at five velocity levels, eight directions, and two normal-force levels. Each surface was tested under 160 conditions, yielding 18,880 synchronized recordings of audio, acceleration, force, position, and visual data. Validation using convolutional neural networks demonstrates classification accuracies of 96.0% for texture recognition, 88.8% for velocity estimation, and 78.8% for direction estimation, confirming the dataset’s utility for machine learning applications. This resource enables research in haptic rendering, texture recognition algorithms, and human tactile perception mechanisms, supporting the development of realistic haptic interfaces for virtual reality and robotic applications.

   @article{eguchi2026hapticdataset,
     month = mar,
     author = {Eguchi, Michikuni and Hayase, Tomohiro and Hiroi, Yuichi and Hiraki, Takefumi},
     journal = {Nature Scientific Data},
     title = {Cluster Haptic Texture Dataset: Haptic Texture Dataset with Varied Velocity-Direction Sliding Contacts},
     year = {2026},
     volume = {13},
     number = {},
     pages = {756},
     doi = {10.1038/s41597-026-06760-z},
     publisher = {},
   }

4. IEEE TVCG

   

   J15 BeamStellar: Beaming Display with Low-Latency, 6-DoF Glasses Tracking via Spatio-Temporal LED Encoding

   Jonas Weigand, Christian Eichhorn, Yuichi Hiroi, and Yuta Itoh

   IEEE Transactions on Visualization and Computer Graphics, Mar 2026

   Abs Bib DOI

   Conventional Augmented Reality (AR) head-mounted displays (HMDs) integrate all essential components within the wearable unit, imposing limitations on weight, heat dissipation, and battery life. Beaming Displays (BDs) address these constraints by externalizing computation and projection; however, achieving reliable low-latency six-degrees-of-freedom (6-DoF) tracking remains a challenge. We present BeamStellar, a BD system that enables low-latency 6-DoF head tracking by combining a position-sensitive detector (PSD) with a spatio-temporally modulated infrared LED array mounted on passive glasses. The system integrates tracking and projection via a co-aligned optical path and performs real-time signal processing on an FPGA to minimize tracking latency. We describe the system architecture, pose reconstruction pipeline, and evaluate the pose-tracking latency of approximately 430 µs. BeamStellar provides a replicable, lightweight AR tracking platform, establishing a new paradigm for low-latency environment-based AR systems.

   @article{weigand2026stellar,
     author = {Weigand, Jonas and Eichhorn, Christian and Hiroi, Yuichi and Itoh, Yuta},
     title = {BeamStellar: Beaming Display with Low-Latency, 6-DoF Glasses Tracking via Spatio-Temporal LED Encoding},
     journal = {IEEE Transactions on Visualization and Computer Graphics},
     year = {2026},
     volume = {32},
     number = {5},
     pages = {3222-3228},
     doi = {10.1109/TVCG.2026.3680746},
     month = mar,
   }

5. IEICE Trans. Inf. & Syst.

   

   J14 MetaProxy: Bridging Agents and Environments in Commercial Metaverse Platforms through Remote Intermediary Monitoring

   Ryutaro Kurai, Takefumi Hiraki, Yuichi Hiroi, Yutaro Hirao, Monica Perusquia-Hernandez, Hideaki Uchiyama, and Kiyoshi Kiyokawa

   IEICE Transactions on Information and Systems, Jun 2026

   Abs Bib DOI

   Metaverse platforms have emerged as central hubs for virtual communication and entertainment, attracting large numbers of users. Integrating autonomous agent avatars into these spaces could enable diverse applications, including educational support, psychological assistance, and enhanced entertainment experiences. However, commercial metaverse platforms typically lack external Application Programming Interfaces (APIs) for agent control, creating significant implementation barriers for researchers and developers. This paper presents a network proxy module that monitors client-server communication in commercial metaverse platforms, enabling avatar control without platform-side modifications. Our approach extracts spatial information from network packets and controls avatars via keyboard-mouse emulation, requiring no changes to the underlying platform architecture. We validated our method by implementing a Large Language Model (LLM)-integrated agent avatar on Cluster, a commercial metaverse platform. The experimental evaluation demonstrated the agent’s ability to respond to user proximity, comments, and emotional expressions with an average response time of 1.34 seconds.

   @article{kurai2026metaproxy,
     month = jun,
     title = {MetaProxy: Bridging Agents and Environments in Commercial Metaverse Platforms through Remote Intermediary Monitoring},
     author = {Kurai, Ryutaro and Hiraki, Takefumi and Hiroi, Yuichi and Hirao, Yutaro and Perusquia-Hernandez, Monica and Uchiyama, Hideaki and Kiyokawa, Kiyoshi},
     journal = {IEICE Transactions on Information and Systems},
     volume = {E109.D},
     number = {6},
     pages = {870-878},
     year = {2026},
     doi = {10.1587/transinf.2025HCP0008},
   }

6. IEEE ISMAR

   CF20 HaptoFlow: High-Fidelity Real-Time Vibrotactile Generation via Flow Matching for Virtual Reality

   Michikuni Eguchi, Yuichi Hiroi, and Takefumi Hiraki

   In 2026 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), Oct 2026

   Abs Bib arXiv Code Website

   Haptic feedback is widely employed to enhance immersion in Virtual Reality (VR) environments. However, designing haptic stimuli that cover diverse interaction conditions remains a significant scalability challenge. Data-driven haptic generation has emerged as a promising approach, yet existing models face an inherent trade-off between waveform expressiveness and inference responsiveness, which becomes increasingly critical as training data grow in scale and diversity. To address this challenge, we propose HaptoFlow, a vibrotactile generative model based on Flow Matching, designed for interactive real-time haptic rendering in VR. Flow Matching learns a continuous vector field that transforms a base distribution into the target data distribution, enabling efficient representation of complex haptic data distributions and thereby facilitating both high-quality generation and computational efficiency. We train HaptoFlow conditioned on material labels and interaction parameters (stroking velocity and applied force), and integrate it into a VR system. Technical evaluation demonstrates that HaptoFlow outperforms all baseline methods in both waveform reproduction accuracy and inference latency. Furthermore, user studies confirm that the system latency falls well within the perceptual threshold of visual-haptic delay, and statistically significant improvements in perceived haptic quality are observed for a subset of materials. These findings establish a practical foundation for scalable, data-driven haptic content creation in VR, and provide latency benchmarks that inform the design of future real-time haptic rendering systems.

   @inproceedings{eguchi2026haptoflow,
     author = {Eguchi, Michikuni and Hiroi, Yuichi and Hiraki, Takefumi},
     title = {HaptoFlow: High-Fidelity Real-Time Vibrotactile Generation via Flow Matching for Virtual Reality},
     booktitle = {2026 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)},
     year = {2026},
     pages = {to appear},
     month = oct,
   }

7. IEEE VR

   CF19 ImmersiveDMT: A VR System for Expressive Body Movement and Stress Reduction through Dance Movement Therapy

   Satomi Tokida, Yong-Hao Hu, Yuta Itoh, Jean-Marie Normand, Rebecca Fribourg, Jean-Philippe Riviere, Yuichi Hiroi, Takefumi Hiraki, and Yoshio Ishiguro

   In 2026 IEEE Conference on Virtual Reality and 3D User Interfaces (IEEE VR), Mar 2026

   Abs Bib DOI

   In our daily lives, we are exposed to various stressors, which, even in mild forms, can lead to serious health issues when prolonged. Dance Movement Therapy (DMT) uses bodily movement as psychotherapy to improve health and well-being; however, traditional DMT requires therapists, specific setups, and physical props, limiting daily accessibility. To address this, we conducted a formative study with 10 DMT experts and developed ImmersiveDMT, a VR application that provides real-time visual and auditory feedback in response to users’ movements. A user study with 56 participants examined how visual, auditory, and audiovisual feedback affect expressive movement and stress reduction compared to a non-responsive immersive baseline. Results showed that auditory and audiovisual conditions induced significantly more expressive movement compared to baseline. Both visual and auditory feedback achieved significant stress reduction below baseline (V: 17.1%, p=.001; A: 16.6%, p=.014), while combined audiovisual feedback led to less stress reduction. This research presents a novel approach to accessible daily stress management by integrating VR and DMT, demonstrating new possibilities for self-directed mental health support.

   @inproceedings{tokida2026immersivedmt,
     doi = {10.1109/VR67842.2026.00058},
     author = {Tokida, Satomi and Hu, Yong-Hao and Itoh, Yuta and Normand, Jean-Marie and Fribourg, Rebecca and Riviere, Jean-Philippe and Hiroi, Yuichi and Hiraki, Takefumi and Ishiguro, Yoshio},
     title = {ImmersiveDMT: A VR System for Expressive Body Movement and Stress Reduction through Dance Movement Therapy},
     booktitle = {2026 IEEE Conference on Virtual Reality and 3D User Interfaces (IEEE VR)},
     year = {2026},
     volume = {},
     number = {},
     pages = {369--379},
     month = mar,
   }

8. IEEE AIxVR

   CF18 Large-scale, Longitudinal Field Study of AI-agent-User Interactions in Commercial Metaverse

   Yuichi Hiroi, Yuki Imai, Hikari Yanagawa, and Takefumi Hiraki

   In Proceedings of 2026 IEEE International Conference on Artificial Intelligence and eXtended and Virtual Reality (AIxVR), Jan 2026

   Abs Bib DOI PDF

   Although commercial metaverse platforms have advanced social implementation, they struggle to retain users. Most new users abandon these platforms shortly after their initial experience, which hinders the realization of the metaverse’s social and economic potential. Although AI-agents could promote user interaction and retention by acting as social catalysts, existing research has focused on laboratory-based short-term validation. This approach provides limited evidence of the long-term effectiveness of agents in commercial environments. This study examines the impact of Large Language Model (LLM)-based AI-agents on user continuation behavior by operating AI-agents for 31 days on the commercial metaverse platform Cluster and observing the natural usage behavior of 5,020 unique users. The analysis used two complementary approaches: (1) an aggregate effect analysis at the weekly habit formation level and (2) an within-user effect analysis at the daily decision-making level with individual difference controls. The results revealed that interaction with the AI-agent produces lasting changes in user behavior, especially among new users. Furthermore, the cumulative relationship-building process through continuous contact opportunities rather than single impressive experiences with AI-agents was found to decisively affect users’ continued metaverse usage.

   @inproceedings{hiroi2026aiagent,
     pages = {108--118},
     doi = {10.1109/AIxVR67263.2026.00021},
     author = {Hiroi, Yuichi and Imai, Yuki and Yanagawa, Hikari and Hiraki, Takefumi},
     title = {Large-scale, Longitudinal Field Study of AI-agent-User Interactions in Commercial Metaverse},
     year = {2026},
     booktitle = {Proceedings of 2026 IEEE International Conference on Artificial Intelligence and eXtended and Virtual Reality (AIxVR)},
     month = jan,
     series = {AIxVR '26},
   }

9. ISMAR Workshop

   

   CW11 SnapGrid: A Predictable Snap Rotation Technique with a Visible Phase Cue for Anticipatory Comfort in VR

   Norichika Matsuno, Yutaro Hirao, Yuichi Hiroi, and Takefumi Hiraki

   In 2026 IEEE International Symposium on Mixed and Augmented Reality Abstracts and Workshops (ISMAR 2026 Workshop), Oct 2026

   Abs Bib PDF

   Snap rotation is a widely adopted comfort option that mitigates cybersickness by replacing continuous visual rotation with discrete viewpoint changes. Existing implementations, however, schedule these changes without informing the user in advance: while a user holds a sustained rotational input, each discrete change occurs at a moment the user cannot anticipate. We present SnapGrid, a viewpoint control technique that quantizes continuous rotational input into rhythmic snap rotations and externalizes their phase. A sparse spherical grid rotates around the camera in response to the input, and the scene rotates by one grid interval when a grid line crosses a marker fixed at the center of the view, so the timing and magnitude of every snap are perceptually available before it occurs. We describe the design rationale of SnapGrid with reference to the predictability account of cybersickness, which has so far examined anticipation cues only for passive motion, and report a pilot in-the-wild deployment in which 28 of 33 respondents rated camera rotation as more comfortable with SnapGrid enabled than disabled. We outline the controlled experiment needed to isolate the contribution of the anticipation cue.

   @inproceedings{matsuno2026snapgrid,
     author = {Matsuno, Norichika and Hirao, Yutaro and Hiroi, Yuichi and Hiraki, Takefumi},
     title = {SnapGrid: A Predictable Snap Rotation Technique with a Visible Phase Cue for Anticipatory Comfort in VR},
     booktitle = {2026 IEEE International Symposium on Mixed and Augmented Reality Abstracts and Workshops (ISMAR 2026 Workshop)},
     year = {2026},
     pages = {to appear},
     month = oct,
   }

10. ISMAR Workshop

    

    CW10 AdaptLBE: Toward Co-optimizing Real-Object Placement and Redirected Walking to Host an Existing VR World in an Ordinary Room

    Masanori Ibara, Yutaro Hirao, Yuichi Hiroi, and Takefumi Hiraki

    In 2026 IEEE International Symposium on Mixed and Augmented Reality Abstracts and Workshops (ISMAR 2026 Workshop), Oct 2026

    Abs Bib PDF

    Location-based entertainment (LBE) allows participants to experience VR while walking through a physical space. However, it requires a large dedicated tracking space and permanent equipment, which limits its use to specialized areas. One solution is to reuse the existing furniture as passive-haptic proxies while walking a larger VR world within a confined space. Previous studies address these needs separately, adapting either the virtual or the physical environment, or reusing single proxies via redirected walking (RDW). However, none of these studies designs proxy placement and RDW jointly across a scene. We propose AdaptLBE, which combines the two: from a single room scan, it compresses each piece of furniture toward the anchor centroid and builds an RDW gain map that absorbs the residual within perceptual thresholds, giving a VR area larger than the room without resets. We implement the proof-of-concept on one iPhone and one standalone Quest 3, with no PC or dedicated facility. In a pilot test (one participant, 20 seatings), the avatar walked a 23.3 m² VR area inside a 15.5 m² physical room—a 1.5× larger walkable area (expansion ratio R = 1.50)—with all gains within thresholds and a median chair-alignment error of 0.37 m.

    @inproceedings{ibara2026adaptlbe,
      author = {Ibara, Masanori and Hirao, Yutaro and Hiroi, Yuichi and Hiraki, Takefumi},
      title = {AdaptLBE: Toward Co-optimizing Real-Object Placement and Redirected Walking to Host an Existing VR World in an Ordinary Room},
      booktitle = {2026 IEEE International Symposium on Mixed and Augmented Reality Abstracts and Workshops (ISMAR 2026 Workshop)},
      year = {2026},
      pages = {to appear},
      month = oct,
    }

11. IEEE VR Workshop

    

    CW9 Analyzing User Behaviors in Metaverse Brand Events that Drive Real-World Store Visit Intentions

    Satomi Tokida, Yuichi Hiroi, Miyabi Uesugi, Atsushi Shimono, Haruyuki Suzuki, Shouma Abe, and Takefumi Hiraki

    In 2026 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), Mar 2026

    Abs Bib DOI PDF

    As metaverse platforms evolve into social and commercial spaces, companies increasingly host virtual events to enhance brand awareness and customer engagement. However, how participation in such events translates into real-world outcomes—specifically, increased intention to visit physical stores—remains underexplored. We analyzed behavioral logs and survey responses from 814 participants at "Metaverse AEON," a month-long commercial event on the Cluster platform. Six behavioral metrics were compared between users who reported increased store visit intention and those who reported no change. Results suggest that active engagement—game play, photography, emote use, and revisits—was significantly associated with increased visit intention, while dwell time alone showed inconsistent effects. Moreover, these association patterns varied by user segment: Virtual YouTuber fans showed positive associations across multiple metrics, whereas non-fans responded mainly to general-audience content. These findings suggest that engagement quality may play a key role in the transfer from virtual events to real-world behavioral intentions, offering design implications for metaverse brand experiences.

    @inproceedings{tokida2026analyze-aeon,
      author = {Tokida, Satomi and Hiroi, Yuichi and Uesugi, Miyabi and Shimono, Atsushi and Suzuki, Haruyuki and Abe, Shouma and Hiraki, Takefumi},
      title = {Analyzing User Behaviors in Metaverse Brand Events that Drive Real-World Store Visit Intentions},
      booktitle = {2026 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)},
      year = {2026},
      doi = {10.1109/VRW70859.2026.00138},
      volume = {},
      number = {},
      pages = {714--717},
      month = mar,
    }

12. IEEE VR Workshop

    

    CW8 Verselyzer: A Statistical Analysis Interface Enabling Non-Experts to Assess Metaverse Branding Effectiveness

    Sotaro Yokoi, Satomi Tokida, Yuichi Hiroi, and Takefumi Hiraki

    In 2026 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), Mar 2026

    Abs Bib DOI PDF

    Branding in metaverse spaces offers significant potential for immersive customer engagement through embodied, physical-world-like interactions. However, unlike traditional web-based marketing, metaverse branding involves users’ active, spatial behaviors, requiring different KPI design and evaluation methods that remain underexplored. To address this gap, we propose "Verselyzer," a web interface that enables non-experts to design KPIs and evaluate branding effectiveness in metaverse environments. Based on a formative study with seven branding practitioners, we derived design requirements emphasizing (1) metaverse-specific behavioral data collection and (2) accessible analysis support for non-experts. The resulting system progressively supports hypothesis formulation, data visualization, spatial behavior comprehension, and simplified statistical analysis. A user study with 11 practitioners demonstrated that Verselyzer effectively supports hypothesis verification, while also revealing that statistical tools can uncover hidden insights and help formalize experience-based hypotheses. These findings provide practical implications for improving metaverse branding practices as commercial use of virtual spaces continues to grow.

    @inproceedings{yokoi2026verselyzer,
      author = {Yokoi, Sotaro and Tokida, Satomi and Hiroi, Yuichi and Hiraki, Takefumi},
      title = {Verselyzer: A Statistical Analysis Interface Enabling Non-Experts to Assess Metaverse Branding Effectiveness},
      booktitle = {2026 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)},
      year = {2026},
      doi = {10.1109/VRW70859.2026.00134},
      volume = {},
      number = {},
      pages = {692--695},
      month = mar,
    }

13. IEEE VR Workshop

    

    CW7 Depth Perception in Soft-Edge Occlusion-capable Optical See-through Head-mounted Displays

    Gauthier Bertram, Takefumi Hiraki, Yuichi Hiroi, Etienne Peillard, and Jean-Marie Normand

    In 2026 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), Mar 2026

    Abs Bib DOI PDF

    Commercial optical see-through head-mounted displays (OSTHMDs) are becoming increasingly prevalent. While conventional OST-HMDs use optical combiners to overlay virtual objects semitransparently onto the background, recent commercial products have introduced occlusion-capable OST-HMDs (OCOST-HMDs) that selectively block background light using occlusion masks. However, the depth perception characteristics of OCOST-HMDs, particularly the effect of occlusion on depth perception, remain insufficiently investigated. This study evaluated the effect of soft-edge occlusion masks on distance perception accuracy for virtual objects using Magic Leap 2, a commercial OCOST-HMD. We measured participants’ distance estimates while varying the presentation distance, vertical position, presence of drop shadows, and occlusion state of virtual objects. Our preliminary results confirmed that vertical position significantly affects distance perception, consistent with prior work. However, occlusion alone did not significantly improve distance estimation accuracy. These findings suggest that background occlusion for isolated objects provides limited effectiveness as a depth cue.

    @inproceedings{gauthier2026occlusion,
      author = {Bertram, Gauthier and Hiraki, Takefumi and Hiroi, Yuichi and Peillard, Etienne and Normand, Jean-Marie},
      title = {Depth Perception in Soft-Edge Occlusion-capable Optical See-through Head-mounted Displays},
      booktitle = {2026 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)},
      year = {2026},
      doi = {10.1109/VRW70859.2026.00140},
      volume = {},
      number = {},
      pages = {725--732},
      month = mar,
    }

14. EuroHaptics WiP

    CP15 HapCap: Estimating Haptic Adjectives with Vision-Language Model for Dataset Annotation

    Michikuni Eguchi, Yuichi Hiroi, Norihisa Miki, and Takefumi Hiraki

    In EuroHaptics 2026 Work-in-Progress, Jul 2026

    Abs Bib DOI PDF

    Haptic datasets capturing human haptic perceptual ratings underpin haptic texture rendering and cross-modal understanding. However, such datasets are scarce and hard to scale, because annotating objects with human perceptual ratings requires participants to physically handle each item. On the other hand, human haptic perception is closely tied to an object’s visual appearance and linguistic description, suggesting that tactile attributes may be inferable without physical contact. We propose HapCap, a haptic-adjective annotation method using Vision-Language Model (VLM) trained on large-scale image-text corpora. We feed each object’s photograph and name from the dataset into the model to estimate adjective ratings. To investigate how well VLMs can reproduce human haptic judgments, we benchmark three open-weight models on a haptic dataset with human perceptual ratings. All three models stayed within roughly one point of human ratings on the 1–5 scale on average. They broadly captured human haptic perception. These preliminary results position VLMs as a promising alternative that can substantially reduce the cost of haptic dataset annotation.

    @inproceedings{eguchi2026hapcap,
      pages = {1--3},
      doi = {10.5281/zenodo.20272101},
      author = {Eguchi, Michikuni and Hiroi, Yuichi and Miki, Norihisa and Hiraki, Takefumi},
      title = {HapCap: Estimating Haptic Adjectives with Vision-Language Model for Dataset Annotation},
      booktitle = {EuroHaptics 2026 Work-in-Progress},
      year = {2026},
      month = jul,
    }

15. APMAR Demo

    CD4 Saccaidance: Saccade-Aware Pattern Embedding for Gaze Guidance on High-Speed Displays

    Masahiro Nara, Ryusuke Miyazaki, Yuichi Hiroi, Takefumi Hiraki, Yuta Itoh, and Shio Miyafuji

    In Asia-Pacific Workshop on Mixed and Augmented Reality (APMAR 2026), Taiwan, Aug 2026

    Technology demonstration

    Bib

    @inproceedings{nara2026saccaidance-demo,
      author = {Nara, Masahiro and Miyazaki, Ryusuke and Hiroi, Yuichi and Hiraki, Takefumi and Itoh, Yuta and Miyafuji, Shio},
      title = {Saccaidance: Saccade-Aware Pattern Embedding for Gaze Guidance on High-Speed Displays},
      booktitle = {Asia-Pacific Workshop on Mixed and Augmented Reality (APMAR 2026)},
      year = {2026},
      month = aug,
      location = {Taiwan},
      note = {Technology demonstration},
    }

2025

1. IEEE Perv. Comp.

   

   J13 Cross-Reality Lifestyle: Integrating Physical and Virtual Lives Through Multiplatform Metaverse

   Yuichi Hiroi, Yuji Hatada, and Takefumi Hiraki

   IEEE Pervasive Computing, Oct 2025

   Abs Bib DOI arXiv

   Technological advances are redefining the relationship between physical and virtual spaces. Traditionally, when users engage in virtual reality, they are completely cutoff from the physical space. Similarly, they are unable to access virtual experiences while engaged in physical activities. However, modern multiplatform metaverse environments allow simultaneous participation through mobile devices, creating new opportunities for integrated experiences. This study introduces the concept of “cross-reality lifestyles” to examine how users actively combine their physical and virtual activities. We identify three patterns of integration: first, Amplification: one space enhances experiences in the other; second, Complementary: spaces offer different but equally valuable alternatives, and third, Emergence: simultaneous engagement creates entirely new experiences. We propose the ACE cube framework that analyzes these patterns as continuous characteristics, and by integrating this analysis with technical requirements of commercial platforms, we provide practical guidelines for platform selection, technical investment prioritization, and cross-reality application development.

   @article{hiroi2025crossreality,
     month = oct,
     author = {Hiroi, Yuichi and Hatada, Yuji and Hiraki, Takefumi},
     journal = {IEEE Pervasive Computing},
     title = {Cross-Reality Lifestyle: Integrating Physical and Virtual Lives Through Multiplatform Metaverse},
     year = {2025},
     volume = {24},
     number = {4},
     pages = {19-28},
     keywords = {Metaverse;Avatars;Three-dimensional displays;Digital twins;Training;Smart phones;Robots;Pervasive computing;Technical requirements;Fans},
     doi = {10.1109/MPRV.2025.3610749},
     url = {https://doi.ieeecomputersociety.org/10.1109/MPRV.2025.3610749},
     publisher = {IEEE Computer Society},
   }

2. IEEE Perv. Comp.

   

   J12 MetaGadget: An Accessible Framework for IoT Integration Into Commercial Metaverse Platforms

   Ryutaro Kurai, Hikari Yanagawa, Yuichi Hiroi, and Takefumi Hiraki

   IEEE Pervasive Computing, Sep 2025

   Award Good Experience Design Award 2025, DAXX.

   Abs Bib DOI arXiv Code

   While the integration of Internet of Things (IoT) devices in virtual spaces is becoming increasingly common, technical barriers to controlling custom devices in multiuser virtual reality (VR) environments remain high, particularly limiting new applications in educational and prototyping settings. We propose MetaGadget, a framework for connecting IoT devices to commercial metaverse platforms that implements device control through HTTP-based event triggers without requiring persistent client connections. Through two workshops focused on smart home control and custom device integration, we explored the potential application of IoT connectivity in multiuser metaverse environments. Participants successfully implemented new interactions unique to the metaverse, such as environmental sensing and remote control systems that support simultaneous operation by multiple users, and reported positive feedback on the ease of system development. We verified that our framework provides a new approach to controlling IoT devices in the metaverse while reducing technical requirements and provides a foundation for creative practice that connects multiuser VR environments and physical spaces.

   @article{kurai2025metagadget,
     month = sep,
     author = {Kurai, Ryutaro and Yanagawa, Hikari and Hiroi, Yuichi and Hiraki, Takefumi},
     journal = {IEEE Pervasive Computing},
     title = {MetaGadget: An Accessible Framework for IoT Integration Into Commercial Metaverse Platforms},
     year = {2025},
     volume = {24},
     number = {4},
     pages = {63-73},
     keywords = {Metaverse;Internet of Things;Servers;Switches;Conferences;Aerospace electronics;Fans;Control systems;HTTP;Digital twins},
     doi = {10.1109/MPRV.2025.3602289},
   }

3. IEEE Access

   

   J11 MagicCraft: Natural Language-Driven Generation of Dynamic and Interactive 3D Objects for Commercial Metaverse Platforms

   Ryutaro Kurai, Takefumi Hiraki, Yuichi Hiroi, Yutaro Hirao, Monica Perusquía-Hernández, Hideaki Uchiyama, and Kiyoshi Kiyokawa

   IEEE Access, Jul 2025

   Abs Bib DOI arXiv

   Metaverse platforms are rapidly evolving to provide immersive spaces for user interaction and content creation. However, the generation of dynamic and interactive 3D objects remains challenging due to the need for advanced 3D modeling and programming skills. To address this challenge, we present MagicCraft, a system that generates functional 3D objects from natural language prompts for metaverse platforms. MagicCraft uses generative AI models to manage the entire content creation pipeline: converting user text descriptions into images, transforming images into 3D models, predicting object behavior, and assigning necessary attributes and scripts. It also provides an interactive interface for users to refine generated objects by adjusting features such as orientation, scale, seating positions, and grip points. Implemented on Cluster, a commercial metaverse platform, MagicCraft was evaluated by 7 expert CG designers and 51 general users. Results show that MagicCraft significantly reduces the time and skill required to create 3D objects. Users with no prior experience in 3D modeling or programming successfully created complex, interactive objects and deployed them in the metaverse. Expert feedback highlighted the system’s potential to improve content creation workflows and support rapid prototyping. By integrating AI-generated content into metaverse platforms, MagicCraft makes 3D content creation more accessible.

   @article{kurai2025magiccraft,
     month = jul,
     author = {Kurai, Ryutaro and Hiraki, Takefumi and Hiroi, Yuichi and Hirao, Yutaro and Perusquía-Hernández, Monica and Uchiyama, Hideaki and Kiyokawa, Kiyoshi},
     journal = {IEEE Access},
     title = {MagicCraft: Natural Language-Driven Generation of Dynamic and Interactive 3D Objects for Commercial Metaverse Platforms},
     year = {2025},
     volume = {13},
     number = {},
     pages = {132459-132474},
     keywords = {Three-dimensional displays;Metaverse;Solid modeling;Generative AI;Codes;Programming;Natural languages;Biological system modeling;Visualization;Pipelines;Metaverse;3D object generation;generative AI;AI-assisted design},
     doi = {10.1109/ACCESS.2025.3587232},
   }

4. IEEE TVCG

   

   J10 ChromaGazer: Unobtrusive Visual Modulation using Imperceptible Color Vibration for Visual Guidance

   Rinto Tosa, Shingo Hattori, Yuichi Hiroi, Yuta Itoh, and Takefumi Hiraki

   IEEE Transactions on Visualization and Computer Graphics, Mar 2025

   Abs Bib DOI arXiv PDF

   Visual guidance (VG) plays an essential role in directing user attention in virtual reality (VR) and augmented reality (AR) environments. However, traditional approaches rely on explicit visual annotations, which often compromise visual clarity and increase user cognitive load. To address this issue, we propose an unobtrusive VG technique based on color vibration, a phenomenon in which rapidly alternating colors at frequencies above 25 Hz are perceived as a single intermediate color. Our work explores a perceptual state that exists between complete color fusion and visible flicker, where color differences remain detectable without conscious awareness of vibration. Through two experimental studies, we first identified the thresholds separating complete fusion, this intermediate perceptual state, and visible flicker by systematically varying color vibration parameters. Subsequently, we applied color vibrations with derived thresholds to natural image regions and validated their attention-guiding capabilities using eye-tracking measurements. The results demonstrate that controlled color vibration successfully directs user attention while maintaining low cognitive demand, providing an effective method for implementing unobtrusive VG in VR and AR systems.

   @article{tosa2025chromagazer,
     month = mar,
     author = {Tosa, Rinto and Hattori, Shingo and Hiroi, Yuichi and Itoh, Yuta and Hiraki, Takefumi},
     journal = {IEEE Transactions on Visualization and Computer Graphics},
     title = {ChromaGazer: Unobtrusive Visual Modulation using Imperceptible Color Vibration for Visual Guidance},
     year = {2025},
     volume = {31},
     number = {5},
     pages = {3450-3458},
     keywords = {Visual Guidance;Imperceptible Color Vibration;Color Perception;Augmented Reality},
     doi = {10.1109/TVCG.2025.3549173}
   }

5. IEEE Access

   

   J9 MagicItem: Dynamic Behavior Design of Virtual Objects With Large Language Models in a Commercial Metaverse Platform

   Ryutaro Kurai, Takefumi Hiraki, Yuichi Hiroi, Yutaro Hirao, Monica Perusquia-Hernandez, Hideki Uchiyama, and Kiyoshi Kiyokawa

   IEEE Access, Jan 2025

   Abs Bib DOI arXiv

   To create rich experiences in virtual reality (VR) environments, it is essential to define the behavior of virtual objects through programming. However, programming in 3D spaces requires a wide range of background knowledge and programming skills. Although Large Language Models (LLMs) have provided programming support, they are still primarily aimed at programmers. In metaverse platforms, where many users inhabit VR spaces, most users are unfamiliar with programming, making it difficult for them to modify the behavior of objects in the VR environment easily. Existing LLM-based script generation methods for VR spaces require multiple lengthy iterations to implement the desired behaviors and are difficult to integrate into the operation of metaverse platforms. To address this issue, we propose a tool that generates behaviors for objects in VR spaces from natural language within Cluster, a metaverse platform with a large user base. By integrating LLMs with the Cluster Script provided by this platform, we enable users with limited programming experience to define object behaviors within the platform freely. We have also integrated our tool into a commercial metaverse platform and are conducting online experiments with 63 general users of the platform. The experiments show that even users with no programming background can successfully generate behaviors for objects in VR spaces, resulting in a highly satisfying system. Our research contributes to democratizing VR content creation by enabling non-programmers to design dynamic behaviors for virtual objects in metaverse platforms.

   @article{kurai2025magicitem,
     month = jan,
     author = {Kurai, Ryutaro and Hiraki, Takefumi and Hiroi, Yuichi and Hirao, Yutaro and Perusquia-Hernandez, Monica and Uchiyama, Hideki and Kiyokawa, Kiyoshi},
     journal = {IEEE Access},
     title = {MagicItem: Dynamic Behavior Design of Virtual Objects With Large Language Models in a Commercial Metaverse Platform},
     year = {2025},
     volume = {13},
     number = {},
     pages = {19132-19143},
     doi = {10.1109/ACCESS.2025.3530439},
     keywords = {Metaverse;Codes;Three-dimensional displays;Large language models;Natural languages;Aerospace electronics;Web servers;Usability;Programming profession;Hands;Large-language model;low-code programming;metaverse platform;virtual reality},
   }

6. ACM VRST

   CF17 Saccaidance: Saccade-Aware Pattern Embedding for Gaze Guidance on High-Speed Displays

   Masahiro Nara, Ryusuke Miyazaki, Yuichi Hiroi, Takefumi Hiraki, Yuta Itoh, and Shio Miyafuji

   In Proceedings of the 2025 31st ACM Symposium on Virtual Reality Software and Technology, Nov 2025

   Abs Bib DOI PDF

   Gaze guidance is essential for directing user attention to specific areas of interest. However, conventional visual cues generate persistent visual noise that hinders concentration during tasks. We propose Saccaidance, a gaze-guidance method that appears only when users move their gaze. Saccaidance employs temporal additive color mixing and 480 Hz high-speed displays to shift the color phase of guidance patterns. This renders the patterns barely visible during fixation and makes them appear transiently when users move their gaze as a color-breaking effect. This intermittent gaze guidance appears only during gaze transitions, providing effective guidance without interfering with focused work or requiring eye-tracking hardware. We conducted experiments with 24 participants under four conditions that involved search tasks: an unmodified baseline, conventional explicit guidance, and our proposed method using oval and radial patterns. The results show that our approach effectively constrains the exploration area while preserving subjective naturalness. We also outline application scenarios of our method, including document highlighting.

   @inproceedings{nara2025saccaidance,
     pages = {1--11},
     author = {Nara, Masahiro and Miyazaki, Ryusuke and Hiroi, Yuichi and Hiraki, Takefumi and Itoh, Yuta and Miyafuji, Shio},
     title = {Saccaidance: Saccade-Aware Pattern Embedding for Gaze Guidance on High-Speed Displays},
     year = {2025},
     isbn = {9798400721182},
     publisher = {Association for Computing Machinery},
     address = {New York, NY, USA},
     url = {https://doi.org/10.1145/3756884.3766051},
     doi = {10.1145/3756884.3766051},
     booktitle = {Proceedings of the 2025 31st ACM Symposium on Virtual Reality Software and Technology},
     articleno = {36},
     numpages = {11},
     keywords = {Gaze Guidance, Temporal Additive Color Mixing, Color Vibration, Augmented Reality},
     series = {VRST '25},
     month = nov,
   }

7. IEEE ISEMV

   CF16 Community Analysis of Social Virtual Reality Based on Large-Scale Log Data of a Commercial Metaverse Platform

   Hiroto Tsutsui, Takefumi Hiraki, Yuichi Hiroi, and Shoichi Hasegawa

   In Proceedings of 2025 IEEE International Symposium on Emergence Metaverse, Oct 2025

   Best paper Best Paper Award, IEEE ISEMV 2025.

   Abs Bib DOI arXiv

   This study quantitatively analyzes the structural characteristics of user communities within Social Virtual Reality (Social VR) platforms supporting head-mounted displays (HMDs), based on large-scale log data. By detecting and evaluating community structures from data on substantial interactions (defined as prolonged co-presence in the same virtual space), we found that Social VR platforms tend to host numerous, relatively small communities characterized by strong internal cohesion and limited inter-community connections. This finding contrasts with the large-scale, broadly connected community structures typically observed in conventional Social Networking Services (SNS). Furthermore, we identified a user segment capable of mediating between communities, despite these users not necessarily having numerous direct connections. We term this user segment ‘community hoppers’ and discuss their characteristics. These findings contribute to a deeper understanding of the community structures that emerge within the unique communication environment of Social VR and the roles users play within them.

   @inproceedings{tsutsui2025communityanalysis,
     title = {Community Analysis of Social Virtual Reality Based on Large-Scale Log Data of a Commercial Metaverse Platform},
     author = {Tsutsui, Hiroto and Hiraki, Takefumi and Hiroi, Yuichi and Hasegawa, Shoichi},
     year = {2025},
     booktitle = {Proceedings of 2025 IEEE International Symposium on Emergence Metaverse},
     month = oct,
     pages = {95-103},
     doi = {10.1109/ISEMV67326.2025.00026},
   }

8. IEEE ISEMV

   CF15 Implementation and Application of Multi-Format 3D Data Integration in a Cross-Device Commercial Metaverse Platform

   Masanori Ibara, Yuichi Hiroi, Takushi Kamegai, and Takefumi Hiraki

   In Proceedings of 2025 IEEE International Symposium on Emergence Metaverse, Oct 2025

   Abs Bib DOI arXiv

   Traditionally, specialized 3D design data, such as BIM and CAD, have been accessible only to a select group of experts, creating significant barriers that prevent general users from participating in decision-making processes. This paper provides a systematic overview of practical insights for utilizing 3D data in industrial and architectural domains by presenting implementation cases of the industrial metaverse on Cluster, a commercial cross-device metaverse platform. This paper analyzes the characteristics and constraints of major data formats in the industrial and architectural fields and organizes integration workflows for the metaverse. Through application cases utilizing 3D data across multiple domains, we present practical examples of collaborative decision-making support enabled by the fusion of metaverse and digital twin technologies. Specifically, we demonstrate that multi-device access and simultaneous multi-user participation capabilities foster democratic environments in the industrial metaverse, which are challenging to achieve with conventional, expert-dependent systems.

   @inproceedings{ibara2025implementation,
     title = {Implementation and Application of Multi-Format 3D Data Integration in a Cross-Device Commercial Metaverse Platform},
     author = {Ibara, Masanori and Hiroi, Yuichi and Kamegai, Takushi and Hiraki, Takefumi},
     year = {2025},
     booktitle = {Proceedings of 2025 IEEE International Symposium on Emergence Metaverse},
     month = oct,
     pages = {104-113},
     doi = {10.1109/ISEMV67326.2025.00027},
   }

9. IEEE ISEMV

   CF14 Weather Synchronization in Digital Twin Environments for Shared VR Experience Using Commercial Metaverse Platforms

   Masanori Ibara, Yuichi Hiroi, Takushi Kamegai, Yusuke Masubuchi, Kazuki Matsutani, Megumi Zaizen, Junya Morita, and Takefumi Hiraki

   In Proceedings of 2025 IEEE International Symposium on Emergence Metaverse, Oct 2025

   Abs Bib DOI

   Digital twin technology creates bidirectional synchronization between physical and virtual environments, yet current implementations fail to provide authentic environmental experiences that enhance user presence in shared virtual spaces. While digital twin environments using commercial metaverse platforms for IoT sensor data visualization have been proposed, translating environmental information into meaningful sensory experiences remains largely unexplored, particularly lacking approaches for weather conditions that significantly influence spatial perception. We developed a weather synchronization system that integrates real-time environmental data from “The GEAR” smart building with the Cluster metaverse platform, enabling shared VR experiences with authentic atmospheric immersion. Our system processes temperature, humidity, precipitation, wind speed, and solar radiation measurements to generate corresponding virtual environmental effects including dynamic sky rendering, precipitation particles, and ambient audio modulation. Performance evaluation demonstrated practical response times of 0.8-1.0 seconds for weather data transmission and reflection in the virtual environment. This work establishes a foundation for seamless physical-digital space integration, potentially enhancing remote collaboration efficiency and facilitating more dynamic discussions in shared virtual workspaces.

   @inproceedings{ibara2025weather,
     author = {Ibara, Masanori and Hiroi, Yuichi and Kamegai, Takushi and Masubuchi, Yusuke and Matsutani, Kazuki and Zaizen, Megumi and Morita, Junya and Hiraki, Takefumi},
     title = {Weather Synchronization in Digital Twin Environments for Shared VR Experience Using Commercial Metaverse Platforms},
     booktitle = {Proceedings of 2025 IEEE International Symposium on Emergence Metaverse},
     year = {2025},
     volume = {},
     number = {},
     pages = {159-162},
     doi = {10.1109/ISEMV67326.2025.00037},
     month = oct,
   }

10. ACM ICMI

    CF13 Pinching Visuo-haptic Display: Investigating Cross-Modal Effects of Visual Textures on Electrostatic Cloth Tactile Sensations

    Takekazu Kitagishi, Chun Wei Ooi, Yuichi Hiroi, and Jun Rekimoto

    In Proceedings of the 27th International Conference on Multimodal Interaction, Oct 2025

    Abs Bib DOI

    Tactile representations of fabrics are critical in virtual retail and VR fitting applications. A pinching tactile display (PTD) using electrostatic adhesion is a fabric-type display that can flexibly adjust tactile sensations by adjusting voltage and frequency, and is expected to be applied to the remote transmission of fabric tactile sensations. However, no significant differences in tactile sensation other than roughness were observed. Building on established research in cross-modal perception, we investigated whether overlaying visual fabric textures could alter the tactile expressiveness of PTDs. We developed a system that integrated a systematically controlled conductive fabric-based PTD with spatially registered VR fabric simulations. Participants rated four tactile properties (roughness, stiffness, thickness, warmth) while touching the same conductive substrate under different visual texture conditions (denim, gauze, toweling, voile). The results showed that voile texture significantly affected stiffness (p=.00887); toweling texture tended to affect warmth (p=.0658); and denim texture tended to affect roughness (p=.0972). These results suggest that virtual texture overlay can affect the warmth, stiffness, and roughness of the PTD. Since the PTD can only change roughness, these results suggest that texture overlay enables the display of tactile sensations that the PTD alone cannot express. On the other hand, no significant influence of texture on thickness was observed. This suggests the pinching motion provided a highly reliable tactile cue, which may have led to visual information being underrated and thus not reaching a significant level. These findings offer strategic insights for employing visual textures to enhance perceived fabric properties, paving the way for deployment in virtual fashion retail and VR fitting applications.

    @inproceedings{kitagishi2025pinching,
      author = {Kitagishi, Takekazu and Ooi, Chun Wei and Hiroi, Yuichi and Rekimoto, Jun},
      title = {Pinching Visuo-haptic Display: Investigating Cross-Modal Effects of Visual Textures on Electrostatic Cloth Tactile Sensations},
      year = {2025},
      isbn = {9798400714993},
      publisher = {Association for Computing Machinery},
      address = {New York, NY, USA},
      doi = {10.1145/3716553.3750810},
      booktitle = {Proceedings of the 27th International Conference on Multimodal Interaction},
      pages = {624–633},
      numpages = {10},
      keywords = {MultiModal; Cross-modal Effect; VR; Physical Simulation; Haptic Display;},
      series = {ICMI '25},
      month = oct,
    }

11. IEEE ISMAR

    CF12 Navigation Pixie: Implementation and Empirical Study Toward On-demand Navigation Agents in Commercial Metaverse

    Hikari Yanagawa, Yuichi Hiroi, Satomi Tokida, Yuji Hatada, and Takefumi Hiraki

    In Proceedings of 2025 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), Oct 2025

    Abs Bib DOI arXiv PDF Video Website

    While commercial metaverse platforms offer diverse user-generated content, they lack effective navigation assistance that can dynamically adapt to users’ interests and intentions. Although previous research has investigated on-demand agents in controlled environments, implementation in commercial settings with diverse world configurations and platform constraints remains challenging.We present Navigation Pixie, an on-demand navigation agent employing a loosely coupled architecture that integrates structured spatial metadata with LLM-based natural language processing while minimizing platform dependencies, which enables experiments on the extensive user base of commercial metaverse platforms. Our cross-platform experiments on commercial metaverse platform Cluster with 99 PC client and 94 VR-HMD participants demonstrated that Navigation Pixie significantly increased dwell time and free exploration compared to fixed-route and no-agent conditions across both platforms. Subjective evaluations revealed consistent on-demand preferences in PC environments versus context-dependent social perception advantages in VR-HMD. This research contributes to advancing VR interaction design through conversational spatial navigation agents, establishes cross-platform evaluation methodologies revealing environment-dependent effectiveness, and demonstrates empirical experimentation frameworks for commercial metaverse platforms.

    @inproceedings{yanagawa2025navigationpixie,
      pages = {1137--1147},
      title = {Navigation Pixie: Implementation and Empirical Study Toward On-demand Navigation Agents in Commercial Metaverse},
      author = {Yanagawa, Hikari and Hiroi, Yuichi and Tokida, Satomi and Hatada, Yuji and Hiraki, Takefumi},
      year = {2025},
      booktitle = {Proceedings of 2025 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)},
      month = oct,
      doi = {10.1109/ISMAR67309.2025.00120},
    }

12. Augmented Humans

    CF11 ChromaGazer-HMD: Visual Modulation using Unobtrusive Color Vibration for Gaze Guidance with Head-Mounted Displays

    Youfang Shen, Rinto Tosa, Yuji Hatada, Yuichi Hiroi, Takefumi Hiraki, and Takeshi Naemura

    In Proceedings of the Augmented Humans International Conference 2025, Mar 2025

    Abs Bib DOI PDF

    VR content often involves vast spaces for users to explore, making it uncertain whether they will notice areas or objects highlighted by content creators. While various methods for guiding users’ gaze have been proposed, those that significantly alter the appearance of the content can reduce its naturalness and negatively impact the user experience. In this study, we explored a Visual Guidance (VG) approach using color vibration, based on the human visual system’s inability to perceive rapid chromatic changes. We evaluated its applicability to HMD environments and introduced a method called complementary color vibration, where opposite-phase vibrations are applied to the left and right displays of the HMD. We further investigated optimal parameters for VG and assessed its effectiveness in search tasks. Our experiments revealed that color vibration in HMD environments significantly reduced search times and areas compared to conditions without guidance, while maintaining the naturalness of the content. Furthermore, complementary color vibration was found to preserve naturalness significantly better than conventional synchronous color vibration. These findings indicate that VG using color vibration is effective in HMD environments and that complementary color vibration is less obtrusive than traditional synchronous color vibration.

    @inproceedings{shen2025chromagazerhmd,
      author = {Shen, Youfang and Tosa, Rinto and Hatada, Yuji and Hiroi, Yuichi and Hiraki, Takefumi and Naemura, Takeshi},
      title = {ChromaGazer-HMD: Visual Modulation using Unobtrusive Color Vibration for Gaze Guidance with Head-Mounted Displays},
      year = {2025},
      isbn = {9798400715662},
      publisher = {Association for Computing Machinery},
      address = {New York, NY, USA},
      doi = {10.1145/3745900.3746090},
      booktitle = {Proceedings of the Augmented Humans International Conference 2025},
      pages = {43–54},
      numpages = {12},
      month = mar,
    }

13. IEEE VR

    CF10 Slim Diffractive Waveguide Glasses for Beaming Displays with Enhanced Head Orientation Tolerance

    Yuta Itoh, Tomoya Nakamura, Yuichi Hiroi, and Kaan Aksit

    In 2025 IEEE Conference on Virtual Reality and 3D User Interfaces (IEEE VR), Mar 2025

    Abs Bib DOI PDF

    Augmented Reality (AR) glasses must be slim, lightweight, and energy-efficient to achieve widespread adoption. Beaming Displays present a promising solution by offloading active components, such as the power-supplied light engine, into the surrounding environment while leaving only passive elements, like the eyepiece, in the wearable device. However, existing approaches still struggle to achieve both a slim design and a wide tolerance for projection angles relative to the user’s head orientation. In this work, we introduce a design for light-receiving glasses using a diffractive waveguide with in-coupling and out-coupling gratings. Our approach expands the allowable range of incident angles while maintaining a compact, lightweight form factor. We developed a proof-of-concept prototype and demonstrated an incident angle tolerance of approximately 20-30 degrees range, overcoming the previous design of 5 degrees.

    @inproceedings{itoh2025hoebeaming,
      author = {Itoh, Yuta and Nakamura, Tomoya and Hiroi, Yuichi and Aksit, Kaan},
      title = {Slim Diffractive Waveguide Glasses for Beaming Displays with Enhanced Head Orientation Tolerance},
      booktitle = {2025 IEEE Conference on Virtual Reality and 3D User Interfaces (IEEE VR)},
      year = {2025},
      volume = {},
      number = {},
      pages = {351-358},
      keywords = {beaming display; augmented reality; waveguide; near-eye displays; DOEs},
      month = mar,
      doi = {10.1109/VR59515.2025.00059},
    }

14. ISMAR Workshop

    

    CW6 MagicPen: Interactive Sketch-to-3D Generation in Commertial Metaverse Platforms

    Ryutaro Kurai, Yuji Hatada, Takefumi Hiraki, and Yuichi Hiroi

    In 2025 IEEE International Symposium on Mixed and Augmented Reality Abstracts and Workshops (ISMAR 2025 Workshop), Oct 2025

    Abs Bib DOI PDF

    Although User Generated Content (UGC) distinguishes metaverse platforms from conventional VR, 3D content creation remains limited to users with specialized modeling expertise and platform-specific technical knowledge. While generative AI advances have improved accessibility, existing approaches fail to provide both VR-native operation and intuitive input capabilities simultaneously, maintaining barriers for novice users. We propose MagicPen, a system enabling direct conversion of hand-drawn sketches into 3D objects within VR environments through a three-stage AI pipeline comprising image refinement, 3D model generation, and metaverse integration. Evaluation with 68 diverse sketch inputs revealed 35.5-second average processing times and high-fidelity geometric reproduction, validating seamless Cluster platform integration. The system enables collaborative 3D content creation in social VR spaces, allowing users without modeling expertise to transform individual creative workflows into shared social experiences.

    @inproceedings{kurai2025magicpen,
      author = {Kurai, Ryutaro and Hatada, Yuji and Hiraki, Takefumi and Hiroi, Yuichi},
      title = {MagicPen: Interactive Sketch-to-3D Generation in Commertial Metaverse Platforms},
      booktitle = {2025 IEEE International Symposium on Mixed and Augmented Reality Abstracts and Workshops (ISMAR 2025 Workshop)},
      year = {2025},
      volume = {},
      number = {},
      pages = {602--604},
      keywords = {Metaverse;Sketch-based Modeling;3D Generative AI},
      month = oct,
      doi = {10.1109/ISMAR-Adjunct68609.2025.00119},
    }

15. ISMAR Workshop

    

    CW5 Dynamic Gimmick Leaning for Navigation Agents in Social VR Through User-Agent Dialogue

    Atsuya Matsumoto, Hikari Yanagawa, Yuichi Hiroi, Yuji Hatada, Takuji Narumi, and Takefumi Hiraki

    In 2025 IEEE International Symposium on Mixed and Augmented Reality Abstracts and Workshops (ISMAR 2025 Workshop), Oct 2025

    Abs Bib DOI PDF

    User-generated interactive mechanisms (“gimmicks”) in metaverse platforms offer unique experiences, but their complexity hinders player exploration. Existing navigation agents, reliant on static metadata, cannot comprehend these dynamic gimmicks. We propose the “Conversational Knowledge Acquisition Loop,” a method for agents to dynamically learn gimmicks through player conversation. A preliminary study showed that our agent successfully acquired knowledge via conversation and applied it to solve analogous gimmicks.

    @inproceedings{matsumoto2025dynamic,
      author = {Matsumoto, Atsuya and Yanagawa, Hikari and Hiroi, Yuichi and Hatada, Yuji and Narumi, Takuji and Hiraki, Takefumi},
      title = {Dynamic Gimmick Leaning for Navigation Agents in Social VR Through User-Agent Dialogue},
      booktitle = {2025 IEEE International Symposium on Mixed and Augmented Reality Abstracts and Workshops (ISMAR 2025 Workshop)},
      year = {2025},
      volume = {},
      number = {},
      pages = {605--607},
      keywords = {Autonomous Agent;Virtual Reality;Embodied Conversational Agent;Social VR.},
      month = oct,
      doi = {10.1109/ISMAR-Adjunct68609.2025.00120},
    }

16. IEEE VR Workshop

    

    CW4 BeamStellar: Low-Latency, 6-DoF Head Tracking for Beaming Displays with Spatio-Temporal LED Encoding

    Jonas Weigand, Yuichi Hiroi, and Yuta Itoh

    In 2025 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), Mar 2025

    Abs Bib DOI PDF

    Conventional augmented reality (AR) head-mounted displays (HMDs) require the integration of all components, including optics and processors, which presents significant barriers to practical implementation due to weight and heat generation concerns. While Beaming Displays address these challenges by offloading computing and projection functions to the environment, existing solutions leaving 6-degrees of freedom (6DoF) image projection with low motion-to-photon (M2P) latency. In this position paper, we introduce our trial BeamStellar, a Beaming Display system that integrates 6 DoF head tracking by combining spatio-temporally encoded LED patterns with a position-sensing detector. Through digital signal processing using an FPGA, the system aims a theoretical latency of less than 200 microseconds. This implementation alleviates the hardware burden on users, facilitating seamless AR experiences that seamlessly blend virtual content with the physical world.

    @inproceedings{weigand2025stellar,
      author = {Weigand, Jonas and Hiroi, Yuichi and Itoh, Yuta},
      title = {BeamStellar: Low-Latency, 6-DoF Head Tracking for Beaming Displays with Spatio-Temporal LED Encoding},
      booktitle = {2025 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)},
      year = {2025},
      volume = {},
      number = {},
      pages = {884--886},
      keywords = {Beaming Displays;Optical See-through Display;Low latency head tracking;Spatio-temporal Light pattern},
      month = mar,
      doi = {10.1109/VRW66409.2025.00180},
    }

17. IEEE VR Workshop

    

    CW3 Development of Digital Twin Environment through Integration of Commercial Metaverse Platform and IoT Sensors of Smart Building

    Yusuke Masubuchi, Takefumi Hiraki, Yuichi Hiroi, Masanori Ibara, Kazuki Matsutani, Megumi Zaizen, and Junya Morita

    In 2025 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), Mar 2025

    Abs Bib DOI

    The digital transformation of smart cities and workplaces requires effective integration of physical and cyber spaces, yet existing digital twin solutions remain limited in supporting real-time, multi-user collaboration. While metaverse platforms enable shared virtual experiences, they have not supported comprehensive integration of IoT sensors on physical spaces, especially for large-scale smart architectural environments. This paper presents a digital twin environment that integrates Kajima Corp.’s smart building facility "The GEAR" in Singapore with a commercial metaverse platform Cluster. Our system consists of three key components: a standardized IoT sensor platform, a real-time data relay system, and an environmental data visualization framework. Quantitative end-to-end latency measurements confirm the feasibility of our approach for real-world applications in large architectural spaces. The proposed framework enables new forms of collaboration that transcend spatial constraints, advancing the development of next-generation interactive environments.

    @inproceedings{masubuchi2025digitaltwin,
      author = {Masubuchi, Yusuke and Hiraki, Takefumi and Hiroi, Yuichi and Ibara, Masanori and Matsutani, Kazuki and Zaizen, Megumi and Morita, Junya},
      title = {Development of Digital Twin Environment through Integration of Commercial Metaverse Platform and IoT Sensors of Smart Building},
      booktitle = {2025 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)},
      year = {2025},
      volume = {},
      number = {},
      pages = {852-855},
      keywords = {Digital Twin;Metaverse Integration;Smart Building;IoT Sensors},
      month = mar,
      doi = {10.1109/VRW66409.2025.00173}
    }

18. SA Poster

    

    CP14 Tactile Data Recording System for Clothing with Motion-Controlled Robotic Sliding

    Michikuni Eguchi, Takekazu Kitagishi, Yuichi Hiroi, and Takefumi Hiraki

    In Proceedings of the SIGGRAPH Asia 2025 Posters, Hong Kong, Dec 2025

    Abs Bib DOI arXiv

    The tactile sensation of clothing is critical to wearer comfort. To reveal physical properties that make clothing comfortable, systematic collection of tactile data during sliding motion is required. Recent studies demonstrate that tactile data obtained through stroking with systematically varied speeds and directions encode material properties more accurately. However, prior methods are optimized for small samples and inadequately address non-destructive recording of intact garments, limiting their scalability to large clothing databases where non-destructive measurement is essential. We propose a robotic arm-based system for collecting tactile data from clothing. The system performs stroking measurements with a simulated fingertip while precisely controlling speeds and directions without cutting clothing, thus enabling the creation of motion-labeled, multimodal tactile databases. Machine learning evaluation showed that including motion-related parameters (velocity and direction) improved identification accuracy for audio and/or acceleration, demonstrating the efficacy of motion-related labels for characterizing clothing tactile sensation.

    @inproceedings{eguchi2025tactile,
      pages = {1--3},
      author = {Eguchi, Michikuni and Kitagishi, Takekazu and Hiroi, Yuichi and Hiraki, Takefumi},
      title = {Tactile Data Recording System for Clothing with Motion-Controlled Robotic Sliding},
      year = {2025},
      isbn = {9798400721342},
      publisher = {Association for Computing Machinery},
      address = {New York, NY, USA},
      url = {https://doi.org/10.1145/3757374.3771452},
      doi = {10.1145/3757374.3771452},
      booktitle = {Proceedings of the SIGGRAPH Asia 2025 Posters},
      articleno = {31},
      numpages = {3},
      keywords = {Haptics, Tactile Sensation, Textiles, Clothes, Machine Learning},
      location = {Hong Kong},
      series = {SA Posters '25},
      month = dec,
    }

19. ISMAR Poster

    

    CP13 RAPHO: A Rapid Occluding Photochromic Occlusion Capable See-Through Display

    Chun Wei Ooi, Yuichi Hiroi, Takefumi Hiraki, and Yuta Itoh

    In 2025 IEEE International Symposium on Mixed and Augmented Reality Abstracts and Workshops (ISMAR 2025 Poster), Oct 2025

    Abs Bib DOI PDF

    Occlusion capability (OC) has been one of the important topics in designing the next-generation optical see-through near-eye displays (OST NED). OC broadens OST-NED applications including outdoor ones and improves virtual objects’ spatial perception and their perceived realism. However, conventional OC with optoelectronic masks adds bulkiness and visual artifacts in the real-world view. As an emerging direction, photochromic-based occlusion (PO) has the potential to address these issues, yet current implementations have a critical limitation in their response time. In this paper, we provide a real-time PO-OST NED based on fast-response photochromic materials, bringing it one step closer to realizing a lightweight, high transparency, and visual artifact-free occlusion solution.

    @inproceedings{ooi2025rapho,
      author = {Ooi, Chun Wei and Hiroi, Yuichi and Hiraki, Takefumi and Itoh, Yuta},
      title = {RAPHO: A Rapid Occluding Photochromic Occlusion Capable See-Through Display},
      booktitle = {2025 IEEE International Symposium on Mixed and Augmented Reality Abstracts and Workshops (ISMAR 2025 Poster)},
      year = {2025},
      volume = {},
      number = {},
      pages = {648--649},
      month = oct,
      doi = {10.1109/ISMAR-Adjunct68609.2025.00133}
    }

20. WHC WiP

    CP12 Cluster Haptic Texture Database: Haptic Texture Database with Controlled Sliding-Contact Interactions

    Michikuni Eguchi, Yuichi Hiroi, Tomohiro Hayase, and Takefumi Hiraki

    In Adjunct Proceedings of the 2025 IEEE World Haptics Conference (WHC), Jul 2025

    Abs Bib arXiv PDF Code HTML

    Haptic sciences and technologies benefit greatly from comprehensive datasets that capture tactile stimuli under controlled, systematic conditions. However, existing haptic databases collect data through uncontrolled exploration, which hinders the systematic analysis of how motion parameters (e.g., motion direction and velocity) influence tactile perception. This paper introduces Cluster Haptic Texture Database, a multimodal dataset recorded using a 3-axis machine with an artificial finger to precisely control sliding velocity and direction. The dataset encompasses 118 textured surfaces across 9 material categories, with recordings at 5 velocity levels (20-60 mm/s) and 8 directions. Each surface was tested under 160 conditions, yielding 18,880 synchronized recordings of audio, acceleration, force, position, and visual data. Validation using convolutional neural networks demonstrates classification accuracies of 96% for texture recognition, 88.76% for velocity estimation, and 78.79% for direction estimation, confirming the dataset’s utility for machine learning applications. This resource enables research in haptic rendering, texture recognition algorithms, and human tactile perception mechanisms, supporting the development of realistic haptic interfaces for virtual reality systems and robotic applications.

    @inproceedings{eguchi2025whc_wip,
      pages = {WP2.09:1--2},
      title = {Cluster Haptic Texture Database: Haptic Texture Database with Controlled Sliding-Contact Interactions},
      author = {Eguchi, Michikuni and Hiroi, Yuichi and Hayase, Tomohiro and Hiraki, Takefumi},
      booktitle = {Adjunct Proceedings of the 2025 IEEE World Haptics Conference (WHC)},
      year = {2025},
      organization = {IEEE},
      month = jul,
    }

21. IEEE VR Poster

    CP11 An implementation of MagicCraft: Generating Interactive 3D Objects and Their Behaviors from Text for Commercial Metaverse Platforms

    Ryutaro Kurai, Takefumi Hiraki, Yuichi Hiroi, Yutaro Hirao, Monica Perusquia-Hernandez, Hideki Uchiyama, and Kiyoshi Kiyokawa

    In 2025 IEEE Conference on Virtual Reality and 3D User Interfaces Adjunct 2025, Mar 2025

    Abs Bib DOI PDF

    Metaverse platforms are rapidly evolving to provide immersive spaces. However, the generation of dynamic and interactive 3D objects remains a challenge due to the need for advanced 3D modeling and programming skills. We present MagicCraft, a system that generates functional 3D objects from natural language prompts. MagicCraft uses generative AI models to manage the entire content creation pipeline: converting user text descriptions into images, transforming images into 3D models, predicting object behavior, and assigning necessary attributes and scripts. It also provides an interactive interface for users to refine generated objects by adjusting features like orientation, scale, seating positions, and grip points.

    @inproceedings{kurai2025magiccraft-poster,
      author = {Kurai, Ryutaro and Hiraki, Takefumi and Hiroi, Yuichi and Hirao, Yutaro and Perusquia-Hernandez, Monica and Uchiyama, Hideki and Kiyokawa, Kiyoshi},
      title = {An implementation of MagicCraft: Generating Interactive 3D Objects and Their Behaviors from Text for Commercial Metaverse Platforms},
      booktitle = {2025 IEEE Conference on Virtual Reality and 3D User Interfaces Adjunct 2025},
      year = {2025},
      volume = {},
      number = {},
      pages = {1284-1285},
      keywords = {Solid modeling;Three-dimensional displays;Metaverse;Generative AI;Natural languages;Refining;Pipelines;Programming;Predictive models;User interfaces;Metaverse;3D Object Generation;Generative AI;AI-Assisted Design},
      month = mar,
      doi = {10.1109/VRW66409.2025.00288},
    }

2024

1. IEEE TVCG

   

   J8 HaptoFloater: Visuo-Haptic Augmented Reality by Embedding Imperceptible Color Vibration Signals for Tactile Display Control in a Mid-Air Image

   Rina Nagano, Takahiro Kinoshita, Shingo Hattori, Yuichi Hiroi, Yuta Itoh, and Takefumi Hiraki

   IEEE Transactions on Visualization and Computer Graphics, Nov 2024

   Abs Bib DOI PDF Video

   We propose HaptoFloater, a low-latency mid-air visuo-haptic augmented reality (VHAR) system that utilizes imperceptible color vibrations. When adding tactile stimuli to the visual information of a mid-air image, the user should not perceive the latency between the tactile and visual information. However, conventional tactile presentation methods for mid-air images, based on camera-detected fingertip positioning, introduce latency due to image processing and communication. To mitigate this latency, we use a color vibration technique; humans cannot perceive the vibration when the display alternates between two different color stimuli at a frequency of 25 Hz or higher. In our system, we embed this imperceptible color vibration into the mid-air image formed by a micromirror array plate, and a photodiode on the fingertip device directly detects this color vibration to provide tactile stimulation. Thus, our system allows for the tactile perception of multiple patterns on a mid-air image in 59.5 ms. In addition, we evaluate the visual-haptic delay tolerance on a mid-air display using our VHAR system and a tactile actuator with a single pattern and faster response time. The results of our user study indicate a visual-haptic delay tolerance of 110.6 ms, which is considerably larger than the latency associated with systems using multiple tactile patterns.

   @article{nagano2024haptofloater,
     author = {Nagano, Rina and Kinoshita, Takahiro and Hattori, Shingo and Hiroi, Yuichi and Itoh, Yuta and Hiraki, Takefumi},
     journal = {IEEE Transactions on Visualization and Computer Graphics},
     title = {HaptoFloater: Visuo-Haptic Augmented Reality by Embedding Imperceptible Color Vibration Signals for Tactile Display Control in a Mid-Air Image},
     year = {2024},
     volume = {30},
     number = {11},
     pages = {7463-7472},
     keywords = {Haptic interfaces;Vibrations;Visualization;Image color analysis;Actuators;Delays;Tactile sensors;Visuo-haptic displays;mid-air images;LCD displays;imperceptible color vibration},
     doi = {10.1109/TVCG.2024.3456175},
     issn = {1941-0506},
     month = nov,
   }

2. IEEE TVCG

   

   J7 Towards Co-Operative Beaming Displays: Dual Steering Projectors for Extended Projection Volume and Head Orientation Range

   Hiroto Aoki, Takumi Tochimoto, Yuichi Hiroi, and Yuta Itoh

   IEEE Transactions on Visualization and Computer Graphics, May 2024

   Honorable Mention Best Paper Honorable Mention, IEEE VR 2024.

   Abs Bib DOI PDF

   Existing near-eye displays (NEDs) have trade-offs related to size, weight, computational resources, battery life, and body temperature. A recent paradigm, beaming display, addresses these trade-offs by separating the NED into a steering projector (SP) for image presentation and a passive headset worn by the user. However, the beaming display has issues with the projection area of a single SP and has severe limitations on the head orientation and pose that the user can move. In this study, we distribute dual steering projectors in the scene to extend the head orientation and pose of the beaming display by coordinating the dual projections on a passive headset. For cooperative control of each SP, we define a geometric model of the SPs and propose a calibration and projection control method designed for multiple projectors. We present implementations of the system along with evaluations showing that the precision and delay are 1.8 ∼ 5.7 mm and 14.46 ms, respectively, at a distance of about 1 m from the SPs. From this result, our prototype with multiple SPs can project images in the projection area (20 \textmm \times 30 \textmm) of the passive headset while extending the projectable head orientation. Furthermore, as applications of cooperative control by multiple SPs, we show the possibility of multiple users, improving dynamic range and binocular presentation.

   @article{aoki2024dualbeaming,
     author = {Aoki, Hiroto and Tochimoto, Takumi and Hiroi, Yuichi and Itoh, Yuta},
     journal = {IEEE Transactions on Visualization and Computer Graphics},
     title = {Towards Co-Operative Beaming Displays: Dual Steering Projectors for Extended Projection Volume and Head Orientation Range},
     year = {2024},
     volume = {30},
     number = {5},
     pages = {2309-2318},
     keywords = {Mirrors;Cameras;Headphones;Calibration;Optical imaging;High-speed optical techniques;Head;Near-eye display;Augmented reality;Projectors},
     doi = {10.1109/TVCG.2024.3372118},
     issn = {1941-0506},
     month = may,
   }

3. IEEE TVCG

   

   J6 StainedSweeper: Compact, Variable-Intensity Light-Attenuation Display with Sweeping Tunable Retarders

   Yuichi Hiroi, Takefumi Hiraki, and Yuta Itoh

   IEEE Transactions on Visualization and Computer Graphics, May 2024

   Abs Bib DOI PDF Supp Video

   Light Attenuation Displays (LADs) are a type of Optical See-Through Head-Mounted Display (OST-HMD) that present images by attenuating incoming light with a pixel-wise polarizing color filter. Although LADs can display images in bright environments, there is a trade-off between the number of Spatial Light Modulators (SLMs) and the color gamut and contrast that can be expressed, making it difficult to achieve both high-fidelity image display and a small form factor. To address this problem, we propose StainedSweeper, a LAD that achieves both the wide color gamut and the variable intensity with a single SLM. Our system synchronously controls a pixel-wise Digital Micromirror Device (DMD) and a nonpixel polarizing color filter to pass light when each pixel is the desired color. By sweeping this control at high speed, the human eye perceives images in a time-multiplexed, integrated manner. To achieve this, we develop the OST-HMD design using a reflective Solc filter as a polarized color filter and a color reproduction algorithm based on the optimization of the time-multiplexing matrix for the selected primary color filters. Our proof-of-concept prototype showed that our single SLM design can produce subtractive images with variable contrast and a wider color gamut than conventional LADs.

   @article{hiroi2024stainedsweeper,
     author = {Hiroi, Yuichi and Hiraki, Takefumi and Itoh, Yuta},
     journal = {IEEE Transactions on Visualization and Computer Graphics},
     title = {StainedSweeper: Compact, Variable-Intensity Light-Attenuation Display with Sweeping Tunable Retarders},
     year = {2024},
     volume = {30},
     number = {5},
     pages = {2682-2692},
     keywords = {Image color analysis;Optical filters;Filtering algorithms;Light sources;Optical imaging;Optical reflection;Optical polarization;Light attenuation display;see-through display;augmented reality;time-multiplexing;polarized color filter},
     doi = {10.1109/TVCG.2024.3372058},
     issn = {1941-0506},
     month = may,
   }

4. IEEE ISMAR

   CF9 FactoredSweeper: Optical See-Through Display Integrating Light Attenuation and Addition with a Single Spatial Light Modulator

   Yuichi Hiroi, Takefumi Hiraki, and Yuta Itoh

   In 2024 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), Oct 2024

   Abs Bib DOI PDF Supp Video

   Light Attenuation Displays (LADs), a subset of Optical See-Through Head-Mounted Displays (OST-HMDs), enable image display in bright environments by filtering incident light at the pixel level. Although recent methods have proposed single-DMD light attenuation, they do not consider additive color display and background compensation, limiting their applicability in real-world scenarios. We present FactoredSweeper, a single digital micromirror device (DMD) system that incorporates both light attenuation and addition. By synchronizing the DMD, color filter, and light source, our system generates an additive virtual image, light attenuation, and occlusion through time multiplexing. To produce the target image while compensating for the background, we optimize time-multiplexed binary DMD patterns and LED/color filter schedules using perceptually-driven non-negative matrix factorization. Simulations and prototypes demonstrate that our integrated attenuation-addition single-SLM system achieves superior dynamic range and perceptual image quality compared to conventional occlusion-capable OST-HMDs using grayscale occlusion masks.

   @inproceedings{hiroi2024factored,
     author = {Hiroi, Yuichi and Hiraki, Takefumi and Itoh, Yuta},
     title = {FactoredSweeper: Optical See-Through Display Integrating Light
     Attenuation and Addition with a Single Spatial Light Modulator},
     booktitle = {2024 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)},
     year = {2024},
     volume = {},
     number = {},
     pages = {61--70},
     keywords = {Light attenuation display; see-through display; augmented reality, time-multiplexing; perceptual-driven matrix factorization},
     month = oct,
     doi = {10.1109/ISMAR62088.2024.00020},
   }

5. IEEE VR Workshop

   

   CW2 Design and Implementation of Agent APIs for Large-Scale Social VR Platforms

   Ryutaro Kurai, Takefumi Hiraki, Yuichi Hiroi, Yutaro Hirao, Monica Perusquia-Hernandez, Hideaki Uchiyama, and Kiyoshi Kiyokawa

   In 2024 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW) , Mar 2024

   Abs Bib DOI

   Implementing an autonomous agent on a social VR platform where many users share space requires diverse information. In particular, it is required to recognize the distance from other users, their orientation toward each other, the avatar’s pose, and text and voice messages, and to behave accordingly. This paper proposes an API to obtain the above information on “Cluster,” a multi-device social VR platform in operation, and an agent that uses the API. We have implemented this API using a network proxy. The agent using this API can connect to ChatGPT [7] and have a conversation in real time. We measured the latency required for the conversation and confirmed that the response time was about 1 second.

   @inproceedings{kurai2024agentapi,
     author = {Kurai, Ryutaro and Hiraki, Takefumi and Hiroi, Yuichi and Hirao, Yutaro and Perusquia-Hernandez, Monica and Uchiyama, Hideaki and Kiyokawa, Kiyoshi},
     booktitle = { 2024 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW) },
     title = {{ Design and Implementation of Agent APIs for Large-Scale Social VR Platforms }},
     year = {2024},
     volume = {},
     issn = {},
     pages = {584-587},
     keywords = {Three-dimensional displays;Text recognition;Oral communication;Virtual reality;Speech recognition;User interfaces;Software},
     doi = {10.1109/VRW62533.2024.00112},
     url = {https://doi.ieeecomputersociety.org/10.1109/VRW62533.2024.00112},
     publisher = {IEEE Computer Society},
     address = {Los Alamitos, CA, USA},
     month = mar,
   }

6. ISMAR Poster

   

   CP10 Beaming Display Using Thin Holographic Waveguides for Wider Head Orientation Angle Range

   Yuta Itoh, Tomoya Nakamura, Yuichi Hiroi, and Kaan Aksit

   In 2024 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), Oct 2024

   Abs Bib DOI PDF

   Augmented Reality (AR) glasses face fundamental challenges related to technical trade-offs. Emerging Beaming Displays (BDs) offer a compelling solution by separating the active and passive components. However, existing BD-based AR glasses have yet to achieve a thin and lightweight design with wide incident projection angles. This work proposes an eyepiece for BDs, including a holographic waveguide with tailored in- and out-coupling gratings. The proposed design aims to achieve a millimeter-thin form factor with a wide tolerance for incident angles, thus overcoming the limitations of existing designs. We have constructed proof-of-concept passive AR glasses prototypes, all approximately 2mm thick, including one in the form of conventional eyeglasses, and demonstrated an acceptable lateral angle of incidence of up to 90 degrees.

   @inproceedings{itoh2024thinholographic,
     author = {Itoh, Yuta and Nakamura, Tomoya and Hiroi, Yuichi and Aksit, Kaan},
     title = {Beaming Display Using Thin Holographic Waveguides for Wider Head Orientation Angle Range},
     booktitle = {2024 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct)},
     year = {2024},
     volume = {},
     number = {},
     pages = {475-476},
     keywords = {Beaming display; Augmented reality; Near-eye display; waveguide; HOEs},
     doi = {10.1109/ISMAR-Adjunct64951.2024.00138},
     month = oct,
   }

7. SIGGRAPH Poster

   

   CP9 Measurement of the Imperceptible Threshold for Color Vibration Pairs Selected by using MacAdam Ellipse

   Shingo Hattori, Yuichi Hiroi, and Takefumi Hiraki

   In ACM SIGGRAPH 2024 Posters, Denver, CO, USA, Jul 2024

   Award 1st Place, Student Research Competition (Undergraduate), SIGGRAPH 2024.

   Abs Bib DOI arXiv

   We propose an efficient method for searching for color vibration pairs that are imperceptible to the human eye based on the MacAdam ellipse, an experimentally determined color-difference range that is indistinguishable to the human eye. We created color pairs by selecting eight colors within the sRGB color space specified by the ellipse, and conducted experiments to confirm the threshold of the amplitude of color vibration amplitude at which flicker becomes imperceptible to the human eye. The experimental results indicate a general guideline for acceptable amplitudes for pair selection.

   @inproceedings{hattori2024macadam,
     pages = {1--2},
     month = jul,
     author = {Hattori, Shingo and Hiroi, Yuichi and Hiraki, Takefumi},
     title = {Measurement of the Imperceptible Threshold for Color Vibration Pairs Selected by using MacAdam Ellipse},
     year = {2024},
     isbn = {9798400705168},
     publisher = {Association for Computing Machinery},
     address = {New York, NY, USA},
     url = {https://doi.org/10.1145/3641234.3671041},
     doi = {10.1145/3641234.3671041},
     booktitle = {ACM SIGGRAPH 2024 Posters},
     articleno = {68},
     numpages = {2},
     keywords = {MacAdam ellipse, color perception, imperceptible color vibration},
     location = {Denver, CO, USA},
     series = {SIGGRAPH '24},
   }

8. ISMAR Demo

   CD3 MetaGadget: IoT Framework for Event-Triggered Integration of User-Developed Devices into Commercial Metaverse Platforms

   Ryutaro Kurai, Yuichi Hiroi, and Takefumi Hiraki

   In 2024 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) , Oct 2024

   Abs Bib DOI Video Code

   This demonstration introduces MetaGadget, an IoT framework designed to integrate user-developed devices into commercial metaverse platforms. Synchronizing virtual reality (VR) environments with physical devices has traditionally required a constant connection to VR clients, limiting flexibility and resource efficiency. MetaGadget overcomes these limitations by configuring user-developed devices as IoT units with server capabilities, supporting communication via HTTP protocols within the commercial metaverse platform, Cluster. This approach enables event-triggered device control without the need for persistent connections from metaverse clients. Through the demonstration, users will experience event-triggered interaction between VR and physical devices, as well as real-world device control through the VR space by multiple people. Our framework is expected to reduce technical barriers to integrating VR spaces and custom devices, contribute to interoperability, and increase resource efficiency through event-triggered connections.

   @inproceedings{kurai2024metagadget,
     author = {Kurai, Ryutaro and Hiroi, Yuichi and Hiraki, Takefumi},
     booktitle = { 2024 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) },
     title = {{ MetaGadget: IoT Framework for Event-Triggered Integration of User-Developed Devices into Commercial Metaverse Platforms }},
     year = {2024},
     volume = {},
     issn = {},
     pages = {632-633},
     keywords = {Virtual Reality; Metaverse Platform; User-Developed Devices},
     month = oct,
     doi = {10.1109/ISMAR-Adjunct64951.2024.00185},
   }

9. arXiv

   

   P1 PanoTree: Autonomous Photo-Spot Explorer in Virtual Reality Scenes

   Tomohiro Hayase, Sacha Braun, Hikari Yanagawa, Itsuki Orito, and Yuichi Hiroi

   May 2024

   Abs Bib DOI arXiv Video Code Website

   Social VR platforms enable social, economic, and creative activities by allowing users to create and share their own virtual spaces. In social VR, photography within a VR scene is an important indicator of visitors’ activities. Although automatic identification of photo spots within a VR scene can facilitate the process of creating a VR scene and enhance the visitor experience, there are challenges in quantitatively evaluating photos taken in the VR scene and efficiently exploring the large VR scene. We propose PanoTree, an automated photo-spot explorer in VR scenes. To assess the aesthetics of images captured in VR scenes, a deep scoring network is trained on a large dataset of photos collected by a social VR platform to determine whether humans are likely to take similar photos. Furthermore, we propose a Hierarchical Optimistic Optimization (HOO)-based search algorithm to efficiently explore 3D VR spaces with the reward from the scoring network. Our user study shows that the scoring network achieves human-level performance in distinguishing randomly taken images from those taken by humans. In addition, we show applications using the explored photo spots, such as automatic thumbnail generation, support for VR world creation, and visitor flow planning within a VR scene.

   @misc{hayase2024panotree,
     doi = {10.48550/arXiv.2405.17136},
     month = may,
     title = {PanoTree: Autonomous Photo-Spot Explorer in Virtual Reality Scenes},
     author = {Hayase, Tomohiro and Braun, Sacha and Yanagawa, Hikari and Orito, Itsuki and Hiroi, Yuichi},
     year = {2024},
     eprint = {2405.17136},
     archiveprefix = {arXiv},
     primaryclass = {cs.CV},
   }

2023

1. IEEE TVCG

   

   J5 Low-Latency Beaming Display: Implementation of Wearable, 133 μs Motion-to-Photon Latency Near-Eye Display

   Yuichi Hiroi , Akira Watanabe, Yuri Mikawa, and Yuta Itoh

   IEEE Transactions on Visualization and Computer Graphics, Nov 2023

   Abs Bib DOI PDF Video Slides

   This paper presents a low-latency Beaming Display system with a 133 μ\mathrms motion-to-photon (M2P) latency, the delay from head motion to the corresponding image motion. The Beaming Display represents a recent near-eye display paradigm that involves a steerable remote projector and a passive wearable headset. This system aims to overcome typical trade-offs of Optical See-Through Head-Mounted Displays (OST-HMDs), such as weight and computational resources. However, since the Beaming Display projects a small image onto a moving, distant viewpoint, M2P latency significantly affects displacement. To reduce M2P latency, we propose a low-latency Beaming Display system that can be modularized without relying on expensive high-speed devices. In our system, a 2D position sensor, which is placed coaxially on the projector, detects the light from the IR-LED on the headset and generates a differential signal for tracking. An analog closed-loop control of the steering mirror based on this signal continuously projects images onto the headset. We have implemented a proof-of-concept prototype, evaluated the latency and the augmented reality experience through a user-perspective camera, and discussed the limitations and potential improvements of the prototype.

   @article{hiroi2023lowlatency,
     author = {Hiroi, Yuichi and Watanabe, Akira and Mikawa, Yuri and Itoh, Yuta},
     journal = {IEEE Transactions on Visualization and Computer Graphics},
     title = {Low-Latency Beaming Display: Implementation of Wearable, 133 μs Motion-to-Photon Latency Near-Eye Display},
     year = {2023},
     volume = {29},
     number = {11},
     pages = {4761-4771},
     keywords = {Mirrors;Low latency communication;Headphones;Head;Cameras;Tracking;Two dimensional displays;Low-Latency Display;Beaming Display;Motion-to-Photon Latency;Lateral-effect Photodiodes},
     doi = {10.1109/TVCG.2023.3320212},
     issn = {1941-0506},
     month = nov,
   }

2. ACM VRST

   CF8 Retinal Homing Display: Head-Tracking Auto-stereoscopic Retinal Projection Display

   Hiroto Aoki, Yuichi Hiroi, Yuta Itoh, and Jun Rekimoto

   In Proceedings of the 29th ACM Symposium on Virtual Reality Software and Technology, Christchurch, New Zealand, Oct 2023

   Abs Bib DOI PDF

   This paper introduces Retinal Homing Display, which presents focus-free stereoscopic images via retinal projection, thus eliminating the need for the user to wear additional equipment. Traditional 3D displays, typically classified as either naked-eye stereoscopic or wearable, present inherent challenges: the former involves a compromise between resolution and accurate depth perception, while the latter imposes an additional burden on the user. Our proposed display employs optical and mechanical mechanisms to converge projector light at the user’s pupil center, simultaneously tracking eye movements. This lets the user perceive focus-free, high-resolution stereoscopic images without wearable equipment. We implemented a proof-of-concept system utilizing a robotic arm and a Dihedral Corner Reflector Array (DCRA), subsequently evaluating image quality and its eyebox. Finally, we discuss the limitations of the current prototype and outline potential directions for future research.

   @inproceedings{aoki2023retinal,
     pages = {1--10},
     month = oct,
     author = {Aoki, Hiroto and Hiroi, Yuichi and Itoh, Yuta and Rekimoto, Jun},
     title = {Retinal Homing Display: Head-Tracking Auto-stereoscopic Retinal Projection Display},
     year = {2023},
     isbn = {9798400703287},
     publisher = {Association for Computing Machinery},
     address = {New York, NY, USA},
     url = {https://doi.org/10.1145/3611659.3615715},
     doi = {10.1145/3611659.3615715},
     booktitle = {Proceedings of the 29th ACM Symposium on Virtual Reality Software and Technology},
     articleno = {13},
     numpages = {10},
     keywords = {autostereoscopic display, motion-following display, retinal projection},
     location = {Christchurch, New Zealand},
     series = {VRST '23},
   }

3. ACM UIST

   CF7 Telextiles: End-to-end Remote Transmission of Fabric Tactile Sensation

   Takekazu Kitagishi, Yuichi Hiroi, Yuna Watanabe, Yuta Itoh, and Jun Rekimoto

   In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, San Francisco, CA, USA, Oct 2023

   Abs Bib DOI Video Website

   The tactile sensation of textiles is critical in determining the comfort of clothing. For remote use, such as online shopping, users cannot physically touch the textile of clothes, making it difficult to evaluate its tactile sensation. Tactile sensing and actuation devices are required to transmit the tactile sensation of textiles. The sensing device needs to recognize different garments, even with hand-held sensors. In addition, the existing actuation device can only present a limited number of known patterns and cannot transmit unknown tactile sensations of textiles. To address these issues, we propose Telextiles, an interface that can remotely transmit tactile sensations of textiles by creating a latent space that reflects the proximity of textiles through contrastive self-supervised learning. We confirm that textiles with similar tactile features are located close to each other in the latent space through a two-dimensional plot. We then compress the latent features for known textile samples into the 1D distance and apply the 16 textile samples to the rollers in the order of the distance. The roller is rotated to select the textile with the closest feature if an unknown textile is detected.

   @inproceedings{kitagishi2023telextiles,
     pages = {1--10},
     month = oct,
     author = {Kitagishi, Takekazu and Hiroi, Yuichi and Watanabe, Yuna and Itoh, Yuta and Rekimoto, Jun},
     title = {Telextiles: End-to-end Remote Transmission of Fabric Tactile Sensation},
     year = {2023},
     isbn = {9798400701320},
     publisher = {Association for Computing Machinery},
     address = {New York, NY, USA},
     url = {https://doi.org/10.1145/3586183.3606764},
     doi = {10.1145/3586183.3606764},
     booktitle = {Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology},
     articleno = {67},
     numpages = {10},
     keywords = {Haptic feedback, Machine learning, Passive haptic feedback, Self supervised learning, Tactile Display, Tactile perception, Texture, Texture perception, Texture recognition},
     location = {San Francisco, CA, USA},
     series = {UIST '23},
   }

4. IEEE VR

   CF6 A Compact Photochromic Occlusion Capable See-through Display with Holographic Lenses

   Chun-Wei Ooi, Yuichi Hiroi, and Yuta Itoh

   In 2023 IEEE Conference Virtual Reality and 3D User Interfaces (VR), Mar 2023

   Abs Bib DOI PDF

   Occlusion is a crucial visual element in optical see-through (OST) augmented reality, however, implementing occlusion in OST displays while addressing various design trade-offs is a difficult problem. In contrast to the traditional method of using spatial light modulators (SLMs) for the occlusion mask, using photochromic materials as occlusion masks can effectively eliminate diffraction artifacts in see-through views due to the lack of electronic pixels, thus providing superior see-through image quality. However, this design requires UV illumination to activate the photochromic mate-rial, which traditionally requires multiple SLMs, resulting in a larger form factor for the system. This paper presents a compact photochromic occlusion-capable OST design using multilayer, wavelength-dependent holographic optical lenses (HOLs). Our approach employs a single digital mi-cromirror display (DMD) to form both the occlusion mask with UV light and a virtual image with visible light in a time-multiplexed man-ner. We demonstrate our proof-of-concept system on a bench-top setup and assess the appearance and contrasts of the displayed image. We also suggest potential improvements for current prototypes to encourage the community to explore this occlusion approach.

   @inproceedings{ooi2023photochromic,
     author = {Ooi, Chun-Wei and Hiroi, Yuichi and Itoh, Yuta},
     booktitle = {2023 IEEE Conference Virtual Reality and 3D User Interfaces (VR)},
     title = {A Compact Photochromic Occlusion Capable See-through Display with Holographic Lenses},
     year = {2023},
     volume = {},
     number = {},
     pages = {237-242},
     keywords = {Visualization;Optical diffraction;Three-dimensional displays;Optical design;Prototypes;Holography;User interfaces;Human-centered computing-Human computer interaction (HCI)-Interaction paradigms-Mixed / augmented reality;Human-centered computing-Communication hardware;interfaces and storage-Displays and imagers},
     doi = {10.1109/VR55154.2023.00039},
     issn = {2642-5254},
     month = mar,
   }

5. ISMAR Poster

   CP8 Dual Beaming Display for Extended Head Orientation and Projection Volume

   Takumi Tochimoto, Yuichi Hiroi, and Yuta Itoh

   In 2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), Oct 2023

   Abs Bib DOI

   Existing near-eye displays (NEDs) come with various tradeoffs, including size, weight, and battery life. The new "beaming display" model attempts to address these by splitting the NED into a steering projector (SP) and a passive headset. However, it has limitations in terms of projection area and user head movement. In this research, we utilize two steering projectors to expand the beaming display’s capabilities. We introduce a geometric model and a control method specifically for multiple projectors. Tests indicate an accuracy of 3-19mm and a delay of 14.46 ms from a 1m distance to the SPs, allowing projection onto a 20mm x 30mm area of the headset. This method not only expands head movement possibilities but also shows potential for multiple users and improved dynamic range and binocular presentation.

   @inproceedings{tochimoto2023dualbeaming,
     author = {Tochimoto, Takumi and Hiroi, Yuichi and Itoh, Yuta},
     booktitle = {2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct)},
     title = {Dual Beaming Display for Extended Head Orientation and Projection Volume},
     year = {2023},
     volume = {},
     number = {},
     pages = {377-378},
     keywords = {Headphones;Design methodology;Geometric modeling;Display systems;Focusing;Dynamic range;Delays;Human-centered computing;Visualization;Visualization techniques;Treemaps; Human-centered computing;Visualization design and evaluation methods},
     doi = {10.1109/ISMAR-Adjunct60411.2023.00081},
     issn = {2771-1110},
     month = oct,
   }

6. AHs Poster

   

   CP7 Brain-Computer Interface using Directional Auditory Perception

   Yuto Koike, Yuichi Hiroi, Yuta Itoh, and Jun Rekimoto

   In Proceedings of the Augmented Humans International Conference 2023, Glasgow, United Kingdom, Mar 2023

   Honorable Mention Best Poster Honorable Mention, Augmented Humans 2023.

   Abs Bib DOI

   We investigate the potential of brain-computer interface (BCI) using electroencephalogram (EEG) induced by listening (or recalling) auditory stimuli of different directions. In the initial attempt, we apply a time series classification model based on deep learning to the EEG to demonstrate whether each EEG can be classified by recognizing binary (left or right) auditory directions. The results showed high classification accuracy when trained and tested on the same users. Discussion is provided to further explore this topic.

   @inproceedings{koike2023braincomputer,
     month = mar,
     author = {Koike, Yuto and Hiroi, Yuichi and Itoh, Yuta and Rekimoto, Jun},
     title = {Brain-Computer Interface using Directional Auditory Perception},
     year = {2023},
     isbn = {9781450399845},
     publisher = {Association for Computing Machinery},
     address = {New York, NY, USA},
     url = {https://doi.org/10.1145/3582700.3583713},
     doi = {10.1145/3582700.3583713},
     booktitle = {Proceedings of the Augmented Humans International Conference 2023},
     pages = {342–345},
     numpages = {4},
     keywords = {BCI, BMI, Brain-computer Interface, Brain-machine Interface, Directional Auditory Sensation, EEG, Electroencephalogram, Electroencephalography},
     location = {Glasgow, United Kingdom},
     series = {AHs '23},
   }

7. ACM UIST Demo

   CD2 Telextiles: End-to-end Remote Transmission of Fabric Tactile Sensation

   Takekazu Kitagishi, Yuichi Hiroi, Yuna Watanabe, Yuta Itoh, and Jun Rekimoto

   In Adjunct Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, San Francisco, USA, Oct 2023

   Technology demonstration

   Bib DOI

   @inproceedings{kitagishi2023telextile-demo,
     publisher = {Association for Computing Machinery},
     pages = {1--3},
     doi = {10.1145/3586182.3615770},
     author = {Kitagishi, Takekazu and Hiroi, Yuichi and Watanabe, Yuna and Itoh, Yuta and Rekimoto, Jun},
     title = {Telextiles: End-to-end Remote Transmission of Fabric Tactile Sensation},
     booktitle = {Adjunct Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology},
     year = {2023},
     month = oct,
     location = {San Francisco, USA},
     note = {Technology demonstration},
   }

2022

1. Opt. Exp.

   

   J4 Neural distortion fields for spatial calibration of wide field-of-view near-eye displays

   Yuichi Hiroi, Kiyosato Someya, and Yuta Itoh

   Opt. Express, Oct 2022

   Award Selected as one of Japan’s leading optics research achievements of 2022, Optical Society of Japan (2023).

   Abs Bib DOI Video

   We propose a spatial calibration method for wide field-of-view (FoV) near-eye displays (NEDs) with complex image distortions. Image distortions in NEDs can destroy the reality of the virtual object and cause sickness. To achieve distortion-free images in NEDs, it is necessary to establish a pixel-by-pixel correspondence between the viewpoint and the displayed image. Designing compact and wide-FoV NEDs requires complex optical designs. In such designs, the displayed images are subject to gaze-contingent, non-linear geometric distortions, which explicit geometric models can be difficult to represent or computationally intensive to optimize. To solve these problems, we propose neural distortion field (NDF), a fully-connected deep neural network that implicitly represents display surfaces complexly distorted in spaces. NDF takes spatial position and gaze direction as input and outputs the display pixel coordinate and its intensity as perceived in the input gaze direction. We synthesize the distortion map from a novel viewpoint by querying points on the ray from the viewpoint and computing a weighted sum to project output display coordinates into an image. Experiments showed that NDF calibrates an augmented reality NED with 90° FoV with about 3.23 pixel (5.8 arcmin) median error using only 8 training viewpoints. Additionally, we confirmed that NDF calibrates more accurately than the non-linear polynomial fitting, especially around the center of the FoV.

   @article{hiroi2022neural,
     author = {Hiroi, Yuichi and Someya, Kiyosato and Itoh, Yuta},
     journal = {Opt. Express},
     keywords = {Image metrics; Lens design; Near eye displays; Optical aberration; Optical systems; Systems design},
     number = {22},
     pages = {40628--40644},
     publisher = {Optica Publishing Group},
     title = {Neural distortion fields for spatial calibration of wide field-of-view near-eye displays},
     volume = {30},
     month = oct,
     year = {2022},
     url = {https://opg.optica.org/oe/abstract.cfm?URI=oe-30-22-40628},
     doi = {10.1364/OE.472288},
   }

2. ACM VRST

   CF5 NeARportation: A Remote Real-time Neural Rendering Framework

   Yuichi Hiroi, Yuta Itoh, and Jun Rekimoto

   In Proceedings of the 28th ACM Symposium on Virtual Reality Software and Technology, Tsukuba, Japan, Nov 2022

   Abs Bib DOI PDF Video

   While presenting a photorealistic appearance plays a major role in immersion in Augmented Virtuality environment, displaying that of real objects remains a challenge. Recent developments in photogrammetry have facilitated the incorporation of real objects into virtual space. However, reproducing complex appearances, such as subsurface scattering and transparency, still requires a dedicated environment for measurement and possesses a trade-off between rendering quality and frame rate. Our NeARportation framework combines server–client bidirectional communication and neural rendering to resolve these trade-offs. Neural rendering on the server receives the client’s head posture and generates a novel-view image with realistic appearance reproduction that is streamed onto the client’s display. By applying our framework to a stereoscopic display, we confirm that it can display a high-fidelity appearance on full-HD stereo videos at 35-40 frames per second (fps) according to the user’s head motion.

   @inproceedings{hiroi2022nearportation,
     pages = {1--5},
     month = nov,
     author = {Hiroi, Yuichi and Itoh, Yuta and Rekimoto, Jun},
     title = {NeARportation: A Remote Real-time Neural Rendering Framework},
     year = {2022},
     isbn = {9781450398893},
     publisher = {Association for Computing Machinery},
     address = {New York, NY, USA},
     url = {https://doi.org/10.1145/3562939.3565616},
     doi = {10.1145/3562939.3565616},
     booktitle = {Proceedings of the 28th ACM Symposium on Virtual Reality Software and Technology},
     articleno = {23},
     numpages = {5},
     keywords = {remote rendering, real-time rendering, neural rendering, augmented virtuality, appearance reproduction},
     location = {Tsukuba, Japan},
     series = {VRST '22},
   }

3. IEEE ISMAR

   CF4 Towards Spatial Airflow Interaction: Schlieren Imaging for Augmented Reality

   Zhang Zhibin, Yuichi Hiroi, and Yuta Itoh

   In 2022 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), Oct 2022

   Abs Bib DOI PDF Video

   This work integrates Schlieren Imaging, a unique sensing modality, into Augmented Reality (AR) to explore ways to utilize invisible airflows for AR. Schlieren imaging is an imaging technique that visualizes the flow of fluids, which is normally invisible to the eyes. Theoretically, the technique can calculate the motion, pressure, temperature, and density of the airflow in our physical world. This unique, but less applied modality may expand interaction paradigms in AR and VR. We build a proof-of-concept AR system combined with Schlieren imaging that allows real airflow to affect virtual objects. The results of quantitative analyses show that our system can integrate different types of airflow with pressure values ranging from weak breathing actions to a heat gun up to 10m/s or 0.25m3/min airflow. We also showcase AR use cases including blowing out a virtual candle and a heat gun.

   @inproceedings{zhibin2022airflow,
     author = {Zhibin, Zhang and Hiroi, Yuichi and Itoh, Yuta},
     booktitle = {2022 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)},
     title = {Towards Spatial Airflow Interaction: Schlieren Imaging for Augmented Reality},
     year = {2022},
     volume = {},
     number = {},
     pages = {215-223},
     keywords = {Heating systems;Temperature sensors;Photography;Visualization;Sensitivity;Statistical analysis;User interfaces;Schlieren imaging;augmented reality;spatial airflow interaction},
     doi = {10.1109/ISMAR55827.2022.00036},
     issn = {1554-7868},
     month = oct,
   }

2021

1. Opt. Exp.

   

   J3 Focal surface occlusion

   Yuichi Hiroi, Takumi Kaminokado, Shunsuke Ono, and Yuta Itoh

   Opt. Express, Oct 2021

   Abs Bib DOI

   This paper proposes focal surface occlusion to provide focal cues of occlusion masks for multiple virtual objects at continuous depths in an occlusion-capable optical see-through head-mounted display. A phase-only spatial light modulator (PSLM) that acts as a dynamic free-form lens is used to conform the focal surface of an occlusion mask to the geometry of the virtual scene. To reproduce multiple and continuous focal blurs while reducing the distortion of the see-through view, an optical design based on afocal optics and edge-based optimization to exploit a property of the occlusion mask is established. The prototype with the PSLM and transmissive liquid crystal display can reproduce the focus blur of occluded objects at multiple and continuous depths with a field of view of 14.6°.

   @article{hiroi2021focalsurface,
     author = {Hiroi, Yuichi and Kaminokado, Takumi and Ono, Shunsuke and Itoh, Yuta},
     journal = {Opt. Express},
     keywords = {Head mounted displays; Microlens arrays; Near eye displays; Optical components; Optical systems; Spatial light modulators},
     number = {22},
     pages = {36581--36597},
     publisher = {Optica Publishing Group},
     title = {Focal surface occlusion},
     volume = {29},
     month = oct,
     year = {2021},
     url = {https://opg.optica.org/oe/abstract.cfm?URI=oe-29-22-36581},
     doi = {10.1364/OE.440024},
   }

2. Augmented Humans

   CF3 CircadianVisor: Image Presentation with an Optical See-Through Display in Consideration of Circadian Illuminance

   Takumi Tochimoto, Yuichi Hiroi, and Yuta Itoh

   In Proceedings of the Augmented Humans International Conference 2021, Rovaniemi, Finland, Feb 2021

   Abs Bib DOI PDF

   In modern society, the impact of nighttime artificial lighting on the human sleep/wake cycle (the circadian rhythm), has long been an important issue. In augmented reality, such health hazards should be prevented if we are to become a society that wears optical see-through head-mounted displays (OST-HMDs) on a daily basis. We present CircadianVisor, an OST display system that controls circadian performance. Our system combines an OST-HMD with a liquid crystal (LC) shutter and a spectrometer to control the circadian illuminance (CIL, biolux) of light incident on the user’s eyes. To prevent the CIL at the eyes from exceeding the threshold, correct for the displayed image based on RGB values, and adjust the transmittance of the LC visor to pass through the environment light based on spectral measurements, we build a proof-of-concept system to evaluate the feasibility of the system’s CIL control and test it with a spectrometer installed at the user’s viewpoint. The evaluation shows that the CIL at the user’s viewpoint can be kept below the threshold.

   @inproceedings{tochimoto2021circadian,
     month = feb,
     author = {Tochimoto, Takumi and Hiroi, Yuichi and Itoh, Yuta},
     title = {CircadianVisor: Image Presentation with an Optical See-Through Display in Consideration of Circadian Illuminance},
     year = {2021},
     isbn = {9781450384285},
     publisher = {Association for Computing Machinery},
     address = {New York, NY, USA},
     url = {https://doi.org/10.1145/3458709.3458938},
     doi = {10.1145/3458709.3458938},
     booktitle = {Proceedings of the Augmented Humans International Conference 2021},
     pages = {66–76},
     numpages = {11},
     keywords = {Augmented reality, Blue-light-blocking glasses, Circadian rhythm, Head-mounted displays},
     location = {Rovaniemi, Finland},
     series = {AHs '21},
   }

3. ISMAR Poster

   

   CP6 Focus-Aware Retinal Projection-based Near-Eye Display

   Mayu Kaneko, Yuichi Hiroi, and Yuta Itoh

   In 2021 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), Oct 2021

   Abs Bib DOI PDF Video

   The primary challenge in optical see-through near-eye displays lies in providing correct optical focus cues. Established approaches such as varifocal or light field display essentially sacrifice temporal or spatial resolution of the resulting 3D images. This paper explores a new direction to address the trade-off by combining a retinal projection display (RPD) with ocular wavefront sensing (OWS). Our core idea is to display a depth of field-simulated image on an RPD to produce visually consistent optical focus cues while maintaining the spatial and temporal resolution of the image. To obtain the current accommodation of the eye, we integrate OWS. We demonstrate that our proof-of-concept system successfully renders virtual contents with proper depth cues while covering the eye accommodation range from 28.5 cm (3.5 D) to infinity (0.0 D).

   @inproceedings{kaneko2021focusaware,
     author = {Kaneko, Mayu and Hiroi, Yuichi and Itoh, Yuta},
     booktitle = {2021 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct)},
     title = {Focus-Aware Retinal Projection-based Near-Eye Display},
     year = {2021},
     volume = {},
     number = {},
     pages = {207-208},
     keywords = {Three-dimensional displays;Optical variables measurement;Optical imaging;Retina;Rendering (computer graphics);Adaptive optics;Light fields;Augmented reality;accommodation sensing;retinal projection;near-eye display},
     doi = {10.1109/ISMAR-Adjunct54149.2021.00049},
     issn = {},
     month = oct,
   }

2020

1. IEEE TVCG

   

   J2 StainedView: Variable-Intensity Light-Attenuation Display with Cascaded Spatial Color Filtering for Improved Color Fidelity

   Takumi Kaminokado, Yuichi Hiroi, and Yuta Itoh

   IEEE Transactions on Visualization and Computer Graphics, Dec 2020

   Abs Bib DOI PDF

   We present StainedView, an optical see-through display that spatially filters the spectral distribution of light to form an image with improved color fidelity. Existing light-attenuation displays have limited color fidelity and contrast, resulting in a degraded appearance of virtual images. To use these displays to present virtual images that are more consistent with the real world, we require three things: intensity modulation of incoming light, spatial color filtering with narrower bandwidth, and appropriate light modulation for incoming light with an arbitrary spectral distribution. In StainedView, we address the three requirements by cascading two phase-only spatial light modulators (PSLMs), a digital micromirror device, and polarization optics to control both light intensity and spectrum distribution. We show that our design has a 1.8 times wider color gamut fidelity (75.8% fulfillment of sRGB color space) compared to the existing single-PSLM approach (41.4%) under a reference white light. We demonstrated the design with a proof-of-concept display system. We further introduce our optics design and pixel-selection algorithm for the given light input, evaluate the spatial color filter, and discuss the limitation of the current prototype.

   @article{kamino2020stainedview,
     author = {Kaminokado, Takumi and Hiroi, Yuichi and Itoh, Yuta},
     journal = {IEEE Transactions on Visualization and Computer Graphics},
     title = {StainedView: Variable-Intensity Light-Attenuation Display with Cascaded Spatial Color Filtering for Improved Color Fidelity},
     year = {2020},
     volume = {26},
     number = {12},
     pages = {3576-3586},
     keywords = {Image color analysis;Optical imaging;Optical polarization;Adaptive optics;Optical distortion;Optical modulation;Nonlinear optics;Light attenuation display;phase modulation;see-through display;vision augmentation;augmented reality},
     doi = {10.1109/TVCG.2020.3023569},
     issn = {1941-0506},
     month = dec,
   }

2. Augmented Humans

   CF2 DehazeGlasses: Optical Dehazing with an Occlusion Capable See-Through Display

   Yuichi Hiroi, Takumi Kaminokado, Atsushi Mori, and Yuta Itoh

   In Proceedings of the Augmented Humans International Conference, Kaiserslautern, Germany, Mar 2020

   Abs Bib DOI PDF

   We present DehazeGlasses, a see-through visual haze removal system that optically dehazes the user’s field of vision. Human vision suffers from a degraded view due to aspects of the scene environment, such as haze. Such degradation may interfere with our behavior or judgement in daily tasks. We focus on hazy scenes as one common degradation source, which whitens the view due to certain atmospheric conditions. Unlike typical computer vision systems that process recorded images, we aim to realize a see-through glasses system that can optically manipulate our field of view to dehaze the perceived scene. Our system selectively modulates the intensity of the light entering the eyes via occlusion-capable optical see-through head-mounted displays (OST-HMD). We built a proof-of-concept system to evaluate the feasibility of our haze removal method by combining a digital micromirror device (DMD) and an OST-HMD, and tested it with a user-perspective viewpoint camera. A quantitative evaluation with 80 scenes from a haze removal dataset shows that our system realizes a dehazed view that is significantly closer to the ground truth scene compared to the native view under a perceptual image similarity metric. This evaluation shows that our system achieves perceptually natural haze removal while maintaining the see-through view of actual scenes.

   @inproceedings{hiroi2020dehaze,
     pages = {1--11},
     month = mar,
     author = {Hiroi, Yuichi and Kaminokado, Takumi and Mori, Atsushi and Itoh, Yuta},
     title = {DehazeGlasses: Optical Dehazing with an Occlusion Capable See-Through Display},
     year = {2020},
     isbn = {9781450376037},
     publisher = {Association for Computing Machinery},
     address = {New York, NY, USA},
     url = {https://doi.org/10.1145/3384657.3384781},
     doi = {10.1145/3384657.3384781},
     booktitle = {Proceedings of the Augmented Humans International Conference},
     articleno = {3},
     numpages = {11},
     keywords = {Vision Augmentation, Occlusion-Capable HMD, Head-Mounted Displays, Haze Removal, Augmented Reality},
     location = {Kaiserslautern, Germany},
     series = {AHs '20},
   }

3. CVPR Workshop

   CW1 DehazeGlasses: Optical Dehazing with an Occlusion Capable See-Through Display

   Yuichi Hiroi, Takumi Kaminokado, Atsushi Mori, and Yuta Itoh

   In 9th IEEE International Workshop on Computational Cameras and Displays (CCD, CVPR Workshops), 2020

   Workshop presentation

   Bib Website

   @inproceedings{hiroi2020dehaze-workshop,
     author = {Hiroi, Yuichi and Kaminokado, Takumi and Mori, Atsushi and Itoh, Yuta},
     title = {DehazeGlasses: Optical Dehazing with an Occlusion Capable See-Through Display},
     booktitle = {9th IEEE International Workshop on Computational Cameras and Displays (CCD, CVPR Workshops)},
     year = {2020},
     note = {Workshop presentation},
   }

4. ISMAR Poster

   

   CP5 Stencil Marker: Designing Partially Transparent Markers for Stacking Augmented Reality Objects

   Xuan Zhang, Jonathan Lundgren, Yoya Mesaki, Yuichi Hiroi, and Yuta Itoh

   In 2020 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), Nov 2020

   Abs Bib DOI PDF Video

   We propose a transparent colored AR marker that allows 3D objects to be stacked in space. Conventional AR markers make it difficult to display multiple objects in the same position in space, or to manipulate the order or rotation of objects. The proposed transparent colored markers are designed to detect the order and rotation direction of each marker in the stack from the observed image, based on mathematical constraints. We describe these constraints to design markers, the implementation to detect its stacking order and rotation of each marker, and a proof-of-concept application Totem Poles. We also discuss the limitations of the current prototype and possible research directions.

   @inproceedings{zhang2020stencil,
     author = {Zhang, Xuan and Lundgren, Jonathan and Mesaki, Yoya and Hiroi, Yuichi and Itoh, Yuta},
     booktitle = {2020 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct)},
     title = {Stencil Marker: Designing Partially Transparent Markers for Stacking Augmented Reality Objects},
     year = {2020},
     volume = {},
     number = {},
     pages = {255-257},
     keywords = {Human computer interaction;Three-dimensional displays;Image color analysis;Stacking;Prototypes;Augmented reality;Human-centered computing;Visualization;Visualization techniques;Treemaps;Human-centered computing;Visualization;Visualization design and evaluation methods},
     doi = {10.1109/ISMAR-Adjunct51615.2020.00073},
     issn = {},
     month = nov,
   }

2019

1. ISMAR Poster

   

   CP4 OSTNet: Calibration Method for Optical See-Through Head-Mounted Displays via Non-Parametric Distortion Map Generation

   Kiyosato Someya, Yuichi Hiroi, Makoto Yamada, and Yuta Itoh

   In 2019 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), Oct 2019

   Abs Bib DOI PDF

   We propose a spatial calibration method for Optical See-Through Head-Mounted Displays (OST-HMDs) having complex optical distortion such as wide field-of-view (FoV) designs. Viewpoint-dependent non-linear optical distortion makes existing spatial calibration methods either impossible to handle or difficult to compensate without intensive computation. To overcome this issue, we propose OSTNet, a non-parametric data-driven calibration method that creates a generative 2D distortion model for a given six-degree-of-freedom viewpoint pose.

   @inproceedings{someya2019ostnet,
     author = {Someya, Kiyosato and Hiroi, Yuichi and Yamada, Makoto and Itoh, Yuta},
     booktitle = {2019 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct)},
     title = {OSTNet: Calibration Method for Optical See-Through Head-Mounted Displays via Non-Parametric Distortion Map Generation},
     year = {2019},
     volume = {},
     number = {},
     pages = {259-260},
     keywords = {Optical distortion;Calibration;Cameras;Nonlinear distortion;Two dimensional displays;Decoding;optical see through head mounted display;calibration;Variational Autoencoder},
     doi = {10.1109/ISMAR-Adjunct.2019.00-34},
     issn = {},
     month = oct,
   }

2018

1. IEEE TVCG

   

   J1 HySAR: Hybrid Material Rendering by an Optical See-Through Head-Mounted Display with Spatial Augmented Reality Projection

   Takumi Hamasaki, Yuta Itoh, Yuichi Hiroi, Daisuke Iwai, and Maki Sugimoto

   IEEE Transactions on Visualization and Computer Graphics, Apr 2018

   Abs Bib DOI PDF

   Spatial augmented reality (SAR) pursues realism in rendering materials and objects. To advance this goal, we propose a hybrid SAR (HySAR) that combines a projector with optical see-through head-mounted displays (OST-HMD). In an ordinary SAR scenario with co-located viewers, the viewers perceive the same virtual material on physical surfaces. In general, the material consists of two components: a view-independent (VI) component such as diffuse reflection, and a view-dependent (VD) component such as specular reflection. The VI component is static over viewpoints, whereas the VD should change for each viewpoint even if a projector can simulate only one viewpoint at one time. In HySAR, a projector only renders the static VI components. In addition, the OST-HMD renders the dynamic VD components according to the viewer’s current viewpoint. Unlike conventional SAR, the HySAR concept theoretically allows an unlimited number of co-located viewers to see the correct material over different viewpoints. Furthermore, the combination enhances the total dynamic range, the maximum intensity, and the resolution of perceived materials. With proof-of-concept systems, we demonstrate HySAR both qualitatively and quantitatively with real objects. First, we demonstrate HySAR by rendering synthetic material properties on a real object from different viewpoints. Our quantitative evaluation shows that our system increases the dynamic range by 2.24 times and the maximum intensity by 2.12 times compared to an ordinary SAR system. Second, we replicate the material properties of a real object by SAR and HySAR, and show that HySAR outperforms SAR in rendering VD specular components.

   @article{hamasaki2018hysar,
     author = {Hamasaki, Takumi and Itoh, Yuta and Hiroi, Yuichi and Iwai, Daisuke and Sugimoto, Maki},
     journal = {IEEE Transactions on Visualization and Computer Graphics},
     title = {HySAR: Hybrid Material Rendering by an Optical See-Through Head-Mounted Display with Spatial Augmented Reality Projection},
     year = {2018},
     volume = {24},
     number = {4},
     pages = {1457-1466},
     keywords = {Augmented realtiy;Head-mounted displays;Rendering (computer graphics);Optical reflection;Adaptive optics;Optical see-through displays;hybrid material rendering;spatial augmented reality},
     doi = {10.1109/TVCG.2018.2793659},
     issn = {1941-0506},
     month = apr,
   }

2017

1. Augmented Human

   CF1 AdaptiVisor: assisting eye adaptation via occlusive optical see-through head-mounted displays

   Yuichi Hiroi, Yuta Itoh, Takumi Hamasaki, and Maki Sugimoto

   In Proceedings of the 8th Augmented Human International Conference, Silicon Valley, California, USA, Mar 2017

   Best paper Best Paper, 3rd Place, Augmented Human 2017.

   Abs Bib DOI PDF Video

   Brightness adaptation is a fundamental ability in human visual system, and adjusts various levels of darkness and light. While this ability is continuously used, and it can mostly handle sudden lighting changes in the environment, the adaptation could still take several minutes. Moreover, during the adaptation, the color perception changes as well. This slow reactivity and perception change of the eyes could lead to mistakes for tasks performed in dazzling or temporally high-contrast environments such as when driving into the sun or during a welding process.We present AdaptiVisor, a vision augmentation system that assists the brightness adaptation of the eye. Our system selectively modulates the intensity of the light coming into the eyes via occlusion-capable Optical See-Through Head-Mounted Displays (OST-HMD). An integrated camera captures highlights and brightness in the environment via high-dynamic range capture, and our display system selectively dims or enhances part of field of views so that the user would not perceive rapid brightness changes. We build a proof-of-concept system to evaluate the feasibility of the adaptation assistance by combining a transmissive LCD panel and an OST-HMD, and test it with a user-perspective, view-point camera. The evaluation shows that the system decreases the overexposed area in a scene to 1/15th, and enhances the color by reducing majorly underexposed area to half. We also include a preliminary user trial and it indicates that the system also works for real eyes for the HMD part and to some extent for the LCD.

   @inproceedings{hiroi2017adaptivisor,
     pages = {1--9},
     month = mar,
     author = {Hiroi, Yuichi and Itoh, Yuta and Hamasaki, Takumi and Sugimoto, Maki},
     title = {AdaptiVisor: assisting eye adaptation via occlusive optical see-through head-mounted displays},
     year = {2017},
     isbn = {9781450348355},
     publisher = {Association for Computing Machinery},
     address = {New York, NY, USA},
     url = {https://doi.org/10.1145/3041164.3041178},
     doi = {10.1145/3041164.3041178},
     booktitle = {Proceedings of the 8th Augmented Human International Conference},
     articleno = {9},
     numpages = {9},
     keywords = {AdaptiVisor, augmented reality, brightness adaptation, head-mounted displays, near-eye displays, occlusive HMD, optical see-through, vision augmentation},
     location = {Silicon Valley, California, USA},
     series = {AH '17},
   }

2. IEEE VR Poster

   CP3 HySAR: Hybrid material rendering by an optical see-through head-mounted display with spatial augmented reality projection

   Yuichi Hiroi, Yuta Itoh, Takumi Hamasaki, Daisuke Iwai, and Maki Sugimoto

   In 2017 IEEE Virtual Reality (VR), Mar 2017

   Abs Bib DOI PDF

   We propose a hybrid SAR concept combining a projector and Optical See-Through Head-Mounted Displays (OST-HMD). Our proposed hybrid SAR system utilizes OST-HMD as an extra rendering layer to render a view-dependent property in OST-HMDs according to the viewer’s viewpoint. Combined with view-independent components created by a static projector, the viewer can see richer material contents. Unlike conventional SAR systems, our system theoretically allows unlimited number of viewers seeing enhanced contents in the same space while keeping the existing SAR experiences. Furthermore, the system enhances the total dynamic range, the maximum intensity, and the resolution of perceived materials. With a proof-of-concept system that consists of a projector and an OST-HMD, we qualitatively demonstrate that our system successfully creates hybrid rendering on a hemisphere object from five horizontal viewpoints. Our quantitative evaluation also shows that our system increases the dynamic range by 2.1 times and the maximum intensity by 1.9 times compared to an ordinary SAR system.

   @inproceedings{7892251,
     author = {Hiroi, Yuichi and Itoh, Yuta and Hamasaki, Takumi and Iwai, Daisuke and Sugimoto, Maki},
     booktitle = {2017 IEEE Virtual Reality (VR)},
     title = {HySAR: Hybrid material rendering by an optical see-through head-mounted display with spatial augmented reality projection},
     year = {2017},
     volume = {},
     number = {},
     pages = {211-212},
     keywords = {Rendering (computer graphics);Dynamic range;Spatial resolution;Cameras;Optical imaging;Electronic mail;H.5.1 [Information Interfaces and Presentation]: Multimedia Information Systems — Artificial, augmented, and virtual realities},
     doi = {10.1109/VR.2017.7892251},
     issn = {2375-5334},
     month = mar,
   }

2016

1. ACM TEI

   CP2 MARCut: Marker-based Laser Cutting for Personal Fabrication on Existing Objects

   Takashi Kikuchi, Yuichi Hiroi, Ross T. Smith, Bruce H. Thomas, and Maki Sugimoto

   In Proceedings of the TEI ’16: Tenth International Conference on Tangible, Embedded, and Embodied Interaction, Eindhoven, Netherlands, Feb 2016

   Abs Bib DOI

   Typical personal fabrication using a laser cutter allows objects to be created from raw material and the engraving of existing objects. Current methods to precisely align an object with the laser is a difficult process due to indirect manipulations. In this paper, we propose a marker-based system as a novel paradigm for direct interactive laser cutting on existing objects. Our system, MARCut, performs the laser cutting based on tangible markers that are applied directly onto the object to express the design. Two types of markers are available; hand constructed Shape Markers that represent the desired geometry, and Command Markers that indicate the operational parameters such as cut, engrave or material.

   @inproceedings{kikuchi2016marcut,
     month = feb,
     author = {Kikuchi, Takashi and Hiroi, Yuichi and Smith, Ross T. and Thomas, Bruce H. and Sugimoto, Maki},
     title = {MARCut: Marker-based Laser Cutting for Personal Fabrication on Existing Objects},
     year = {2016},
     isbn = {9781450335829},
     publisher = {Association for Computing Machinery},
     address = {New York, NY, USA},
     url = {https://doi.org/10.1145/2839462.2856549},
     doi = {10.1145/2839462.2856549},
     booktitle = {Proceedings of the TEI '16: Tenth International Conference on Tangible, Embedded, and Embodied Interaction},
     pages = {468–474},
     numpages = {7},
     keywords = {Laser Cutting, Marker-based, Personal Fabrication},
     location = {Eindhoven, Netherlands},
     series = {TEI '16},
   }

2. SIGGRAPH E-tech

   

   CD1 Laplacian vision: augmenting motion prediction via optical see-through head-mounted displays and projectors

   Yuta Itoh, Yuichi Hiroi, Jiu Otsuka, Maki Sugimoto, Jason Orlosky, Kiyoshi Kiyokawa, and Gudrun Klinker

   In ACM SIGGRAPH 2016 Emerging Technologies, Anaheim, California, Jul 2016

   Abs Bib DOI Video

   Naı̈ve physics [7], or folk physics, is our ability to understand physical phenomena. We regularly use this ability in life to avoid collisions in traffic, follow a tennis ball and time the return shot, or while working in dynamic industrial settings. Though this skill improves with practice, it is still imperfect, which leads to mistakes and misjudgments for time intensive tasks. People still often miss a tennis shot, which might cause them to lose the match, or fail to avoid a car or pedestrian, which can lead to injury or even death.As a step towards reducing these errors in human judgement, we present Laplacian Vision (LV), a vision augmentation system which assists the human ability to predict future trajectory information. By tracking real world objects and estimating their trajectories, we can improve a users’s prediction of the landing spot of a ball or the path of an oncoming car. We have designed a system that can track a flying ball in real time, predict its future trajectory, and visualize it in the user’s field of view. The system is also calibrated to account for end-to-end delays so that the trajectory appears to emanate forward from the moving object. We also conduct a user study where 29 subjects predict an object’s landing spot, and show that prediction accuracy improves 3 fold using LV.

   @inproceedings{itoh2016laplacian,
     pages = {1},
     month = jul,
     author = {Itoh, Yuta and Hiroi, Yuichi and Otsuka, Jiu and Sugimoto, Maki and Orlosky, Jason and Kiyokawa, Kiyoshi and Klinker, Gudrun},
     title = {Laplacian vision: augmenting motion prediction via optical see-through head-mounted displays and projectors},
     year = {2016},
     isbn = {9781450343725},
     publisher = {Association for Computing Machinery},
     address = {New York, NY, USA},
     url = {https://doi.org/10.1145/2929464.2949028},
     doi = {10.1145/2929464.2949028},
     booktitle = {ACM SIGGRAPH 2016 Emerging Technologies},
     articleno = {13},
     location = {Anaheim, California},
     series = {SIGGRAPH '16},
   }

2015

1. ISMAR Poster

   

   CP1 Remote Welding Robot Manipulation Using Multi-view Images

   Yuichi Hiroi, Kei Obata, Katsuhiro Suzuki, Naoto Ienaga, Maki Sugimoto, Hideo Saito, and Tadashi Takamaru

   In 2015 IEEE International Symposium on Mixed and Augmented Reality, Sep 2015

   Abs Bib DOI

   This paper proposes a remote welding robot manipulation system by using multi-view images. After an operator specifies two-dimensional path on images, the system transforms it into three-dimensional path and displays the movement of the robot by overlaying graphics with images. The accuracy of our system is sufficient to weld objects when combining with a sensor in the robot. The system allows the non-expert operator to weld objects remotely and intuitively, without the need to create a 3D model of a processed object beforehand.

   @inproceedings{hiroi2015remote,
     author = {Hiroi, Yuichi and Obata, Kei and Suzuki, Katsuhiro and Ienaga, Naoto and Sugimoto, Maki and Saito, Hideo and Takamaru, Tadashi},
     booktitle = {2015 IEEE International Symposium on Mixed and Augmented Reality},
     title = {Remote Welding Robot Manipulation Using Multi-view Images},
     year = {2015},
     volume = {},
     number = {},
     pages = {128-131},
     keywords = {Cameras;Welding;Robot vision systems;Three-dimensional displays;Robot kinematics;Remote robot control;multi-view images;industrial robot;augmented reality},
     doi = {10.1109/ISMAR.2015.38},
     issn = {},
     month = sep,
   }