AI Agents in Social VR & Field Studies

2026

1. IEICE Trans. Inf. & Syst.

   

   J14 MetaProxy: Bridging Agents and Environments in Commercial Metaverse Platforms through Remote Intermediary Monitoring

   Ryutaro Kurai, Takefumi Hiraki, Yuichi Hiroi, Yutaro Hirao, Monica Perusquia-Hernandez, Hideaki Uchiyama, and Kiyoshi Kiyokawa

   IEICE Transactions on Information and Systems, Jun 2026

   Abs Bib DOI

   Metaverse platforms have emerged as central hubs for virtual communication and entertainment, attracting large numbers of users. Integrating autonomous agent avatars into these spaces could enable diverse applications, including educational support, psychological assistance, and enhanced entertainment experiences. However, commercial metaverse platforms typically lack external Application Programming Interfaces (APIs) for agent control, creating significant implementation barriers for researchers and developers. This paper presents a network proxy module that monitors client-server communication in commercial metaverse platforms, enabling avatar control without platform-side modifications. Our approach extracts spatial information from network packets and controls avatars via keyboard-mouse emulation, requiring no changes to the underlying platform architecture. We validated our method by implementing a Large Language Model (LLM)-integrated agent avatar on Cluster, a commercial metaverse platform. The experimental evaluation demonstrated the agent’s ability to respond to user proximity, comments, and emotional expressions with an average response time of 1.34 seconds.

   @article{kurai2026metaproxy,
     month = jun,
     title = {MetaProxy: Bridging Agents and Environments in Commercial Metaverse Platforms through Remote Intermediary Monitoring},
     author = {Kurai, Ryutaro and Hiraki, Takefumi and Hiroi, Yuichi and Hirao, Yutaro and Perusquia-Hernandez, Monica and Uchiyama, Hideaki and Kiyokawa, Kiyoshi},
     journal = {IEICE Transactions on Information and Systems},
     volume = {E109.D},
     number = {6},
     pages = {870-878},
     year = {2026},
     doi = {10.1587/transinf.2025HCP0008},
   }

2. IEEE AIxVR

   CF18 Large-scale, Longitudinal Field Study of AI-agent-User Interactions in Commercial Metaverse

   Yuichi Hiroi, Yuki Imai, Hikari Yanagawa, and Takefumi Hiraki

   In Proceedings of 2026 IEEE International Conference on Artificial Intelligence and eXtended and Virtual Reality (AIxVR), Jan 2026

   Abs Bib DOI PDF

   Although commercial metaverse platforms have advanced social implementation, they struggle to retain users. Most new users abandon these platforms shortly after their initial experience, which hinders the realization of the metaverse’s social and economic potential. Although AI-agents could promote user interaction and retention by acting as social catalysts, existing research has focused on laboratory-based short-term validation. This approach provides limited evidence of the long-term effectiveness of agents in commercial environments. This study examines the impact of Large Language Model (LLM)-based AI-agents on user continuation behavior by operating AI-agents for 31 days on the commercial metaverse platform Cluster and observing the natural usage behavior of 5,020 unique users. The analysis used two complementary approaches: (1) an aggregate effect analysis at the weekly habit formation level and (2) an within-user effect analysis at the daily decision-making level with individual difference controls. The results revealed that interaction with the AI-agent produces lasting changes in user behavior, especially among new users. Furthermore, the cumulative relationship-building process through continuous contact opportunities rather than single impressive experiences with AI-agents was found to decisively affect users’ continued metaverse usage.

   @inproceedings{hiroi2026aiagent,
     pages = {108--118},
     doi = {10.1109/AIxVR67263.2026.00021},
     author = {Hiroi, Yuichi and Imai, Yuki and Yanagawa, Hikari and Hiraki, Takefumi},
     title = {Large-scale, Longitudinal Field Study of AI-agent-User Interactions in Commercial Metaverse},
     year = {2026},
     booktitle = {Proceedings of 2026 IEEE International Conference on Artificial Intelligence and eXtended and Virtual Reality (AIxVR)},
     month = jan,
     series = {AIxVR '26},
   }

2025

1. IEEE ISMAR

   CF12 Navigation Pixie: Implementation and Empirical Study Toward On-demand Navigation Agents in Commercial Metaverse

   Hikari Yanagawa, Yuichi Hiroi, Satomi Tokida, Yuji Hatada, and Takefumi Hiraki

   In Proceedings of 2025 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), Oct 2025

   Abs Bib DOI arXiv PDF Video Website

   While commercial metaverse platforms offer diverse user-generated content, they lack effective navigation assistance that can dynamically adapt to users’ interests and intentions. Although previous research has investigated on-demand agents in controlled environments, implementation in commercial settings with diverse world configurations and platform constraints remains challenging.We present Navigation Pixie, an on-demand navigation agent employing a loosely coupled architecture that integrates structured spatial metadata with LLM-based natural language processing while minimizing platform dependencies, which enables experiments on the extensive user base of commercial metaverse platforms. Our cross-platform experiments on commercial metaverse platform Cluster with 99 PC client and 94 VR-HMD participants demonstrated that Navigation Pixie significantly increased dwell time and free exploration compared to fixed-route and no-agent conditions across both platforms. Subjective evaluations revealed consistent on-demand preferences in PC environments versus context-dependent social perception advantages in VR-HMD. This research contributes to advancing VR interaction design through conversational spatial navigation agents, establishes cross-platform evaluation methodologies revealing environment-dependent effectiveness, and demonstrates empirical experimentation frameworks for commercial metaverse platforms.

   @inproceedings{yanagawa2025navigationpixie,
     pages = {1137--1147},
     title = {Navigation Pixie: Implementation and Empirical Study Toward On-demand Navigation Agents in Commercial Metaverse},
     author = {Yanagawa, Hikari and Hiroi, Yuichi and Tokida, Satomi and Hatada, Yuji and Hiraki, Takefumi},
     year = {2025},
     booktitle = {Proceedings of 2025 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)},
     month = oct,
     doi = {10.1109/ISMAR67309.2025.00120},
   }

2. ISMAR Workshop

   

   CW5 Dynamic Gimmick Leaning for Navigation Agents in Social VR Through User-Agent Dialogue

   Atsuya Matsumoto, Hikari Yanagawa, Yuichi Hiroi, Yuji Hatada, Takuji Narumi, and Takefumi Hiraki

   In 2025 IEEE International Symposium on Mixed and Augmented Reality Abstracts and Workshops (ISMAR 2025 Workshop), Oct 2025

   Abs Bib DOI PDF

   User-generated interactive mechanisms (“gimmicks”) in metaverse platforms offer unique experiences, but their complexity hinders player exploration. Existing navigation agents, reliant on static metadata, cannot comprehend these dynamic gimmicks. We propose the “Conversational Knowledge Acquisition Loop,” a method for agents to dynamically learn gimmicks through player conversation. A preliminary study showed that our agent successfully acquired knowledge via conversation and applied it to solve analogous gimmicks.

   @inproceedings{matsumoto2025dynamic,
     author = {Matsumoto, Atsuya and Yanagawa, Hikari and Hiroi, Yuichi and Hatada, Yuji and Narumi, Takuji and Hiraki, Takefumi},
     title = {Dynamic Gimmick Leaning for Navigation Agents in Social VR Through User-Agent Dialogue},
     booktitle = {2025 IEEE International Symposium on Mixed and Augmented Reality Abstracts and Workshops (ISMAR 2025 Workshop)},
     year = {2025},
     volume = {},
     number = {},
     pages = {605--607},
     keywords = {Autonomous Agent;Virtual Reality;Embodied Conversational Agent;Social VR.},
     month = oct,
     doi = {10.1109/ISMAR-Adjunct68609.2025.00120},
   }

2024

1. IEEE VR Workshop

   

   CW2 Design and Implementation of Agent APIs for Large-Scale Social VR Platforms

   Ryutaro Kurai, Takefumi Hiraki, Yuichi Hiroi, Yutaro Hirao, Monica Perusquia-Hernandez, Hideaki Uchiyama, and Kiyoshi Kiyokawa

   In 2024 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW) , Mar 2024

   Abs Bib DOI

   Implementing an autonomous agent on a social VR platform where many users share space requires diverse information. In particular, it is required to recognize the distance from other users, their orientation toward each other, the avatar’s pose, and text and voice messages, and to behave accordingly. This paper proposes an API to obtain the above information on “Cluster,” a multi-device social VR platform in operation, and an agent that uses the API. We have implemented this API using a network proxy. The agent using this API can connect to ChatGPT [7] and have a conversation in real time. We measured the latency required for the conversation and confirmed that the response time was about 1 second.

   @inproceedings{kurai2024agentapi,
     author = {Kurai, Ryutaro and Hiraki, Takefumi and Hiroi, Yuichi and Hirao, Yutaro and Perusquia-Hernandez, Monica and Uchiyama, Hideaki and Kiyokawa, Kiyoshi},
     booktitle = { 2024 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW) },
     title = {{ Design and Implementation of Agent APIs for Large-Scale Social VR Platforms }},
     year = {2024},
     volume = {},
     issn = {},
     pages = {584-587},
     keywords = {Three-dimensional displays;Text recognition;Oral communication;Virtual reality;Speech recognition;User interfaces;Software},
     doi = {10.1109/VRW62533.2024.00112},
     url = {https://doi.ieeecomputersociety.org/10.1109/VRW62533.2024.00112},
     publisher = {IEEE Computer Society},
     address = {Los Alamitos, CA, USA},
     month = mar,
   }