VR Creation Tools

2025

1. IEEE Perv. Comp.

   

   J12 MetaGadget: An Accessible Framework for IoT Integration Into Commercial Metaverse Platforms

   Ryutaro Kurai, Hikari Yanagawa, Yuichi Hiroi, and Takefumi Hiraki

   IEEE Pervasive Computing, Sep 2025

   Award Good Experience Design Award 2025, DAXX.

   Abs Bib DOI arXiv Code

   While the integration of Internet of Things (IoT) devices in virtual spaces is becoming increasingly common, technical barriers to controlling custom devices in multiuser virtual reality (VR) environments remain high, particularly limiting new applications in educational and prototyping settings. We propose MetaGadget, a framework for connecting IoT devices to commercial metaverse platforms that implements device control through HTTP-based event triggers without requiring persistent client connections. Through two workshops focused on smart home control and custom device integration, we explored the potential application of IoT connectivity in multiuser metaverse environments. Participants successfully implemented new interactions unique to the metaverse, such as environmental sensing and remote control systems that support simultaneous operation by multiple users, and reported positive feedback on the ease of system development. We verified that our framework provides a new approach to controlling IoT devices in the metaverse while reducing technical requirements and provides a foundation for creative practice that connects multiuser VR environments and physical spaces.

   @article{kurai2025metagadget,
     month = sep,
     author = {Kurai, Ryutaro and Yanagawa, Hikari and Hiroi, Yuichi and Hiraki, Takefumi},
     journal = {IEEE Pervasive Computing},
     title = {MetaGadget: An Accessible Framework for IoT Integration Into Commercial Metaverse Platforms},
     year = {2025},
     volume = {24},
     number = {4},
     pages = {63-73},
     keywords = {Metaverse;Internet of Things;Servers;Switches;Conferences;Aerospace electronics;Fans;Control systems;HTTP;Digital twins},
     doi = {10.1109/MPRV.2025.3602289},
   }

2. IEEE Access

   

   J11 MagicCraft: Natural Language-Driven Generation of Dynamic and Interactive 3D Objects for Commercial Metaverse Platforms

   Ryutaro Kurai, Takefumi Hiraki, Yuichi Hiroi, Yutaro Hirao, Monica Perusquía-Hernández, Hideaki Uchiyama, and Kiyoshi Kiyokawa

   IEEE Access, Jul 2025

   Abs Bib DOI arXiv

   Metaverse platforms are rapidly evolving to provide immersive spaces for user interaction and content creation. However, the generation of dynamic and interactive 3D objects remains challenging due to the need for advanced 3D modeling and programming skills. To address this challenge, we present MagicCraft, a system that generates functional 3D objects from natural language prompts for metaverse platforms. MagicCraft uses generative AI models to manage the entire content creation pipeline: converting user text descriptions into images, transforming images into 3D models, predicting object behavior, and assigning necessary attributes and scripts. It also provides an interactive interface for users to refine generated objects by adjusting features such as orientation, scale, seating positions, and grip points. Implemented on Cluster, a commercial metaverse platform, MagicCraft was evaluated by 7 expert CG designers and 51 general users. Results show that MagicCraft significantly reduces the time and skill required to create 3D objects. Users with no prior experience in 3D modeling or programming successfully created complex, interactive objects and deployed them in the metaverse. Expert feedback highlighted the system’s potential to improve content creation workflows and support rapid prototyping. By integrating AI-generated content into metaverse platforms, MagicCraft makes 3D content creation more accessible.

   @article{kurai2025magiccraft,
     month = jul,
     author = {Kurai, Ryutaro and Hiraki, Takefumi and Hiroi, Yuichi and Hirao, Yutaro and Perusquía-Hernández, Monica and Uchiyama, Hideaki and Kiyokawa, Kiyoshi},
     journal = {IEEE Access},
     title = {MagicCraft: Natural Language-Driven Generation of Dynamic and Interactive 3D Objects for Commercial Metaverse Platforms},
     year = {2025},
     volume = {13},
     number = {},
     pages = {132459-132474},
     keywords = {Three-dimensional displays;Metaverse;Solid modeling;Generative AI;Codes;Programming;Natural languages;Biological system modeling;Visualization;Pipelines;Metaverse;3D object generation;generative AI;AI-assisted design},
     doi = {10.1109/ACCESS.2025.3587232},
   }

3. IEEE Access

   

   J9 MagicItem: Dynamic Behavior Design of Virtual Objects With Large Language Models in a Commercial Metaverse Platform

   Ryutaro Kurai, Takefumi Hiraki, Yuichi Hiroi, Yutaro Hirao, Monica Perusquia-Hernandez, Hideki Uchiyama, and Kiyoshi Kiyokawa

   IEEE Access, Jan 2025

   Abs Bib DOI arXiv

   To create rich experiences in virtual reality (VR) environments, it is essential to define the behavior of virtual objects through programming. However, programming in 3D spaces requires a wide range of background knowledge and programming skills. Although Large Language Models (LLMs) have provided programming support, they are still primarily aimed at programmers. In metaverse platforms, where many users inhabit VR spaces, most users are unfamiliar with programming, making it difficult for them to modify the behavior of objects in the VR environment easily. Existing LLM-based script generation methods for VR spaces require multiple lengthy iterations to implement the desired behaviors and are difficult to integrate into the operation of metaverse platforms. To address this issue, we propose a tool that generates behaviors for objects in VR spaces from natural language within Cluster, a metaverse platform with a large user base. By integrating LLMs with the Cluster Script provided by this platform, we enable users with limited programming experience to define object behaviors within the platform freely. We have also integrated our tool into a commercial metaverse platform and are conducting online experiments with 63 general users of the platform. The experiments show that even users with no programming background can successfully generate behaviors for objects in VR spaces, resulting in a highly satisfying system. Our research contributes to democratizing VR content creation by enabling non-programmers to design dynamic behaviors for virtual objects in metaverse platforms.

   @article{kurai2025magicitem,
     month = jan,
     author = {Kurai, Ryutaro and Hiraki, Takefumi and Hiroi, Yuichi and Hirao, Yutaro and Perusquia-Hernandez, Monica and Uchiyama, Hideki and Kiyokawa, Kiyoshi},
     journal = {IEEE Access},
     title = {MagicItem: Dynamic Behavior Design of Virtual Objects With Large Language Models in a Commercial Metaverse Platform},
     year = {2025},
     volume = {13},
     number = {},
     pages = {19132-19143},
     doi = {10.1109/ACCESS.2025.3530439},
     keywords = {Metaverse;Codes;Three-dimensional displays;Large language models;Natural languages;Aerospace electronics;Web servers;Usability;Programming profession;Hands;Large-language model;low-code programming;metaverse platform;virtual reality},
   }

4. ISMAR Workshop

   

   CW6 MagicPen: Interactive Sketch-to-3D Generation in Commertial Metaverse Platforms

   Ryutaro Kurai, Yuji Hatada, Takefumi Hiraki, and Yuichi Hiroi

   In 2025 IEEE International Symposium on Mixed and Augmented Reality Abstracts and Workshops (ISMAR 2025 Workshop), Oct 2025

   Abs Bib DOI PDF

   Although User Generated Content (UGC) distinguishes metaverse platforms from conventional VR, 3D content creation remains limited to users with specialized modeling expertise and platform-specific technical knowledge. While generative AI advances have improved accessibility, existing approaches fail to provide both VR-native operation and intuitive input capabilities simultaneously, maintaining barriers for novice users. We propose MagicPen, a system enabling direct conversion of hand-drawn sketches into 3D objects within VR environments through a three-stage AI pipeline comprising image refinement, 3D model generation, and metaverse integration. Evaluation with 68 diverse sketch inputs revealed 35.5-second average processing times and high-fidelity geometric reproduction, validating seamless Cluster platform integration. The system enables collaborative 3D content creation in social VR spaces, allowing users without modeling expertise to transform individual creative workflows into shared social experiences.

   @inproceedings{kurai2025magicpen,
     author = {Kurai, Ryutaro and Hatada, Yuji and Hiraki, Takefumi and Hiroi, Yuichi},
     title = {MagicPen: Interactive Sketch-to-3D Generation in Commertial Metaverse Platforms},
     booktitle = {2025 IEEE International Symposium on Mixed and Augmented Reality Abstracts and Workshops (ISMAR 2025 Workshop)},
     year = {2025},
     volume = {},
     number = {},
     pages = {602--604},
     keywords = {Metaverse;Sketch-based Modeling;3D Generative AI},
     month = oct,
     doi = {10.1109/ISMAR-Adjunct68609.2025.00119},
   }

5. IEEE VR Poster

   CP11 An implementation of MagicCraft: Generating Interactive 3D Objects and Their Behaviors from Text for Commercial Metaverse Platforms

   Ryutaro Kurai, Takefumi Hiraki, Yuichi Hiroi, Yutaro Hirao, Monica Perusquia-Hernandez, Hideki Uchiyama, and Kiyoshi Kiyokawa

   In 2025 IEEE Conference on Virtual Reality and 3D User Interfaces Adjunct 2025, Mar 2025

   Abs Bib DOI PDF

   Metaverse platforms are rapidly evolving to provide immersive spaces. However, the generation of dynamic and interactive 3D objects remains a challenge due to the need for advanced 3D modeling and programming skills. We present MagicCraft, a system that generates functional 3D objects from natural language prompts. MagicCraft uses generative AI models to manage the entire content creation pipeline: converting user text descriptions into images, transforming images into 3D models, predicting object behavior, and assigning necessary attributes and scripts. It also provides an interactive interface for users to refine generated objects by adjusting features like orientation, scale, seating positions, and grip points.

   @inproceedings{kurai2025magiccraft-poster,
     author = {Kurai, Ryutaro and Hiraki, Takefumi and Hiroi, Yuichi and Hirao, Yutaro and Perusquia-Hernandez, Monica and Uchiyama, Hideki and Kiyokawa, Kiyoshi},
     title = {An implementation of MagicCraft: Generating Interactive 3D Objects and Their Behaviors from Text for Commercial Metaverse Platforms},
     booktitle = {2025 IEEE Conference on Virtual Reality and 3D User Interfaces Adjunct 2025},
     year = {2025},
     volume = {},
     number = {},
     pages = {1284-1285},
     keywords = {Solid modeling;Three-dimensional displays;Metaverse;Generative AI;Natural languages;Refining;Pipelines;Programming;Predictive models;User interfaces;Metaverse;3D Object Generation;Generative AI;AI-Assisted Design},
     month = mar,
     doi = {10.1109/VRW66409.2025.00288},
   }

2024

1. ISMAR Demo

   CD3 MetaGadget: IoT Framework for Event-Triggered Integration of User-Developed Devices into Commercial Metaverse Platforms

   Ryutaro Kurai, Yuichi Hiroi, and Takefumi Hiraki

   In 2024 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) , Oct 2024

   Abs Bib DOI Video Code

   This demonstration introduces MetaGadget, an IoT framework designed to integrate user-developed devices into commercial metaverse platforms. Synchronizing virtual reality (VR) environments with physical devices has traditionally required a constant connection to VR clients, limiting flexibility and resource efficiency. MetaGadget overcomes these limitations by configuring user-developed devices as IoT units with server capabilities, supporting communication via HTTP protocols within the commercial metaverse platform, Cluster. This approach enables event-triggered device control without the need for persistent connections from metaverse clients. Through the demonstration, users will experience event-triggered interaction between VR and physical devices, as well as real-world device control through the VR space by multiple people. Our framework is expected to reduce technical barriers to integrating VR spaces and custom devices, contribute to interoperability, and increase resource efficiency through event-triggered connections.

   @inproceedings{kurai2024metagadget,
     author = {Kurai, Ryutaro and Hiroi, Yuichi and Hiraki, Takefumi},
     booktitle = { 2024 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) },
     title = {{ MetaGadget: IoT Framework for Event-Triggered Integration of User-Developed Devices into Commercial Metaverse Platforms }},
     year = {2024},
     volume = {},
     issn = {},
     pages = {632-633},
     keywords = {Virtual Reality; Metaverse Platform; User-Developed Devices},
     month = oct,
     doi = {10.1109/ISMAR-Adjunct64951.2024.00185},
   }

2. arXiv

   

   P1 PanoTree: Autonomous Photo-Spot Explorer in Virtual Reality Scenes

   Tomohiro Hayase, Sacha Braun, Hikari Yanagawa, Itsuki Orito, and Yuichi Hiroi

   May 2024

   Abs Bib DOI arXiv Video Code Website

   Social VR platforms enable social, economic, and creative activities by allowing users to create and share their own virtual spaces. In social VR, photography within a VR scene is an important indicator of visitors’ activities. Although automatic identification of photo spots within a VR scene can facilitate the process of creating a VR scene and enhance the visitor experience, there are challenges in quantitatively evaluating photos taken in the VR scene and efficiently exploring the large VR scene. We propose PanoTree, an automated photo-spot explorer in VR scenes. To assess the aesthetics of images captured in VR scenes, a deep scoring network is trained on a large dataset of photos collected by a social VR platform to determine whether humans are likely to take similar photos. Furthermore, we propose a Hierarchical Optimistic Optimization (HOO)-based search algorithm to efficiently explore 3D VR spaces with the reward from the scoring network. Our user study shows that the scoring network achieves human-level performance in distinguishing randomly taken images from those taken by humans. In addition, we show applications using the explored photo spots, such as automatic thumbnail generation, support for VR world creation, and visitor flow planning within a VR scene.

   @misc{hayase2024panotree,
     doi = {10.48550/arXiv.2405.17136},
     month = may,
     title = {PanoTree: Autonomous Photo-Spot Explorer in Virtual Reality Scenes},
     author = {Hayase, Tomohiro and Braun, Sacha and Yanagawa, Hikari and Orito, Itsuki and Hiroi, Yuichi},
     year = {2024},
     eprint = {2405.17136},
     archiveprefix = {arXiv},
     primaryclass = {cs.CV},
   }