Turn creative prompts into interactive XR experiences with Gemini

Source: Google Developers Blog

With the release of Canvas in the Gemini web app, our Android XR team began exploring how to make immersive computing accessible to developers and users at large. Gemini proved especially adept at generating interactive 3D web graphics, prompting us to ask: what if you could experience Gemini’s 3D creations in extended reality?

We built an immersive biology simulation where Gemini modeled the complex interactions between blood cells, rendering them with WebGL and Three.js. Converting that content to XR via WebXR APIs added an “Enter XR” button, letting users explore car‑sized blood cells at a sub‑cellular scale. The result demonstrated Gemini’s ability to create fully realized, interactive XR experiences and inspired tools that let you prototype complex 3D models far faster than traditional methods.

Get started building 3D experiences with Gemini

While high‑fidelity XR development traditionally requires a broad skill set, Gemini offers a rapid‑prototyping workflow that moves you from a creative spark to a functional experience you can test immediately. To begin, you’ll need a Samsung Galaxy XR headset and access to Gemini 3 Pro.

1. Set up your XR Gem – Start with the pre‑built XR Blocks Gem or create your own by downloading the XR Blocks ultra‑prompt to give Gemini advanced perception capabilities for realistic AR physics. Upload the prompt to a new Gem in the Gemini web interface and add a description such as “you are a creative and resourceful WebXR developer with superb aesthetic taste and technical execution.” This persona helps Gemini interpret your visual scene and generate polished 3D textures.

2. Launch the experience – Open Gemini in Chrome on your Galaxy XR, start a new chat with your XR Gem, and select Canvas. You can now build interactive components (e.g., a dandelion that reacts to touch) and even embed Gemini Live to talk to Gemini directly without leaving the immersive session.

3. Build and iterate – Think of an idea and ask Gemini to create it. Prompt “create a volcanic environment” to change your surroundings instantly, or “add a butterfly there” to place specific objects. You can give characters backstories and agency. If the result isn’t perfect, try again or layer simple ideas together to add complexity. When finished, your creation becomes a shareable web link.

Ideas to get you building today

• “Make a pen that draws rainbows in 3D.”
• “Make a bunch of bubbles that pop when I touch them.”
• “Make an origami bird that flies around the room for a few seconds, lands on my hand, and repeats when I move my hand.”

Advanced challenges

• Create a smart planetarium using real star data, e.g., NASA exoplanets, Smithsonian star catalog, or ESA’s 3D map of the galaxy.
• Build a clone of your favorite 80s arcade game and re‑skin it with a cat‑themed aesthetic.
• Develop a tool that turns your 3D doodles into lifelike sculptures.

Create something great and share your Canvas so others can try and build on it. Browse the XR Blocks demos for live examples designed for Android XR, and head over to gemini.google.com to get started today using the steps above!