Turn creative prompts into interactive XR experiences with Gemini

Source: Google Developers Blog

Immersive XR with Gemini

With the release of Canvas in the Gemini web app, our Android XR team began exploring how to make immersive computing accessible to developers and users at large. We noticed that Gemini is particularly adept at generating interactive 3D web graphics, and asked a simple question: What if you could experience Gemini’s 3D web creations in extended reality?

To test this, we built an immersive biology simulation. Gemini first modeled the complex interactions between blood cells and rendered them visually using WebGL and Three.js. The shift occurred when we converted that content to XR using WebXR APIs, adding an “Enter XR” button that let users explore car‑sized blood cells at a sub‑cellular scale. This demonstrated Gemini’s capability to build fully realized, interactive XR experiences.

Get started building 3D experiences with Gemini

While high‑fidelity XR development traditionally requires a broad range of skills, Gemini offers a new way to rapidly prototype and explore interactive 3D concepts in real‑time. This approach lets you move quickly from a creative spark to a functional experience you can test immediately.

1. Set up your XR Gem

• Start with the pre‑built XR Blocks Gem or build your own by downloading the XR Blocks ultra‑prompt to give Gemini advanced perception capabilities for realistic AR physics.

• Upload the prompt to a new Gem in the Gemini web interface and add a description such as:

  “You are a creative and resourceful WebXR developer with superb aesthetic taste and technical execution.”

  This persona helps Gemini interpret your visual scene and generate polished 3D textures.

2. Launch the experience

• Open Gemini in Chrome on your Samsung Galaxy XR headset.
• Start a new chat with your XR Gem and select Canvas.
• Build interactive components (e.g., a dandelion that reacts to touch) and embed Gemini Live to talk to Gemini directly or continue creating without leaving the immersive session.

3. Build and iterate

Think of an idea and ask Gemini to build it:

• “Create a volcanic environment.”
• “Add a butterfly there.”
• Give characters backstories and agency within your scene.

If the result isn’t perfect, try again or layer simple ideas together to add complexity. When you’re finished, your creation becomes a web link you can share with anyone.

Ideas to get you building today

Starter prompts

• “Make a pen that draws rainbows in 3D.”
• “Make a bunch of bubbles that pop when I touch them.”
• “Make an origami bird that flies around the room for a few seconds, lands on my hand, and flies away when I move my hand.”

Advanced challenges

• Create a smart planetarium with real star data, e.g., NASA exoplanets, Smithsonian star catalog, ESA’s 3‑D map of the galaxy.
• Clone your favorite 80s arcade game and re‑skin it with a cat theme.
• Build a tool that turns your 3D doodles into lifelike sculptures.

Create something great and share your Canvas so others can try it and build on it. Browse the XR Blocks demos for live examples designed for Android XR, and head over to gemini.google.com to get started today using the steps above.