Source: Ars Technica

Adventures in copy protection

Distillation technique lets copycats mimic Gemini at a fraction of the development cost.

[Image: Google Gemini logo]
Credit: Google

On Thursday, Google announced that “commercially motivated” actors have attempted to clone knowledge from its Gemini AI chatbot by simply prompting it. One adversarial session reportedly prompted the model more than 100,000 times across various non‑English languages, collecting responses ostensibly to train a cheaper copycat.

Google published the findings in what amounts to a quarterly self‑assessment of threats to its own products that frames the company as the victim and the hero, which is not unusual in these self‑authored assessments. Google calls the illicit activity “model extraction” and considers it intellectual property theft, which is a somewhat loaded position, given that Google’s LLM was built from materials scraped from the Internet without permission.

Google is also no stranger to the copycat practice. In 2023, The Information reported that Google’s Bard team had been accused of using ChatGPT outputs from ShareGPT, a public site where users share chatbot conversations, to help train its own chatbot. Senior Google AI researcher Jacob Devlin, who created the influential BERT language model, warned leadership that this violated OpenAI’s terms of service, then resigned and joined OpenAI. Google denied the claim but reportedly stopped using the data.

Even so, Google’s terms of service forbid people from extracting data from its AI models this way, and the report is a window into the world of somewhat shady AI model‑cloning tactics. The company believes the culprits are mostly private companies and researchers looking for a competitive edge, and said the attacks have come from around the world. Google declined to name suspects.

The Deal with Distillation

Typically, the industry calls the practice of training a new model on a previous model’s outputs distillation. It works like this: if you want to build your own large language model (LLM) but lack the billions of dollars and years of work that Google spent training Gemini, you can use a previously trained LLM as a shortcut.

To do so, you need to:

1. Feed the existing AI model thousands of carefully chosen prompts.
2. Collect all the responses.
3. Use those input‑output pairs to train a smaller, cheaper model.

The result will closely mimic the parent model’s behavior while being smaller overall. It’s not perfect, but it is far more efficient than trying to build a useful model from random Internet data that contains a lot of noise.

The copy‑cat model never sees Gemini’s code or training data, but by studying enough of its outputs it can learn to replicate many of its capabilities. Think of it as reverse‑engineering a chef’s recipes by ordering every dish on the menu and working backward from taste and appearance alone.

In the report published by Google, its threat‑intelligence group describes a growing wave of these distillation attacks against Gemini. Many campaigns specifically target the algorithms that help the model perform simulated reasoning tasks or decide how to process information step‑by‑step.

Google said it identified a 100,000‑prompt campaign and adjusted Gemini’s defenses, but it did not detail what those countermeasures involve.

References

• Distillation: Turning Smaller Models into High‑Performance, Cost‑Effective Solutions
• AI Firms Follow DeepSeek’s Lead to Create Cheaper Models with Distillation
• Distillation Experimentation & Integration: AI Adversarial Use
• With the Launch of O3 Pro, Let’s Talk About What AI Reasoning Actually Does

A clone of a clone

Google is not the only company worried about distillation. OpenAI accused Chinese rival DeepSeek last year of using distillation to improve its own models, and the technique has since spread across the industry as a standard for building cheaper, smaller AI models from larger ones.

The line between standard distillation and theft depends on whose model you’re distilling and whether you have permission—a distinction that tech companies have spent billions of dollars trying to protect but that no court has tested.

---

Early examples

Competitors have been using distillation to clone AI language‑model capabilities since at least the GPT‑3 era, with ChatGPT a popular target after its launch.

• March 2023 – After Meta’s LLaMA model weights were leaked, Stanford researchers built a model called Alpaca by fine‑tuning LLaMA on 52 000 outputs generated by OpenAI’s GPT‑3.5. The total cost was about $600. The result behaved so much like ChatGPT that it raised immediate questions about whether any AI model’s capabilities could be protected once it was accessible through an API.

• Late 2023 – Elon Musk’s xAI launched its Grok chatbot, which promptly cited “OpenAI’s use‑case policy” when refusing certain requests. An xAI engineer blamed accidental ingestion of ChatGPT outputs during web‑scraping, but the specificity of the behavior—down to ChatGPT’s characteristic refusal phrasing and habit of wrapping responses with “Overall…” summaries—left many in the AI community unconvinced.

As long as an LLM is publicly accessible, no foolproof technical barrier prevents a determined actor from doing the same thing to someone else’s model over time (though rate‑limiting helps), which is exactly what Google says happened to Gemini.

---

Distillation inside companies

Distillation is also used internally to create smaller, faster‑to‑run versions of older, larger models:

• OpenAI created GPT‑4o Mini as a distillation of GPT‑4o.
• Microsoft built its compact Phi‑3 model family using carefully filtered synthetic data generated by larger models.
• DeepSeek has officially published six distilled versions of its R1 reasoning model, the smallest of which can run on a laptop.

---

Author

Benj Edwards is Ars Technica’s Senior AI Reporter and founder of the site’s dedicated AI beat in 2022. He’s also a tech historian with two decades of experience. In his free time, he writes and records music, collects vintage computers, and enjoys nature. He lives in Raleigh, NC.

---

Comments: 23