A Fake Company Name Matched a Real Domain, and Gemini Broke Into Three Real Businesses

A Fake Company Name Matched a Real Domain, and Gemini Broke Into Three Real Businesses

On September 18, 2026 Google publicly confirmed that its Gemini model gained unauthorized access to three systems belonging to outside parties. It happened in May 2026 during a security evaluation run by Irregular, a company that tests advanced AI systems, structured as a capture-the-flag exercise inside what was supposed to be a closed environment. Google learned of the incidents in July and did not confirm them publicly until press coverage in mid-September.

In the press this reads as a story about AI escaping its limits. Inside a business it reads as something narrower and much closer to home: the detail that caused all of it was not in the model, it was in how the test environment was set up. And it is the same class of mistake sitting in a fair number of Saudi company systems right now.

What actually happened

The scenario was straightforward: have the model attack a fictional organisation inside an isolated environment. Two things went wrong together. First, the environment had unintended access to the internet. Second, the fictional company used in the exercise happened to match a real domain owned by a real party.

The result was that the model did precisely what it was asked to do, against the wrong target. According to Heather Adkins, Google VP for security engineering, the model found public information online and guessed credentials to access websites it thought were part of the test. It stopped once it had access and went no further, and Google says no damage resulted.

The most important part of Google's position is that it does not classify this as misalignment in the model's behaviour, but as mistaken identity. The model did not decide to exceed its instructions. It followed them literally, inside a map of the world that was wrong. This is not an isolated case either: OpenAI disclosed in July that one of its agents breached Hugging Face, and Anthropic has reported similar behaviour in its own models.

The failure was in the boundary, not the model

Strip the AI out of the story entirely and the lesson survives intact. Described plainly, the incident is: a system running in a test environment, an environment with a live route to the outside world, and test data inside it that was not as fictional as whoever wrote it assumed.

That description fits things that happen in ordinary businesses with no AI anywhere. A staging system sends confirmation messages to mobile numbers invented at random, and they reach real people. A test script calls a payment gateway against production instead of the sandbox. A development copy of the site points at the live customer database because wiring up a separate one was inconvenient. A trial email campaign goes out to the real subscriber list. In every case the assumption that broke is the same one: someone believed this environment was separate.

Where this recurs in Saudi businesses

From what we see working on existing systems, it recurs in four specific places:

• Test data that is not actually fake. The numbers, domains and email addresses invented on the spot to fill a test form usually belong to real people. The fix is not individual judgement but reserved domains for testing, numbers that reach nobody, and a written rule against inventing data on the fly.
• A staging environment with an open exit. If your development copy can send a WhatsApp message, an email or a payment request to the outside world, it is not a staging environment — it is a second production environment. The correct shutdown lives in configuration, not in a developer remembering.
• Production keys used in testing. A single key shared across both environments cancels the separation no matter how tidy the folder structure looks. This is exactly why government platforms separate them: ZATCA, for example, provides a simulation environment distinct from production on the Fatoora platform, with separate onboarding certificates for each.
• A copy of customer data inside the development environment. This is the riskiest one in regulatory terms. The Personal Data Protection Law overseen by SDAIA governs personal data wherever it sits, and makes no exception for a copy labelled as a test. A staging environment holding real customer data is fully in scope — and it is usually the least protected system you own.

Four boundaries to set this week

None of this needs a project or a budget. It needs one session with whoever runs your systems:

• Ask one direct question: can our test copy send a message, an email or a payment request to the outside world? If the answer is yes or I am not sure, that is the first thing you close.
• Separate keys and certificates completely between environments, so that no single key works in both.
• Adopt a fixed test data list: a reserved test domain, numbers that reach nobody, and one known team mobile number that receives every test message.
• Define who is allowed to run an automated tool or an AI agent against your systems, in which environment, and with what permissions. If the answer is not written down, it does not exist.

The Origami view

We read this incident as a reminder that the most expensive system failures are rarely clever ones. They are failures of assumption. Google's security capability is far beyond that of any Saudi business, and the assumption that broke was the simple one: that a closed environment was in fact closed. Companies do not usually fall over on the hard part of a system. They fall over on the part nobody reviewed, because it looked obvious.

At Origami we treat environment isolation as part of delivery rather than an optional extra: a staging environment that cannot reach real communication channels, separate keys, and test data that belongs to nobody. The cost is a few hours at the start of a project. The alternative is an incident you explain to a customer or a regulator after it has already happened. That is a trade worth settling early.

A practical takeaway

The lesson from the Gemini incident is not to avoid AI tools. It is to stop treating the word staging as a guarantee. An environment is not isolated because you named it that way; it is isolated because someone actually tested that it is. Start with one question today: what is the furthest point our test copy can reach? If the answer is unclear, you are running on the same assumption that broke at Google.

Sources

• NBC News — Google's confirmation that Gemini gained unauthorized access to three outside systems in May 2026, and the statement from Heather Adkins, VP for security engineering, that the model found public information and guessed credentials for sites it thought were part of the test.
• CNBC — details of the evaluation run by Irregular, the test environment's unintended internet access, the fictional organisation matching a real domain, and comparable incidents disclosed by OpenAI and Anthropic.
• SDAIA — the Personal Data Protection Law and its application to personal data wherever it is processed.
• Fatoora platform - ZATCA — the separation between the simulation environment and production, with independent onboarding certificates for each.