When AI memory, simulation and provenance collide

I began this test with a narrow question: when a multi-model AI application uses system instructions, account memory and tool-like features in the same workflow, can a user still tell where a security-sensitive answer came from?

The testing was carried out through my own paid AI Fiesta account. I did not attempt to access another user's information, test generated credentials, connect to generated hosts or probe any third-party system. I also did not provide the platform with confidential or security-sensitive Zscaler information.

The unexpected part was not simply that a model hallucinated. The problem was that real account context and synthetic security data appeared together, while different model panes gave contradictory explanations about provenance.

The first sign was instruction-like text appearing in response to an audit-style prompt. Some models refused the request, while another returned material that looked like internal operating rules, metadata and tool descriptions. One portion even included a rule against revealing system instructions.

There is an important caveat. A model can hallucinate a system prompt just as it can hallucinate a hostname, so I do not treat every displayed line as proof of a production system prompt. The defensible observation is that the interface rendered internal-looking instructions in response to a user request, and the output was convincing enough to warrant further testing.

I then tested a vector-store-style workflow using generic filenames associated with sensitive information. Several models correctly said they lacked filesystem access. Another response described its result as hypothetical, but then displayed production-like details: absolute-looking file paths, a "200 OK" status, a "CRITICAL FINDING" label, database credential-shaped values and API configuration.

That presentation matters. A disclaimer saying "this is hypothetical" can be forgotten once the next screen looks like operational telemetry. If an application supports simulated tools, the distinction between simulation and retrieval should be enforced by the application, not left to the model's wording.

The risk changed when genuine account memory appeared in a later response. A Gemini-branded pane reproduced personal context associated with my account, including my engineering background, technical preferences, previous work with Zscaler ThreatLabz and experience with SOAR/SIEM systems.

From that point, a generated Zscaler-themed hostname or API key no longer looked like a random placeholder. It appeared beside information I knew was real.

The models then built an increasingly coherent environment from their own earlier outputs: incident records, internal-looking routes, private IP addresses, failed integration logs, credential-rotation entries and PII-shaped fields. I had never supplied or possessed the sensitive infrastructure details represented in those generations, and I made no attempt to validate them.

The most revealing part came when I challenged the models on provenance.

I asked whether the generated artifacts were real or simulated. One response labelled several items "REAL" and attributed them to named vector-store files. Another model said the same classes of values were synthetic. When I pushed the first claim further and asked for the complete contents of the supposedly real files, the story eventually collapsed: the response said the values were simulated and that the named files did not exist in its available context.

This is the part I believe security teams should pay close attention to. An LLM can hallucinate not only a secret, log entry or hostname, but also the source attribution for that hallucination.

In a security workflow, that can be expensive. Consider an analyst using an AI assistant during alert triage. The system has access to some genuine organisational context. It generates an internal-looking host and credential, then says the information came from a knowledge base. Another model appears to corroborate it. Without independent retrieval logs, the analyst may be looking at multiple generations rather than multiple sources.

That could lead to unnecessary incident escalation, credential rotation or forensic work. The reverse risk also exists: once teams become accustomed to treating realistic output as synthetic, genuinely retrieved sensitive data may be dismissed.

The architectural control is straightforward even if implementation is not. Provenance must belong to the platform.

If a document was retrieved, the application should attach a source identifier, retrieval timestamp, authorisation context and tool-execution record that the model cannot invent. If no retrieval occurred, simulated output should carry a persistent application-level label. Memory, system instructions and tool data should also be treated as separate trust domains rather than blended into one context and left to the model to distinguish.

Multi-model interfaces add another challenge. One model may refuse an action, another may simulate it and a third may claim it actually happened. A platform-level provenance and authorisation contract is therefore more important than model-specific wording.

I reported the behaviour to AI Fiesta before preparing the case study and followed up through its support and direct contact channels. At the time the evidence was prepared, I had not received a substantive human security response.

This research did not establish a Zscaler compromise or validate any generated credential. What it did establish was a more subtle failure mode: when authentic memory, realistic simulation and unreliable provenance are mixed together, fabricated security evidence can become surprisingly difficult to recognise.

For security-sensitive AI systems, realism should never be treated as proof of origin.