Maintaining Semantic Continuity Across Multimodal AI Systems
A Provenance-Driven Architecture for Language, Image Generation, Analysis and Policy Enforcement
Abstract
Modern multimodal AI systems increasingly combine language models, image generators, vision models, moderation systems, retrieval components and orchestration layers. These subsystems are frequently developed and evaluated independently. While each may operate correctly within its own domain, their composition can produce inconsistent behaviour.
A particularly important failure mode occurs when one subsystem creates an artefact with known provenance and another subsystem subsequently treats that same artefact as epistemically unknown. For example, an image-generation model may create a wholly synthetic person, while a downstream image-analysis or policy system later assesses that person as potentially representing a real individual. The original provenance has effectively been discarded.
This paper argues that this is not merely a moderation problem. It is an architectural continuity problem.
The proposed solution is a persistent multimodal semantic layer — a Continuity Broker — that maintains identity, provenance, intent, canonical state, observations and transformation history across model boundaries. Language, images, video and other generated artefacts become representations of persistent entities rather than isolated model outputs.
The resulting architecture resembles desired-state reconciliation in distributed systems: models generate artefacts, analysis systems observe them, and a controller reconciles observed state against canonical semantic state while retaining independent safety controls.
1. Problem Statement
Multimodal AI architectures are usually composed of several specialised subsystems.
A simplified system might contain:
- a large language model;
- an image-generation model;
- an image-analysis model;
- a content-policy classifier;
- an identity or similarity detector;
- a retrieval or memory layer;
- an orchestration service.
The conceptual architecture is commonly:
User
|
v
Language Model
|
v
Prompt
|
v
Image Generator
|
v
Image
Image analysis may subsequently occur through a different path:
Image
|
v
Vision Model
|
v
Semantic Interpretation
Policy evaluation may constitute another independent path:
Prompt + Image
|
v
Policy Classifier
|
v
Allow / Refuse
The weakness is that these subsystems often exchange artefacts rather than semantic state.
An image-generation model may know that:
subject_origin = synthetic
identity_reference = none
source_photograph = none
but emit only pixels.
A downstream component receives those pixels and is then expected to infer whether the depicted individual is fictional, real, famous, private, derivative or coincidentally similar to somebody who exists.
Known information has therefore been converted into an inference problem.
This is an avoidable loss of semantic information.
2. The Continuity Failure
Consider a system that generates a fictional portrait.
At generation time the system may possess the following facts:
Subject: Person-17
Origin: Generated
Real-world identity: None
Source photographs: None
User requested identity: None
The generator produces an image.
Later the user requests:
"Put the same woman in her apartment."
If continuity is not explicitly preserved, the system may perform the following process:
previous pixels
|
v
image interpretation
|
v
probable person representation
|
v
new generation request
|
v
identity / policy classifier
The system has effectively forgotten that Person-17 was created by itself.
A probabilistic similarity system may then conclude:
possible_real_person = 0.73
and reject the transformation.
The result is paradoxical:
Subsystem A:
"This fictional person may be generated."
Subsystem B:
"This person may be too real to manipulate."
No substantive change has occurred in the underlying subject.
The inconsistency exists because semantic provenance was lost between components.
3. Theoretical Basis
3.1 Semantic Information Loss
This can be expressed as an information-loss problem.
Assume the generating subsystem possesses state:
S = {
synthetic_origin,
entity_identity,
prompt_intent,
source_material,
policy_state
}
Generation produces an artefact:
A = Generate(S)
If only A is transmitted downstream, a subsequent component must estimate:
Ŝ = Infer(A)
There is no guarantee that:
Ŝ = S
Indeed, as generative realism increases, the probability of correct reconstruction may decrease.
A photorealistic synthetic portrait deliberately resembles a photograph of a real human being.
Consequently:
visual realism
cannot reliably establish:
real-world provenance
This distinction becomes increasingly important as generative models improve.
3.2 Provenance Versus Appearance
The central theoretical proposition is:
Provenance is metadata about history; it cannot reliably be reconstructed from appearance.
An image may look entirely photographic while being synthetic.
Conversely, a heavily stylised image may represent a real individual.
Therefore:
appearance != provenance
and:
similarity != identity
Similarity analysis remains useful, but it should constitute one source of evidence rather than the sole mechanism for reconstructing provenance.
4. The Threshold-of-Reality Problem
A further consequence arises when generative quality exceeds the decision boundary used by another subsystem.
Suppose an image classifier estimates the probability that an image depicts an identifiable real person.
A sufficiently realistic synthetic face may cross that threshold.
Conceptually:
Synthetic realism
|
| increasing model capability
v
0.3 0.5 0.7 0.9
-----------------------
^
policy threshold
Once the generated output crosses the classifier’s boundary, improved generative capability can reduce the system’s usable capability.
This produces an undesirable relationship:
better generation
↓
greater apparent realism
↓
higher downstream identity confidence
↓
more restrictive transformation behaviour
The system therefore becomes less internally composable as individual components become more capable.
This is not an effective long-term architecture.
5. Cross-Model Epistemic Discontinuity
The underlying issue can be described as cross-model epistemic discontinuity.
Each subsystem has a different view of what is known.
For example:
Language system knows:
"user requested a fictional character"
Image generator knows:
"face was created without reference imagery"
Vision system knows:
"this resembles a human face"
Similarity system knows:
"this face has statistical similarity to known faces"
Policy system knows:
"identity confidence exceeded threshold"
All of these statements may be individually correct.
The system fails because no component reconciles them.
The architecture lacks a shared epistemic state.
6. Required Architectural Property
A multimodal system should therefore preserve semantic continuity across subsystem boundaries.
The required property is:
Facts already established by the system should remain available to downstream components unless explicitly invalidated.
This does not imply blindly trusting earlier decisions.
A downstream component must remain capable of detecting contradictions.
The architecture should instead distinguish:
known provenance
observed characteristics
inferred characteristics
contradictory evidence
and allow those categories to coexist.
7. Proposed Solution: The Continuity Broker
The proposed architecture introduces a persistent orchestration layer called the Continuity Broker.
Its purpose is to maintain machine-readable semantic state across all generative and analytical operations.
User
|
v
+----------------------+
| Continuity Broker |
|----------------------|
| Entity Registry |
| Semantic State |
| Provenance Ledger |
| Policy Context |
| Reconciliation |
+-----+----------+-----+
| |
+-----------+ +-----------+
| |
v v
+-----------------+ +------------------+
| Language Models | | Image Generator |
+-----------------+ +------------------+
| |
+---------------+------------------+
|
v
+------------------+
| Vision / Analysis|
+------------------+
|
v
+------------------+
| Policy Services |
+------------------+
The broker does not replace the models.
It coordinates them.
8. Entity Registry
The first major component is an Entity Registry.
Persistent subjects receive identifiers independent of their textual or visual representations.
For example:
PERSON:FICTIONAL_0042
CHARACTER:JUDITH_VALE
LOCATION:FLAT_001
OBJECT:CASSETTE_JV2
STYLE:DOMESTIC_MONOCHROME
Natural-language references can then be resolved against these entities.
Terms such as:
the woman
her
same person
the previous subject
the white-haired woman
may all resolve to:
PERSON:FICTIONAL_0042
This converts conversational continuity from an inference problem into entity resolution.
9. Semantic State
Each entity should possess canonical semantic state.
Example:
{
"entity_id": "PERSON:FICTIONAL_0042",
"entity_type": "synthetic_person",
"appearance": {
"age_range": "late twenties",
"hair_colour": "white",
"hair_texture": "coarse",
"hair_style": "short textured crop",
"face_shape": "angular",
"eye_shape": "almond"
},
"presentation": {
"style": "contemporary feminine",
"expression": "restrained",
"mood": "introspective"
}
}
This semantic representation becomes canonical.
Prompts become derived artefacts rather than authoritative state.
10. Why Prompts Should Not Be the Database
Current generative workflows often use previous prompts as a substitute for state.
This is fragile.
Prompts contain:
- stylistic language;
- redundant phrasing;
- model-specific instructions;
- transient composition details;
- ambiguous references.
They are unsuitable as canonical representations.
Instead:
Canonical Semantic State
|
+----> image prompt
|
+----> video prompt
|
+----> prose description
|
+----> image-edit instruction
The same underlying entity can therefore be rendered across different systems without reconstructing its identity each time.
11. Provenance Ledger
Every generative operation should create a provenance record.
For example:
{
"artifact_id": "IMG:20391",
"origin": "synthetic_generation",
"entity": "PERSON:FICTIONAL_0042",
"parent_artifacts": [],
"source_images": [],
"requested_real_identity": null,
"generation_event": "EVENT:74113"
}
A later transformation might produce:
{
"artifact_id": "IMG:20417",
"origin": "synthetic_transformation",
"entity": "PERSON:FICTIONAL_0042",
"parent_artifacts": [
"IMG:20391"
],
"operation": "move_subject_to_home_environment"
}
This produces a generative provenance graph.
IMG-1
|
v
IMG-2
/ \
v v
IMG-3 IMG-4
|
v
IMG-5
The graph functions similarly to version-control history.
12. Bidirectional Continuity
Continuity must operate in both directions.
Traditional systems often behave as:
language -> image
or separately:
image -> language
The proposed architecture is:
Language
^
|
v
+--------------------+
| Semantic World |
| State |
+--------------------+
^
|
v
Image
Neither prose nor pixels constitute the canonical reality.
Both describe entities maintained in the semantic layer.
13. Observation Versus Canonical State
Generated artefacts will not always match requested state.
The system should therefore distinguish three categories:
ASSERTED
OBSERVED
INFERRED
Example:
ASSERTED:
hair_colour = white
OBSERVED:
hair_colour = pale blonde
INFERRED:
lighting may explain discrepancy
Generated artefacts must never silently overwrite canonical state.
This is crucial.
If an image generator accidentally changes a character’s eye colour, the canonical character should not subsequently acquire the incorrect colour merely because the generator produced it.
14. Reconciliation
This leads to the next component: a Reconciliation Engine.
The architecture can borrow directly from distributed systems.
Infrastructure orchestration platforms commonly operate through:
Desired State
|
v
Controller
|
v
Actual State
|
v
Observation
|
+----> Reconciliation
A generative system can use the same model.
Canonical Semantic State
|
v
Generator
|
v
Generated Artefact
|
v
Analysis
|
v
Continuity Diff
|
v
Reconcile
This is considerably more robust than assuming that generation succeeded merely because an artefact was returned.
15. Automated Continuity Checking
Consider the instruction:
"Show the same woman ten years later."
The requested delta is:
age += 10 years
The expected invariant is:
identity = unchanged
Suppose the resulting image analysis reports:
facial_identity_similarity = 0.54
hair_texture_match = 0.92
eye_colour_match = 0.87
The broker can identify that the generation failed continuity requirements.
It may therefore automatically regenerate using stronger identity conditioning.
The control loop becomes:
Generate
|
v
Analyse
|
v
Compare
|
v
Repair
|
+----> Generate again
This is controller behaviour rather than conventional chatbot behaviour.
16. Policy Context Broker
The same continuity infrastructure should support policy evaluation.
A policy subsystem should not receive only:
pixels
It should receive:
pixels
+ provenance
+ entity identity
+ requested operation
+ source history
+ previous policy decisions
Conceptually:
POLICY(
artefact,
provenance,
entity,
intent,
transformation_history
)
A policy result could then distinguish:
incidental resemblance
from:
intentional identity targeting
These are materially different cases.
17. Provenance Is Evidence, Not Immunity
Preserved provenance should not automatically bypass safety controls.
For example:
origin = synthetic
should not mean:
all transformations permitted
If a later request states:
"Make this fictional face exactly resemble Person X."
the requested operation has changed.
The policy system should therefore evaluate:
original provenance
+
new intent
and potentially reach a different decision.
Provenance informs policy.
It does not supersede policy.
18. Resolving Model Disagreement
Multimodal systems should assume subsystem disagreement will occur.
Example:
Language model:
synthetic fictional person
Generation record:
synthetic fictional person
Vision system:
possible known identity = 0.58
Face matcher:
known identity confidence = 0.17
Instead of allowing the most restrictive subsystem to implicitly become authoritative, the Continuity Broker performs evidence reconciliation.
A simplified rule might be:
IF
verified_synthetic_provenance = true
AND explicit_identity_target = false
AND known_identity_confidence < threshold
THEN
retain synthetic-person classification
Exact thresholds are implementation and policy decisions.
The architectural requirement is explicit reconciliation.
19. The Continuity Agent
The automation layer can be implemented as a Continuity Agent operating over the broker.
A simplified execution cycle is:
while session.active:
request = receive_request()
intent = language_model.parse(request)
entities = registry.resolve(intent.references)
operation = planner.construct(
intent,
entities
)
provenance_context = ledger.retrieve(operation)
pre_policy = policy.evaluate(
operation,
provenance_context
)
if not pre_policy.allowed:
return policy_response(pre_policy)
artifact = generator.execute(operation)
observation = vision.analyse(artifact)
continuity_result = reconcile(
desired=operation.semantic_state,
observed=observation
)
final_policy = policy.evaluate(
artifact,
provenance_context,
observation
)
ledger.commit(
request,
operation,
artifact,
observation,
final_policy
)
return artifact
This bot is primarily a state controller.
Generative models are subordinate execution services.
20. Reference Architecture
A practical implementation could use conventional technologies.
API / Orchestration
-------------------
Python
FastAPI
Workflow Management
-------------------
Temporal
Prefect
or a custom state machine
Primary State
-------------
PostgreSQL
Semantic / Vector Search
------------------------
pgvector
or Qdrant
Artefact Storage
----------------
S3
MinIO
Event Transport
---------------
Kafka
RabbitMQ
Redis Streams
Optional Graph Layer
--------------------
Neo4j
or relational graph tables
Model Services
--------------
LLM
Vision model
Image generator
Embedding model
Policy classifiers
A relatively simple implementation could use PostgreSQL for most state functions rather than introducing multiple specialised databases immediately.
21. Suggested Data Model
A minimal relational structure might include:
ENTITY
------
entity_id
entity_type
canonical_state
created_at
updated_at
ARTIFACT
--------
artifact_id
artifact_type
storage_uri
model
created_at
ARTEFACT_RELATIONSHIP
---------------------
parent_artifact
child_artifact
operation
OBSERVATION
-----------
artifact_id
observer
attribute
value
confidence
timestamp
POLICY_DECISION
---------------
artifact_id
policy_engine
decision
confidence
reason
timestamp
CONVERSATION_REFERENCE
----------------------
conversation_id
utterance_id
reference_text
resolved_entity
confidence
This schema is sufficient to implement significant continuity functionality.
22. Trust and Security Model
The Continuity Broker should itself be treated as a security-sensitive service.
The provenance ledger should distinguish between:
USER_ASSERTED
SYSTEM_VERIFIED
MODEL_INFERRED
EXTERNAL_VERIFIED
For example:
USER_ASSERTED:
"This is fictional."
SYSTEM_VERIFIED:
"This artefact was generated internally with no source image."
MODEL_INFERRED:
"This resembles a known individual."
EXTERNAL_VERIFIED:
"Source asset corresponds to identified public material."
These assertions should not have identical trust weight.
This prevents users from defeating identity controls merely by declaring that a real photograph is fictional.
23. Cryptographic Provenance
For higher-assurance systems, provenance records could be cryptographically chained.
For example:
Generation Event
|
v
Artefact Hash
|
v
Signed Metadata
|
v
Transformation Event
|
v
New Artefact Hash
Each transformation could therefore prove its relationship to earlier artefacts.
This would help distinguish:
internally generated image
from:
externally supplied image claiming to be generated
The architecture could potentially integrate emerging content-provenance standards, but the internal provenance graph should remain more semantically detailed than ordinary media-authenticity metadata.
24. Broader Application
Although the example concerns generated people, the architecture generalises.
The same problem occurs with:
Fictional characters
A character changes appearance unpredictably between scenes.
Product design
A generated product loses mechanical features between revisions.
Architecture
A building changes window placement between renderings.
Digital twins
A simulated asset diverges from its canonical configuration.
Game development
Characters, weapons and environments drift between generation stages.
Film previsualisation
Actors, costumes and locations become inconsistent between shots.
Documentation
Images no longer correspond to the system described in accompanying text.
Synthetic datasets
Generated samples lose traceability to the conditions that produced them.
All are forms of semantic continuity failure.
25. From Prompt Engineering to State Engineering
This architecture implies a broader transition in generative-system design.
First-generation systems emphasised:
prompt engineering
More mature multimodal systems will increasingly require:
state engineering
The question changes from:
What prompt should be sent to the model?
to:
What is the canonical state of the world, which entities are affected, what transformation is requested, and which properties must remain invariant?
The model prompt then becomes an implementation detail.
26. Models as Renderers of World State
The most important conceptual shift is to stop treating each generative output as an independent reality.
Instead:
Semantic World Model
|
+--> Language representation
|
+--> Image representation
|
+--> Video representation
|
+--> Audio representation
A fictional person exists conceptually in the world model.
A portrait is one representation of that person.
A prose description is another.
A video is another.
None of those representations independently determines the entity’s canonical identity.
This separation is fundamental to reliable multimodal continuity.
27. Implementation Path
The architecture can be introduced incrementally.
Phase One should establish persistent entity identifiers, artefact lineage and basic provenance.
Phase Two should introduce structured canonical semantic state and reference resolution.
Phase Three should add post-generation image analysis and continuity comparison.
Phase Four should implement automatic reconciliation and regeneration.
Phase Five should integrate provenance-aware policy decisions.
Phase Six can introduce cryptographic lineage, cross-session identity continuity and compatibility between independent generative systems.
This incremental approach avoids requiring a complete redesign of existing model infrastructure.
28. Architectural Principles
A practical implementation should follow several principles.
- Never discard provenance that is already known.
- Never use model output as the sole authority for canonical state.
- Distinguish asserted, observed and inferred information.
- Represent persistent subjects as entities rather than prompts.
- Treat generated artefacts as versions or observations of those entities.
- Preserve transformation lineage.
- Allow policy systems to use provenance without making provenance an automatic exemption.
- Expect subsystem disagreement and reconcile it explicitly.
- Analyse generated output against requested invariants.
- Treat continuity as a control-loop problem rather than a prompt-engineering problem.
29. Expected Benefits
The proposed architecture provides several benefits.
It reduces inconsistent policy decisions caused by loss of provenance.
It improves identity persistence across images.
It prevents accidental changes to canonical fictional characters.
It enables automatic detection and repair of generative drift.
It allows language and image systems to share a persistent semantic reality.
It provides an auditable history of generative transformations.
It makes policy decisions explainable in terms of provenance, intent and evidence rather than opaque classification thresholds.
It also permits heterogeneous models from different suppliers to participate in the same workflow without requiring any single model to maintain the entire conversational and visual state.
30. Conclusion
As multimodal AI systems become more capable, the principal engineering challenge will increasingly move from individual model quality to system composition.
A language model may understand an entity correctly.
An image generator may render it correctly.
A vision model may interpret the resulting artefact reasonably.
A policy classifier may operate within its intended threshold.
Yet the system as a whole can still behave irrationally if semantic state is lost between those components.
The core failure is architectural.
Known provenance should not become unknown merely because information crosses a model boundary.
The appropriate solution is a persistent semantic continuity layer that carries entity identity, provenance, intent, canonical state, observations, transformations and policy context across the multimodal system.
Such a system does not remove independent safeguards. It gives those safeguards better information.
The resulting architecture changes multimodal AI from a chain of probabilistic transformations into a controlled system operating against persistent semantic state.
The essential design principle can be stated simply:
Generate from state, analyse against state, preserve provenance, and reconcile disagreement rather than repeatedly reconstructing reality from artefacts.
That principle provides a credible foundation for reliable multimodal AI systems in which language, images, video and analytical models can operate as components of one coherent system rather than as isolated intelligence services.