What is your AI actually reading?

Decision one: how the data reaches the model
There are really only three delivery patterns, and they trade off freshness, history and how much infrastructure you agree to run.

1. Batch into a warehouse, lake or object storage. Your integration layer extracts from source systems on a schedule and lands the result somewhere your embedding or retrieval jobs already read. This is the default for RAG corpora, fine-tuning sets and any retrieval that has to reason across time.

• Pros: the deepest history, so the model can compare periods and see how something changed. Every tool that speaks SQL or reads files can consume it, and the corpus stays under your control.

• Cons: freshness is never better than the extraction schedule, so between runs the model is confidently out of date. You operate and pay for the storage.

2. An operational replica kept current by change data capture. Log-based CDC reads the database transaction log instead of repeatedly querying production, keeping a replica continuously close to live state. This is the pattern for latency-sensitive operational AI: a support assistant that has to know the status of an order at the moment the customer asks.

• Pros: near-real-time state without putting query load on the system of record, and deletions are captured rather than silently missing.
• Cons: you get the current state of the replicated tables, not deep history. Log-based capture is available mainly for the major relational databases and needs configuration on the source side, and the replica is one more database to run.
  

3. Direct retrieval with no storage layer at all. The integration platform holds the latest extraction and serves it over a REST interface (JSON or CSV) or a columnar format such as Apache Arrow, increasingly exposed to assistants through the Model Context Protocol, the emerging standard for connecting AI clients to external systems.

• Pros: nothing to operate and the fastest route from question to answer, which makes it the natural fit for agents that need current numbers and nothing else.

• Cons: recent extractions only, so no long history and no heavy joins across sources. You inherit the provider's rate limits and quotas, and the data sits in someone else's cache.
  

These combine well. A support assistant can answer order-status questions from the replica while its knowledge base is re-embedded from the warehouse, and a marketing agent pulls last night’s campaign numbers directly. What you should not do is pick one by accident and then be surprised by its freshness profile.

Decision two: where personal data gets removed
This one has a deadline, and the deadline is before anything gets embedded.

Once a value has been written into a vector index or absorbed into model weights, removing it is not a delete statement. The European Data Protection Board made the regulatory version of this point in Opinion 28/2024, adopted on 17 December 2024: a model trained with personal data cannot, in all cases, be considered anonymous, and any claim of anonymity has to be assessed case by case, including the likelihood that personal data can be extracted through queries.

The practical answer is to handle it at extraction, where there are two blunt and reliable instruments. Exclusion simply never pulls the column: names, addresses, identifiers stay in the source system and cannot leak from a place they never reached. Deterministic hashing replaces the value with a one-way hash, so the same input always produces the same output. Your pipeline can still join, deduplicate and match on that column without anyone downstream seeing the original.

Do it once, in the pipeline, and you are not solving a privacy problem. You are preventing one, permanently. Every system that reads that destination inherits the decision: the warehouse, the index, the assistant your team is piloting this quarter, and the ones nobody has designed yet.

Decision three: what travels alongside the data
Two things should arrive with every row: a guarantee that it is not broken, and enough context to explain what it means.
The first is a validation gate at the flow level rather than a check after the fact. Data quality failures are worse for AI than for dashboards, because a chart that flatlines gets noticed by a human, while a retriever handed a table full of nulls and zeros will cite it, fluently. Column-level rules for nulls, zeros and anomalies can run in one of two modes. In blocking mode, records that fail never land in the table your embedding job reads, so a broken export upstream cannot silently poison the index. In monitoring mode the data proceeds and you are notified, which is also a useful signal for deciding when to re-index. Check that the same rules apply to manual backfills, with a conscious override rather than a silent bypass.

The second is the one that quietly decides whether your AI project takes three weeks or three quarters: business context, carried by the pipeline itself.

Consider a CRM. Your integration layer lands a column called dealstage holding values like qualifiedtobuy and closedwon, next to a custom field named f_status_c. To a model, those are strings. To your revenue team, one of them is the difference between a forecast and a fantasy. Somebody has to explain that f_status_c is the renewal risk flag, that it is only maintained for enterprise accounts, and that stage names changed last spring. The same is true for every source you connect: the ad platform where "conversions" means something specific, the support tool with three different timestamps for the same ticket.

You can supply all of that by hand, in a semantic layer that one person maintains and everyone else quietly distrusts, and redo the work each time a CRM admin adds a field. 

Or the pipeline can carry it, because the connector already knows: dataset descriptions of what a table represents, field descriptions of what each column means, and sensitivity flags marking which ones hold personal data. 

Add the technical metadata too, an extraction timestamp so jobs process only what is new, and a stable row hash to serve as a natural key for deduplication and updates.

That is the difference between an agent that guesses at your schema and one that reads the definition, and between governance checks that someone runs manually and checks that run automatically against what actually landed. It costs a checkbox at source configuration time. It saves a data dictionary nobody wanted to maintain.

Six questions worth asking your own stack

1. How fresh is an answer, honestly? Freshness is extraction schedule plus index refresh, not a property of the model.

2. Where does personal data get removed? If the answer is "in the vector database", it is already too late.

3. What happens when an upstream export breaks? Does the bad data land, or is it blocked before the index sees it?

4. Can the pipeline explain itself: timestamps, stable keys, field descriptions, sensitivity flags?

5. If an assistant queries your data directly, what else can that access reach?

6. How many separate tools does it take to cover all of the above today: one for extraction, another for change data capture, another for masking, another for validation, another for the catalogue? Every hand-off between them is a seam where freshness, context or governance quietly falls out, and a stack that does it in one place is worth more than the sum of its features.

None of this is the exciting part of an AI programme, and none of it will get you applause in a steering committee. But it is usually what separates the pilot that impressed everyone in the demo from the one that actually made it into production, survived a security review, and is still trusted by the business six months later. If you are the person accountable for that outcome, the data path is not a detail you can delegate to the last sprint.

This blog post is contributed by Dataddo, a modern data integration platform built for the AI era. Dataddo helps organizations securely connect data from a wide range of sources for AI, analytics, and reporting, while maintaining full control over their data and avoiding vendor lock-in. Learn more at www.dataddo.com or visit Dataddo at Data Expo.