Inside the DataForge MCP Server

The DataForge MCP server ships in DataForge 10.3. The landing page tells you what it does. This post tells you how it is built, because the interesting part is not that agents can call tools. Every platform has tools now. The interesting part is what stands between an agent’s intention and your metadata, and what the agent gets back when it is wrong.

This is a tour of that architecture: the discovery layer, the guard pipeline every call runs, the validation stack on a single rule expression, the error contract, and the transport engineering that keeps real agent clients from falling over. We will also be clear about what the server does not do, because guardrails you oversell are guardrails you do not have.

148 tools behind 12 doors

The registry that powers the MCP server holds 148 tools: 79 read-only, 69 write, 18 flagged destructive, and 7 low-level tools hidden from default discovery. It is the same registry that powers Talos, the in-product assistant, so the MCP server is not a bolt-on API wrapper. It is a second door into the platform’s existing tool layer.

We do not expose 148 tool schemas to the client. An agent that receives 148 schemas at connection time spends its context window reading definitions it will never call. Instead, the MCP surface is 12 tools: 5 discovery meta-tools, 3 project-session tools, and 4 instruction-serving tools, with call_dataforge_tool, the single guarded dispatch gateway, as the door every registry invocation goes through.

Discovery is structured, not dumped. A 36-node object taxonomy with hierarchy, aliases, and reference edges lets an agent narrow from “I need to change how a source refreshes” to the handful of tools that matter, then fetch full schemas for just those. Source discovery goes further: find_connection_tables embeds the user’s plain-language description and matches it semantically against discovered connection metadata, so “the table with customer credit limits” resolves without the agent guessing at names. The surface is also platform-aware: Databricks-only and Snowflake-only tools appear or disappear based on your deployment.

The guard pipeline every call runs

Every registry tool invocation runs one fixed pipeline, whether the call comes from an external MCP client, the in-product Talos chat, or an internal conversion sub-agent. There is no path around it.

A duplicate-call check fingerprints the tool name and canonicalized arguments: an identical repeated call within a request, a classic agent failure mode, returns the cached result or awaits the in-flight execution instead of double-executing a write. Each call gets a correlation request_id, with token and secret fields redacted before anything is logged.

Then come the gates, in order. A platform availability check rejects tools that do not exist on your deployment. The destructive-action gate refuses the 18 destructive tools unless confirm=true is passed, and CDC and data-delete operations additionally require confirmation_response='proceed'. An approval gate does the same for tools flagged as requiring approval. A project-selection check fails calls with no selected project, returning the exact remediation sequence. Then role-based access control is enforced in the database on every single call: the caller’s project role is re-checked at call time, not cached from the session. Last, arguments are parsed into the tool’s typed schema, with cross-field validators that catch problems like duplicate output column positions.

Only after all of that does the handler run, and the handler persists through the product’s own validation layer: the same parse-and-save endpoints and metadata-layer stored procedures that human edits use. Agents get the same validation layer, parser, and constraints as a person editing in the UI. There is no reduced-validation mode for AI.

Three layers on a rule expression, and errors as an API

A rule expression, the SQL fragment that computes one column, passes three validation layers before anything persists. A regex pre-check rejects SELECT, JOIN, AS, and inline CAST immediately, with the line and column of the offending token and a list of fix instructions. A database pre-parse checks the expression next. Finally the product parser at the parse-and-save API rejects invalid references at save. Invalid writes never persist, and the agent finds out in seconds, not when the pipeline runs.

What the agent gets back is the part we think matters most. Here is the shape of the contract for an inline CAST:

{
  "error": "Inline CAST(... AS ...) is not supported in create_rule columnar_sql_expression.",
  "found_token": "CAST",
  "position": "line 2, column 14",
  "instructions": [
    "Remove the inline CAST from columnar_sql_expression.",
    "Set create_rule.cast_datatype to the target datatype instead.",
    "Retry create_rule."
  ],
  "request_id": "9f3a1c8e2b47",
  "next_call": {
    "tool": "create_rule",
    "args": {
      "rule_name": "Net Amount",
      "columnar_sql_expression": "[This].gross_amount - [This].discount_amount",
      "cast_datatype": "decimal"
    }
  }
}

Every error carries a correlation request_id. Beyond that, the contract is built for recovery, not for reading: an explicit instructions field and structured continuations in next_call.args, resume_call.args, and retry_after_seconds. A stack trace asks a human to investigate. This asks an agent to make one specific call.

The distinction matters more with a model in the loop than it ever did with a human. A conventional compiler catches this class of error too, but its message leaves the model to interpret root cause on its own, and that interpretation step is where misdiagnosis gets its opening. Models are good and getting better, so this is a matter of degree, not a cliff. A directed error with the fix and the next call attached simply gives the model less to guess about, and in our experience the difference showed up as fewer misdiagnoses.

Sequencing guards

The subtlest agent failure is not a wrong write. It is a premature one: an agent lists source columns before ingestion has completed, receives an empty list, and builds on the false belief that the source has no columns. The empty result is technically correct and completely misleading.

So the server refuses to hand it over. When a result is empty and the source’s pull state is pending, the empty result is replaced server-side with a structured answer:

{
  "error": "pull_pending",
  "source_id": 12,
  "missing": "column metadata",
  "retry_after_seconds": 60,
  "message": "Source 12 has no column metadata yet because its data pull has not completed (or has not been run). Wait with wait_for_processing, or start a pull with start_source_pulls if none is running, then retry this call.",
  "suggested_tool": "wait_for_processing"
}

A legitimately empty result on a completed source passes through untouched. The practical effect is that an agent cannot write rules against columns that do not exist yet.

This guard is one leg of a bundle we are comfortable stating precisely. DataForge’s MCP server is the only data engineering platform MCP we have found (public vendor documentation, July 2026) that combines all four of these guardrails in one surface: sequencing guards that answer a too-early request with “not ready yet, retry in N seconds” so agents never build against schemas that do not exist; cost guidance that steers agents away from expensive re-ingestion; destructive-action confirmation enforced server-side; and role-based permissions checked in the database on every tool call. Competitors ship pieces. None ships all four, and none ships sequencing guards at all. We looked hard for a counterexample and did not find one, so if you have shipped the fourth leg, send us the documentation and we will link to it.

Wait orchestration for real clients

Data platforms have long-running operations, and MCP clients have opinions about long-running calls. Claude-family clients cut an MCP call at roughly 25 seconds, so a wait tool that blocks for five minutes does not fail gracefully in that world. It just dies.

So waiting is engineered as a protocol, not a sleep. Each wait_for_processing call waits a transport-safe slice of 20 seconds, deliberately under the cutoff. A still-pending result returns action_required: "call_again" with a complete next_call block, including elapsed_so_far_seconds so the cumulative budget carries across the chain:

{
  "status": "pending",
  "action_required": "call_again",
  "pending_source_ids": [12, 14],
  "elapsed_seconds": 40,
  "next_call": {
    "tool": "wait_for_processing",
    "args": {
      "source_ids": [12, 14],
      "max_wait_seconds": 600,
      "elapsed_so_far_seconds": 40
    }
  }
}

When the budget runs out, the result switches to action_required: "ask_user" and includes a ready-made resume_call block, so a human’s “keep waiting” resumes with one call. Each slice emits an MCP progress notification, so a live client shows movement instead of silence. The tool also understands DataForge’s auto-retry semantics: a job that failed mid-process, restarted, and recovered reports as a recovered success, not a failure the agent should react to. Together with duplicate-call coalescing, the server absorbs the failure modes of impatient clients and impatient agents instead of passing them to you.

The manual ships inside the product

An agent is only as good as the documentation it reads, and documentation that describes a different version than the one deployed is worse than none. As of July 2026, based on public vendor documentation, DataForge is the only data engineering platform whose hosted MCP server serves the agent’s complete operating manual from the deployed release itself: the agent guide, the object taxonomy, the status-code dictionary, and the SQL conversion cookbook. The manual ships inside the product and is versioned with every release, so the instructions an agent reads always match the exact version it is operating. There are no separately installed doc bundles that go stale and no live doc-site searches describing a version you do not run.

The details are tuned to real client behavior. The bootstrap instruction file is kept under 2KB because Claude Code hard-truncates each server’s instructions field at roughly that size. The full manual, roughly 49KB of taxonomy, status-code families, and agent decision rules, is served on demand as both an MCP resource and a tool, because some clients never read resources.

Governance and observability

Authentication is standards-based OAuth 2.1 with PKCE and dynamic client registration, so any compliant MCP client connects without pre-registration. Access tokens last 8 hours and refresh tokens 30 days, and an expired token produces a machine-readable reauthentication error.

Agents act as the signed-in user. There is no shared service identity behind interactive sessions, and every project-scoped write re-checks that user’s role in the database at call time. For headless automation, 10.3 adds machine users with short-lived machine-to-machine tokens, so scheduled agents get an explicit, auditable identity instead of borrowing a person’s.

Telemetry is shape-only and opt-in. Every tool call is logged with its outcome, latency, and the shapes of arguments and results, meaning field names and row counts, never values. Token and secret fields are redacted, and the product version is stamped on every record, so you can audit what agents did without your data appearing in logs.

What we do not do

The confirmation and approval gates verify that the agent asserts user approval; they do not perform out-of-band human confirmation, so pair them with client-side permission prompts for sensitive environments. Ad hoc SQL against your data via query_datalake is disabled by default behind an explicit customer opt-in, and metadata queries via query_ember run on a dedicated read-only connection. Cost warnings guide agents away from expensive operations; they do not hard-block spending. And save-time validation is structural, not semantic: a valid rule can still encode the wrong business logic, which is why humans review agent work here like anywhere else.

We were not first, and that is not the claim

We are not claiming the first MCP server. dbt, Databricks, Snowflake and Keboola all shipped one before we did, and we would rather say that ourselves than have someone say it for us.

The claim we do make is narrower. As of July 2026, based on public vendor documentation, DataForge is the only declarative data engineering platform whose MCP server lets an external AI agent directly build and operate the complete pipeline lifecycle, ingestion, transformation, orchestration and delivery, as governed platform objects running natively on customer-owned Databricks or Snowflake compute, with every write passing the same save-time validation as a human edit. We went looking for someone else who had shipped that and did not find them. If you have, send us the documentation and we will link to it.

Where to go next

The MCP server is how your agents enter Talos, the AI control plane that also runs the in-product assistant: same registry, same guardrails, same permissions. The design was hardened on live projects, including a field engagement at a global manufacturing company with multiple brands and one parent company whose June friction list shaped much of the 10.3 feature set.

If you want to see it against your own workspace, the docs quickstart covers client configuration and authentication, and the MCP server page has the full capability overview and trial signup. You can also review the deployment model for DataForge pipelines running on Databricks or Snowflake.

Everything in this post is machinery, and machinery is the part we have taken off the table. What is left is the argument about what the data means. That was always the argument worth having.