Skip to main content

Untrusted content and prompt injection

A model reads the text a tool publishes and the text a tool returns, and it may follow an instruction it finds in either. The table lists the text that reaches a model through GATool and who writes it:

What a model readsWho writes it
Tool names, titles and descriptionsThe operation files in your repository. A file without a comment takes its description from the schema. Under gatool.dev.experimental.generate-tools the schema names and describes every generated tool
The input schema, and the output schema where a tool publishes oneThe variables of the operation file, with the descriptions, enum values, defaults and deprecation reasons of the schema, and the scalar fragments of your configuration. A scalar fragment is a piece of JSON Schema configured under scalar-schemas
ResultsThe API. A result holds the data the operation selected. Part of that data is text the users of the API wrote, such as the comment of a review
Error textThe API: the errors of a GraphQL response, and the first 2,000 characters of the body of a refused call
The Model Context Protocol (MCP) instructionsGATool, as a fixed text, or your spring.ai.mcp.server.instructions

The owner of the API writes the schema. A schema fetched from a registry is written by whoever can publish to that registry. Access to the registry therefore decides what a model reads as tool text.

What GATool does.

  • A tool sends the document its operation file holds. The model chooses a tool and writes its arguments. The arguments travel as GraphQL variables beside the document, so GraphQL text inside an argument stays a value. executeGraphql is the one tool through which a model writes a document of its own. It is one of the dynamic tools, present while gatool.dev.experimental.generate-tools is dynamic-three-step.

  • A result is written again as data and errors. Its strings stay as the API returned them. GATool replaces the value of gatool.api.credentials.header-value wherever a result or an error text holds it. It also replaces a bearer token in the errors and in the body of a refused call. An application's own ApiCredentialStrategy bean may send another header, such as an API key under its own name. Both rules leave that header alone. An API or a proxy that echoes the request into an error then hands the value to the model. Such a strategy therefore needs an API that keeps request headers out of its errors.

  • gatool.api.max-response-size bounds what GATool reads from the API. gatool.results.max-characters bounds the text one call returns to a model.

  • The default folder of the schema cache is checked before it is used. "Reading the schema from a registry" describes the check.

  • On the MCP endpoint a call needs the scopes its operation file lists. The token of a caller therefore bounds the tools an injected instruction can reach.

  • A mutation is a tool like a query. Every tool publishes readOnlyHint and destructiveHint. A query carries readOnlyHint: true, and a mutation carries destructiveHint: true, so a client can tell the two apart.

What GATool leaves to the client and the application.

  • GATool does not look for instructions in a result, in an error text or in the schema. The client or the application that holds the model treats what a tool returns as untrusted input. Scopes bound what such text can cause. Give the tools that write a scope of their own. Leave that scope out of the token of a caller whose model reads user-written text.

  • An error's extensions reach the model as the API wrote them. Some APIs put an exception class or the lines of a stack trace there. The configuration of the API is where a team switches that off.

  • GATool does not ask a person before a mutation runs. Over MCP that question belongs to the client. MCP asks a client to keep a person in the loop and to confirm a sensitive operation. The hints above tell a client which tools write.

  • In process, Spring AI's ToolCallingManager calls the callback that a model's response names. A mutation in the in-process folder, gatool/in-process/, therefore runs as soon as the model asks for it. Spring AI's ToolMetadata carries returnDirect alone. The application reads which tools write from the catalog. GAToolCatalog.inProcessTools() lists each tool with readOnly(). An application that wants a person to agree first wraps the callbacks of those tools in a ToolCallback of its own. That callback asks the person, then hands the call to GATool's callback.