Thanks for replying!
I follow what you're saying, but one thing I may suggest changing is the way "file" data is reported and recieved.
For the tool result you refer to, I feel it would make more sense to have a new URIResource which can refer to a URI, but importantly does not contain the content blob.
The client is then free to fetch that content blob from the Server if it feels the need to do so.
I see some concepts in the schema that already relate to this kind of behavior, namely ReadResourceResult, and ReferenceResource. These seem like they can be borrowed to make what you describe possible fairly easily.

As for the client to provide a file (or resource) to a server, I would imagine, a similar mechanism must be reported to the server in order to process.

• Model sends a tool request with a URI
• Server can query client for the given URI and retrieve a blob to work with

I believe I need to sit with the protocol for longer to fully understand its limitations, but this seems logical to me right now.

The MCP provides document conversion / text extraction. The result may be huge and must not be loaded directly into LLM context.
That is why it is passed back as a file, stored in a virtual filesystem inside the client.
The LLM then accesses it piecemeal by using additional tools.

The whole process is completely transparent:

1. The user drags a file into the client prompt.
2. It is stored in the virtual file system.
3. Once the user asks questions about it, the LLM tries to access it
4. The LLM realizes that it is non-text and invokes the MCP converter, which places a text file in the file system (as described above)
5. The LLM tries to access the converted file and realizes it is to big
6. The LLM then invokes additional tools (default ones, directly integrated into the client) to do the piecewise access.

So you're depending on the client LLM to recognize all these facts and do the right thing?

Doesn't sound reliable...

Well, yes and no….
The trick is to give the LLM guidance without having to actively manage control flow:

• the file reader tool, when instructed to read a read a file will yield a „file format is not text“ error on the pdf
• it will yield a „file is too big„ error if the file is Text but larger than, say, 50K

this in turn motivates the llm to invoke the file converter tool and the file splitter tool, respectively

It actually works surprisingly reliable, if you’re willing to let the llm trigger those errors once rather than getting it immediately right

In the case of producing a file, I wonder if the server couldn't output to the recently-created output, then return text/markdown with a link to the file either as a remote URL or a local file:/// URL?

At least then the user could have something to click on to view? Especially if this is an endpoint in the LLM tool usage chain?

That's how it's normally done. However I'm bound by company policy that confidential information must not be physically stored in the AI subsystem. This is why I'm trying to pass the data directly to and from the tool and only keep it in session memory.

BTW here's what the LLM does with this toolset:
the query_file tool accesses chunked embeddings that have been asynchronously built as soon as the user uploaded the file.
(I agree, it's quite amazing how the LLM reliably figures all this out by itself)

{ContextElement: actor=ASSISTANT, content="How can I help you?"}
{FileData: actor:USER, content="[context][file id=69d767d0-M-Gruppe-FIBE-FER-2024_de.pdf name=M-Gruppe-FIBE-FER-2024_de.pdf type=application/pdf]"}
{ContextElement: actor=USER, content="how many employees did migros have in 2023"}
{ToolInvocation: id=call_1uEt2EceHOJAp3ZfD9u1kgJz, name=query_file, args={"id":"69d767d0-M-Gruppe-FIBE-FER-2024_de.pdf","query":"number of employees in Migros in 2023"}}
{ToolResult: success=false, id=call_1uEt2EceHOJAp3ZfD9u1kgJz, content=[Error] File ID: 69d767d0-M-Gruppe-FIBE-FER-2024_de.pdf - File is not text.}
{ToolInvocation: id=call_dmCQzDZwMzAgwciB0J2Vk788, name=convert_to_text, args={"file_id":"69d767d0-M-Gruppe-FIBE-FER-2024_de.pdf"}}
{ToolResult: success=true, id=call_dmCQzDZwMzAgwciB0J2Vk788, content=[File] File ID: 8140311b}
{ToolInvocation: id=call_GAKkDyNftVgkV2Tw65nppRve, name=query_file, args={"id":"8140311b","query":"number of employees in Migros in 2023"}}
{ToolResult: success=true, id=call_GAKkDyNftVgkV2Tw65nppRve, content=[Content] File ID: 8140311b - Migros Kompakt > Kennzahlen 5...}
...
{ContextElement: actor=USER, content="give me a summary of the whole document"}
{ToolInvocation: id=call_O90EGlmx1kRIWIDBJiLoKaKY, name=split_file, args={"id":"8140311b"}}
{ToolResult: success=true, id=call_O90EGlmx1kRIWIDBJiLoKaKY, content=File ID: 8140311b split into 18 parts:
8140311b:0
8140311b:1
...}

The pdf extraction generates markdown, as it allows the llm to better analyze (or "vibe" ;-) the content (hierarchy of headings, for example).

The code above is just a proposal, currently there is not even a FileContent return type, only Text and Image.
The ImageContent return type already implements mimeType, the proposal essentially only adds a file name and allows for the type to be literal 'file'.

from SDK types.py:

class ImageContent(BaseModel):
    """Image content for a message."""

    type: Literal["image"]
    data: str
    """The base64-encoded image data."""
    mimeType: str
    """
    The MIME type of the image. Different providers may support different
    image types.
    """
    annotations: Annotations | None = None
    model_config = ConfigDict(extra="allow")

thx for pointing out the PR

You're inherently limited by the protocol for this use case. If you have two separate tools, then the data has to pass through the LLM and do inference. Architecturally this is going to be bad practice - your data may get modified during the inference and you won't know about it.

Some alternatives:

• write one single tool that does both tasks
• update both tools to use a filesystem - e.g. the first tool stores the file to disk, and the second tool reads it from disk again

Please see SEP-1708 which I have introduced and am taking ownership of. I believe it is related and may address the needs mentioned in this thread. Would love to hear your opinion(s).

#1708

@ravi0613 The core bottleneck here is that we’re asking an LLM to edit a very small portion of an Excel file, while LLMs operate under strict context window limits. Converting the entire Excel sheet into a JSON string and asking the model to modify it is both inefficient and unnecessary.

Instead, I’d propose introducing a local MCP server that exposes domain-specific tools for Excel manipulation. These tools would use the Excel API (or a suitable library) to programmatically modify the sheet.

For example:

1. Need to compute the average of column 2? → Expose a dedicated tool for that.
2. Need to insert or update a row? → Expose a tool for row-level mutations.

All the business logic and file-level operations would live inside the MCP server, while the LLM would only orchestrate which tool to call and with what parameters. This keeps the LLM out of large text processing, avoids context window issues, and results in a more scalable, deterministic, and maintainable system.