Skip to content

Images returned by tools (e.g. Gmail attachments) never reach the model as images #887

Open

Description

@mxmzb

What happens

An image a tool returns never reaches the model as an image. Composio's GMAIL_GET_ATTACHMENT answers { file: { name, mimetype, s3url } }. OpenBot turns every tool result into JSON text (resultOf in server/src/plugins/composio.ts, capped at 20,000 characters), and every Bot's history passes tool messages to the provider as strings. The model holds an S3 link it cannot open, so "look at the photo the customer attached" cannot work. Chat attachments the person uploads do reach the model, through inlineAttachments in server/src/copilot.ts, but only for user messages.

Suggested fix (implemented and tested in a demo build)

Providers take images in user messages, not in tool messages. So after each tool result that names an image, add a user message carrying the picture, in the same image/data part shape inlineAttachments already produces (which shared/user-content.ts turns into the provider's image part):

  • A new server/src/tool-images.ts, inlineToolImages(messages, fetch):
    • find each s3url beside an image mimetype (png, jpeg, gif, webp) in tool results;
    • download it on the server, https only, never a private or loopback host, MAX_IMAGE_BYTES per image, the newest 4 per run;
    • insert { role: "user", content: [{ type: "text", text: "The image ... returned by tool call <id> (not a message from the person):" }, { type: "image", source: { type: "data", value, mimeType } }] } right after the tool message;
    • hold downloads in a small cache, so a presigned link that has expired on a later turn still shows the picture.
    • An image that cannot be fetched becomes a sentence telling the model it is not in front of it. Never fail the turn.
  • Call it after inlineAttachments in both places copilot.ts prepares a run's messages: the built-in agent's inline hook and the remote-endpoint middleware. Each tool step is a new run, so the image is in front of the model on the step after the tool returns.

Tests: an image link is found and a non-image one is not; the picture is added straight after the tool result; a link to a private address is never fetched and says so; a history with no tool images comes back unchanged (same reference).

Activity

  1. linear-code commented on Oct 29, 2026

    @linear-code
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions