Skip to content

Add support for images and pdf files#306

Open
Yahiewi wants to merge 1 commit intojupyterlite:mainfrom
Yahiewi:image
Open

Add support for images and pdf files#306
Yahiewi wants to merge 1 commit intojupyterlite:mainfrom
Yahiewi:image

Conversation

@Yahiewi
Copy link
Copy Markdown
Contributor

@Yahiewi Yahiewi commented Apr 2, 2026

This PR fixes #292 by fixing the way attachments are read and sent to the AI.

Main code changes:

  • In the sendMessage function, enhancedMessage is now declared as UserContent and attachmentContents is now replaced by { textContents, binaryParts }
  • Changed _processAttachments return signature changed from Promise<string[]> to Promise<{ textContents: string[] binaryParts: Array<ImagePart | FilePart>}> and replaced const contents with textContents: string[] and binaryParts : Array<ImagePart | FilePart>
  • Defined:
    -imageMimeTypes : .png, .jpg, .jpeg, .gif, .webp, .bmp (which is sent as ImagePart)
    -fileMimeTypes : .pdf (which is sent as FilePart)
  • Defined _readBinaryAttachment function to take in both images and PDFs and return their base64 content.
Screencast.from.2026-04-02.10-51-56.mp4

@Yahiewi
Copy link
Copy Markdown
Contributor Author

Yahiewi commented Apr 2, 2026

GIFs are not/only partially supported by all AIs as input. Claude, for example can see them as still images, Mistral however produces an 'Error generating response' with the following console log: AI_APICallError: Image is an animated GIF.

@Yahiewi Yahiewi marked this pull request as ready for review April 2, 2026 14:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Images are not being parsed correctly

1 participant