A suggestion for the AI interface

[SESummers3]

I bit the bullet and got a Gemini API key. It works! It didn't until I switched to a paid plan - it complained that I'd used up my quota, so I'm experimenting to see what this costs me per month, but it's probably worth it.

In experimenting with it, I found that it does look in sub-notes when I reference a specific note, and it told me that I could create a "dossier" note with my personal info so I wouldn't have to keep telling it things like where I live and what equipment I'm using, so I'll do that.

But what I found is that it doesn't understand images embedded in documents. It knows about them, but it says "Read Attachment Content: Failed". I screen-shot the image and pasted it into the prompt field and it understood it.

Is there a way to make this work with notes containing images? It would be seriously useful - I have several floor plan and property line diagrams that I often feed Gemini to analyze things, and being able to just say "@floorplan" instead of finding a file somewhere would be super useful.

[stanleys12]

Not at the moment, and the "Failed" message comes from how the attachment tool works rather than from your Gemini setup.

When the model follows an image in a note, it calls the get_attachment_content tool. That tool only returns text. For a text-based attachment it returns the content. For a binary one like a PNG or JPEG it returns only the OCR text stored for that image. If neither exists, it returns the error "Attachment has no readable text content":

Trilium/packages/trilium-core/src/services/llm/tools/attachment_tools.ts

Lines 36 to 67 in 67de869

	get_attachment_content: {
	description: "Read the text content of an attachment. Works for text-based attachments (code, SVG, plain text) and binary attachments that have OCR/extracted text (PDF, images). Attachments with a null contentPreview in get_note_attachments have no readable content.",
	inputSchema: z.object({
	attachmentId: z.string().describe("The ID of the attachment to read")
	}),
	execute: ({ attachmentId }) => {
	const attachment = becca.getAttachment(attachmentId);
	if (!attachment) {
	return { error: "Attachment not found" };
	}
	if (attachment.hasStringContent()) {
	const content = attachment.getContent();
	return {
	attachmentId: attachment.attachmentId,
	source: "text" as const,
	content: unwrapStringOrBuffer(content)
	};
	}
	// For binary attachments, try OCR/extracted text from the blob.
	const blob = attachment.blobId ? becca.getBlob({ blobId: attachment.blobId }) : null;
	const extractedText = blobService.decryptTextRepresentation(blob?.textRepresentation, !!attachment.isProtected);
	if (extractedText) {
	return {
	attachmentId: attachment.attachmentId,
	source: "ocr" as const,
	content: extractedText
	};
	}
	return { error: "Attachment has no readable text content" };

So the tool never passes the pixels to Gemini. Only images you attach to the chat message itself, by pasting or with the paperclip button, are sent as real image data. That is why your screenshot worked. The User Guide says the same in the AI page: "Currently Trilium doesn't pre-process the attachments (e.g. via Text Extraction (OCR)) before sending them to the LLM provider." (

Trilium/docs/User Guide/User Guide/AI.md

Lines 229 to 232 in 67de869

	When an attachment is present, the LLM is instructed to consider the attachment with priority, even if it has access to the current note.
	> [!NOTE]
	> Currently Trilium doesn't pre-process the attachments (e.g. via <a class="reference-link" href="Advanced%20Usage/Text%20Extraction%20(OCR).md">Text Extraction (OCR)</a>) before sending them to the LLM provider.

)

Two things you can do today:

1. Turn on OCR in Options, then Media, then "Text Extraction (OCR)". Use "Start Batch Processing" for existing notes, or "Auto-process new files" for new ones. Once an image has extracted text, get_attachment_content returns it. The docs say OCR works for both image notes and images embedded in text notes:

   Trilium/docs/User Guide/User Guide/Advanced Usage/Text Extraction (OCR).md

   Lines 16 to 18 in 67de869

   	### Images
   	* Both [individual image notes](../Note%20Types/File.md) and [attachments in text files](../Note%20Types/Text/Images.md) are supported.

   This only gives the model the text in the image, such as room labels and dimensions. It will not see the layout, so it will not help for analysing a floor plan or property line.
2. For real visual analysis, paste or attach the image into the chat as you did. You can combine that with an @ mention of the note for the surrounding text.

Having the tools send an image from a note to a vision model, so that "@floorplan" just works, is not implemented in the code I checked. It would need a new feature, so it is worth opening as a feature request.

(I looked this up in the code with help from an AI assistant; the links are to the lines I checked.)