[dustinhealy]

Summary

This pull request adds support for uploading, encoding, validating, processing, and displaying audio and video files—specifically for the Google multimodal endpoint—across both the backend and frontend. It introduces new backend services for handling audio/video files, updates file validation logic, and extends the client to allow users to upload these new file types and display them appropriately.

Note: (Requires PR 14 in agents)

Backend: Audio/Video Upload & Processing

• Added new services: encodeAndFormatAudios (api/server/services/Files/Audio/encode.js) and encodeAndFormatVideos (api/server/services/Files/Video/encode.js) to handle encoding, formatting, and validation of uploaded audio and video files for supported endpoints, including Google. [1] [2]
• Updated AgentClient (api/server/controllers/agents/client.js) to categorize attachments by type, process audio and video files along with images and documents, and include them in the message payload for endpoints that support them. [1] [2]

Validation Enhancements

• Extended validation logic in @librechat/api (packages/api/src/files/validation.ts) to include Google-specific file size limits and new validation functions for audio and video files. [1] [2]

Frontend: File Upload & Display

• Updated the file attachment menu (client/src/components/Chat/Input/Files/AttachFileMenu.tsx) to allow users to select audio and video files when using the Google endpoint. [1] [2] [3]
• Added new SVG icons and file type definitions for audio and video files, ensuring they are displayed with appropriate icons and colors in the UI (client/src/utils/files.ts, packages/client/src/svgs/AudioPaths.tsx). [1] [2] [3] [4]

Document Handling Improvements

• Enhanced document encoding to support Google endpoint-specific formatting for PDFs (api/server/services/Files/documents/encode.js). [1] [2] [3]

File Type & Endpoint Configuration

• Expanded the list of supported MIME types and regular expressions to include common video and audio formats. [1] [2]
• Added Google as a document-supported endpoint, enabling multimodal support for video, audio, and document files.

Internal Refactoring

• Exported streamToBuffer for shared use between encoding modules. [1] [2]

Screenshots

Multi-Attach Message

Message using the new Upload to Provider feature for multimodal attachments

---

Multi-Attach Response

Example of conversation parsing a multimodal message correctly

Change Type

• New feature (non-breaking change which adds functionality)
• This change requires a documentation update

Testing

Extensive manual testing of multi-round, multi-attachment conversations with Gemini-2.(0/5)-(flash/pro) models and light testing of other provider endpoints to verify no regressions

Checklist

• My code adheres to this project's style guidelines
• I have performed a self-review of my own code
• I have made pertinent documentation changes
• My changes do not introduce new warnings
• Local unit tests pass with my changes
• A pull request for updating the documentation has been submitted.