Header definitions for requests handled by Buddy

[KlimTodrik]

Proposal:

The problem is that application/json headers are currently hardcoded in searchd.

Even when calling the /cli endpoint—which doesn’t return JSON — the Content-Type: application/json header is still present:

root@232f511affc0:/workdir# curl -v 0:9308/cli -d 'show version'
*   Trying 0.0.0.0:9308...
* Connected to 0.0.0.0 (127.0.0.1) port 9308
> POST /cli HTTP/1.1
> Host: 0.0.0.0:9308
> User-Agent: curl/8.5.0
> Accept: */*
> Content-Length: 12
> Content-Type: application/x-www-form-urlencoded
> 
< HTTP/1.1 200 OK
< Server: 0.0.0 14ad759ed@25062009 (columnar 5.0.2 3813c56@25060612) (secondary 5.0.2 3813c56@25060612) (knn 5.0.2 3813c56@25060612) (embeddings 1.0.0) (buddy v3.30.2-gc16d9c)
< Content-Type: application/json; charset=UTF-8
< Content-Length: 527
< 
+------------+----------------------------------+
| Component  | Version                          |
+------------+----------------------------------+
| Daemon     | 0.0.0 14ad759ed@25062009         |
| Columnar   | columnar 5.0.2 3813c56@25060612  |
| Secondary  | secondary 5.0.2 3813c56@25060612 |
| Knn        | knn 5.0.2 3813c56@25060612       |
| Embeddings | embeddings 1.0.0                 |
| Buddy      | buddy v3.30.2-gc16d9c            |
+------------+----------------------------------+
6 rows in set (0.003 sec)

We need to provide a mechanism for Buddy to define desired headers and pass them upstream, so daemon can attach or override them accordingly

Checklist:

^{To be completed by the assignee. Check off tasks that have been completed or are not applicable.}

Details

• Implementation completed
• Tests developed
• Documentation updated
• Documentation reviewed
• OpenAPI YAML updated and issue created to rebuild clients

[sanikolaev]

We need to update the Daemon/Buddy protocol (https://github.com/manticoresoftware/manticoresearch-buddy?tab=readme-ov-file#communication-protocol-v3), so we can pass custom HTTP headers from Buddy.

[sanikolaev]

This issue blocks manticoresoftware/manticoresearch-buddy#566

[KlimTodrik]

First of all, this needs to be implemented in searchd to support custom headers. On the Buddy side, it only requires changing two lines of code

[tomatolog]

I pushed the proposed fix in the Buddy protocol description at manticoresoftware/manticoresearch-buddy#622 and fix of the daemon and Buddy communication at #3955

After the fix daemon use optional content_type string at the Buddy reply and if it set use its content for the

Content-Type: application/json

to reply to user with the proper context type.

If change in the Buddy specification is ok and Buddy will add that support then daemon from the branch #3955 already has such functionality for testing purpose.