Incorrect UTF-8 character display in stdout logs

[KhAlena]

Russian search results are correct.
When logging to a file, the display of Russian letters is also correct.
But when redirecting query.log to /dev/stdout characters looks like on image:

English letters displayed also correctly.

[sanikolaev]

Please provide more details on how to reproduce the issue and make sure there's no locale issue in the environment. So far I can't reproduce it like this:

msnikolaev@dev:~/issue_20$ cat docker-compose.yml
version: '2.2'

services:
  manticore:
    container_name: manticore
    image: manticoresearch/manticore
    restart: always
    ports:
      - 127.0.0.1:9315:9306
    ulimits:
      nproc: 65535
      nofile:
         soft: 65535
         hard: 65535
      memlock:
        soft: -1
        hard: -1
    environment:
      - QUERY_LOG_TO_STDOUT=true
      - searchd_query_log_format=plain
    volumes:
      - ./data:/var/lib/manticore

mysql> create table t(f text);
Query OK, 0 rows affected (0.01 sec)
mysql> select * from t where match('абв');
Empty set (0.00 sec)

snikolaev@dev:~/issue_20$ docker-compose up
Recreating manticore ... done
Attaching to manticore
manticore    | [Wed Jun 29 14:59:23.670 2022] [1] using config file '/etc/manticoresearch/manticore.conf' (9168 chars)...
manticore    | starting daemon version '5.0.2 348514c86@220530 dev' ...
manticore    | listening on all interfaces for mysql, port=9306
manticore    | listening on UNIX socket /var/run/mysqld/mysqld.sock
manticore    | listening on 192.168.240.2:9312 for sphinx and http(s)
manticore    | listening on all interfaces for sphinx and http(s), port=9308
manticore    | Manticore 5.0.2 348514c86@220530 dev
manticore    | Copyright (c) 2001-2016, Andrew Aksyonoff
manticore    | Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
manticore    | Copyright (c) 2017-2022, Manticore Software LTD (https://manticoresearch.com)
manticore    |
manticore    | precaching index 't'
manticore    | accepting connections
manticore    | prereading 1 indexes
manticore    | prereaded 1 indexes in 0.000 sec
manticore    | [Wed Jun 29 14:59:25.970 2022] 0.000 sec 0.000 sec [ext2/0/ext 0 (0,20)] [t] абв

[KhAlena]

docker-compose

  # Search Service
  tamaranga-manticore:
    container_name: tamaranga-manticore
    image: tamaranga-manticore:${TAG:-latest}
    build:
      context: ./docker/manticore/
      dockerfile: Dockerfile
    restart: unless-stopped
    tty: true
    ulimits:
      nproc: 65535
      nofile:
        soft: 65535
        hard: 65535
      memlock:
        soft: -1
        hard: -1
    volumes:
      - tamaranga-manticore:/var/lib/manticoresearch
    networks:
      - internal-network
    env_file:
      - .env

Dockerfile

FROM manticoresearch/manticore

ARG RU_PACK_URL=https://repo.manticoresearch.com/repository/morphology/ru.pak.tgz
ARG EN_PACK_URL=https://repo.manticoresearch.com/repository/morphology/en.pak.tgz
# Locales installation
RUN apt-get update
RUN apt-get install -y locales-all locales curl xz-utils manticore-columnar-lib
RUN locale-gen ru_RU.UTF-8

# Copy conf file
COPY manticore.conf /etc/manticoresearch/manticore.conf

# Prepare folder for Index files
RUN mkdir -pv /var/lib/manticoresearch
RUN chmod -R 777 /var/lib/manticoresearch

# Install ru.pack
RUN curl -LJO ${RU_PACK_URL}
RUN tar -xvzf ru.pak.tgz -C /usr/share/manticore
RUN rm ru.pak.tgz

# Install en.pack
RUN curl -LJO ${EN_PACK_URL}
RUN tar -xvzf en.pak.tgz -C /usr/share/manticore
RUN rm en.pak.tgz


CMD ["searchd", "--nodetach", "--config", "/etc/manticoresearch/manticore.conf"]

[KhAlena]

environment:
- QUERY_LOG_TO_STDOUT=true
- searchd_query_log_format=plain

with your parameters i have no query logs in stdout

[sanikolaev]

I still can't reproduce the issue based on your docker-compose and your Dockerfile. Unfortunately you didn't provide a fully reproducible example, so I had to guess. Please modify this so the problem starts manifesting. Otherwise it's hard/impossible to fix it:

snikolaev@dev:~/issue_20/repro$ cat docker-compose.yml
version: '2.2'
services:
  tamaranga-manticore:
    container_name: tamaranga-manticore
    image: tamaranga-manticore:${TAG:-latest}
    build:
      context: ./docker/manticore/
      dockerfile: Dockerfile
    restart: unless-stopped
    tty: true
    privileged: true
    ulimits:
      nproc: 65535
      nofile:
        soft: 65535
        hard: 65535
      memlock:
        soft: -1
        hard: -1
    volumes:
      - tamaranga-manticore:/var/lib/manticoresearch
    networks:
      - internal-network
    env_file:
      - .env
volumes:
   tamaranga-manticore:
networks:
   internal-network:

snikolaev@dev:~/issue_20/repro$ cat .env
TAG=a

snikolaev@dev:~/issue_20/repro$ ls -la docker
total 12
drwxrwxr-x 3 snikolaev snikolaev 4096 Jul  1 06:00 .
drwxrwxr-x 3 snikolaev snikolaev 4096 Jul  1 06:14 ..
drwxrwxr-x 2 snikolaev snikolaev 4096 Jul  1 06:19 manticore

snikolaev@dev:~/issue_20/repro$ ls -la docker/manticore/
total 16
drwxrwxr-x 2 snikolaev snikolaev 4096 Jul  1 06:19 .
drwxrwxr-x 3 snikolaev snikolaev 4096 Jul  1 06:00 ..
-rw-rw-r-- 1 snikolaev snikolaev  899 Jul  1 06:19 Dockerfile
-rw-rw-r-- 1 snikolaev snikolaev  453 Jul  1 06:18 manticore.conf

snikolaev@dev:~/issue_20/repro$ cat docker/manticore/Dockerfile
FROM manticoresearch/manticore

ARG RU_PACK_URL=https://repo.manticoresearch.com/repository/morphology/ru.pak.tgz
ARG EN_PACK_URL=https://repo.manticoresearch.com/repository/morphology/en.pak.tgz
# Locales installation
RUN apt-get update
RUN apt-get install -y locales-all locales curl xz-utils manticore-columnar-lib
RUN locale-gen ru_RU.UTF-8

# Copy conf file
COPY manticore.conf /etc/manticoresearch/manticore.conf

# Prepare folder for Index files
RUN mkdir -pv /var/lib/manticoresearch
RUN chmod -R 777 /var/lib/manticoresearch

# Install ru.pack
RUN curl -LJO ${RU_PACK_URL}
RUN tar -xvzf ru.pak.tgz -C /usr/share/manticore
RUN rm ru.pak.tgz

# Install en.pack
RUN curl -LJO ${EN_PACK_URL}
RUN tar -xvzf en.pak.tgz -C /usr/share/manticore
RUN rm en.pak.tgz

RUN ln -sf /dev/stdout /var/log/manticore/query.log

CMD ["searchd", "--nodetach", "--config", "/etc/manticoresearch/manticore.conf"]

snikolaev@dev:~/issue_20/repro$ cat docker/manticore/manticore.conf
common {
    plugin_dir = /usr/local/lib/manticore
    lemmatizer_base = /var/lib/manticore/
}

searchd {
    listen = 127.0.0.1:9312
    listen = 127.0.0.1:9306:mysql
    listen = 127.0.0.1:9308:http
    log = /var/log/manticore/searchd.log
    query_log = /var/log/manticore/query.log
    pid_file = /var/run/manticore/searchd.pid
    data_dir = /var/lib/manticore
    query_log_format = plain
    binlog_max_log_size = 1M
    max_open_files = max
}

build:

snikolaev@dev:~/issue_20/repro$ docker-compose build tamaranga-manticore
Building tamaranga-manticore
Step 1/17 : FROM manticoresearch/manticore
 ---> dda3734281a2
Step 2/17 : ARG RU_PACK_URL=https://repo.manticoresearch.com/repository/morphology/ru.pak.tgz
 ---> Using cache
 ---> 47bad5f977b1
Step 3/17 : ARG EN_PACK_URL=https://repo.manticoresearch.com/repository/morphology/en.pak.tgz
 ---> Using cache
 ---> 8d37a8b799f0
Step 4/17 : RUN apt-get update
 ---> Using cache
 ---> 61a7fb504a52
Step 5/17 : RUN apt-get install -y locales-all locales curl xz-utils manticore-columnar-lib
 ---> Using cache
 ---> 5bdbfc630d5b
Step 6/17 : RUN locale-gen ru_RU.UTF-8
 ---> Using cache
 ---> c5b4fc5f2973
Step 7/17 : COPY manticore.conf /etc/manticoresearch/manticore.conf
 ---> Using cache
 ---> 62911604dd72
Step 8/17 : RUN mkdir -pv /var/lib/manticoresearch
 ---> Using cache
 ---> 2cda4ae78be2
Step 9/17 : RUN chmod -R 777 /var/lib/manticoresearch
 ---> Using cache
 ---> e3aa5d3fecbc
Step 10/17 : RUN curl -LJO ${RU_PACK_URL}
 ---> Using cache
 ---> e2a6ab644e09
Step 11/17 : RUN tar -xvzf ru.pak.tgz -C /usr/share/manticore
 ---> Using cache
 ---> 29fb544c7c0e
Step 12/17 : RUN rm ru.pak.tgz
 ---> Using cache
 ---> 726b4d2dcf06
Step 13/17 : RUN curl -LJO ${EN_PACK_URL}
 ---> Using cache
 ---> 0caec4307a6c
Step 14/17 : RUN tar -xvzf en.pak.tgz -C /usr/share/manticore
 ---> Using cache
 ---> 0110a174c015
Step 15/17 : RUN rm en.pak.tgz
 ---> Using cache
 ---> abe86d6a656c
Step 16/17 : RUN ln -sf /dev/stdout /var/log/manticore/query.log
 ---> Using cache
 ---> 94f0405d7b65
Step 17/17 : CMD ["searchd", "--nodetach", "--config", "/etc/manticoresearch/manticore.conf"]
 ---> Using cache
 ---> bdc1b4250ff3
Successfully built bdc1b4250ff3
Successfully tagged tamaranga-manticore:a

up:

snikolaev@dev:~/issue_20/repro$ docker-compose up
Starting tamaranga-manticore ... done
Attaching to tamaranga-manticore
tamaranga-manticore    | Manticore 5.0.2 348514c86@220530 dev (columnar 1.15.4 2fef34e@220522) (secondary 1.15.4 2fef34e@220522)
tamaranga-manticore    | Copyright (c) 2001-2016, Andrew Aksyonoff
tamaranga-manticore    | Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
tamaranga-manticore    | Copyright (c) 2017-2022, Manticore Software LTD (https://manticoresearch.com)
tamaranga-manticore    |
tamaranga-manticore    | [24:12.139] [1] using config file '/etc/manticoresearch/manticore.conf' (454 chars)...
tamaranga-manticore    | [24:12.141] [1] Set max_open_files to 65535 (previous was 65535), hardlimit is 65535.
tamaranga-manticore    | starting daemon version '5.0.2 348514c86@220530 dev (columnar 1.15.4 2fef34e@220522) (secondary 1.15.4 2fef34e@220522)' ...
tamaranga-manticore    | listening on 127.0.0.1:9312 for sphinx and http(s)
tamaranga-manticore    | listening on 127.0.0.1:9306 for mysql
tamaranga-manticore    | listening on 127.0.0.1:9308 for sphinx and http(s)
tamaranga-manticore    | precaching index 't'
tamaranga-manticore    | precached 1 indexes in 0.001 sec
tamaranga-manticore    | accepting connections
tamaranga-manticore    | prereading 1 indexes
tamaranga-manticore    | prereaded 1 indexes in 0.000 sec
tamaranga-manticore    | [Fri Jul  1 04:24:48.887 2022] 0.000 sec 0.000 sec [ext2/0/ext 0 (0,20)] [t] абв

query:

snikolaev@dev:~/issue_20/repro$ docker-compose exec tamaranga-manticore bash
root@6dbb6590beb8:/var/lib/manticore# mysql -P9306 -h0
mysql> create table t(f text);
ERROR 1064 (42000): index 't': CREATE TABLE failed: index 't' already exists
mysql> select * from t where match('абв');

host's locale:

snikolaev@dev:~/issue_20/repro$ locale
LANG=en_US.UTF-8
LANGUAGE=
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=en_US.UTF-8