[BUG] Docker image 1.4.9 upgrade result in nginx 404 not found

[gabviv73]

Describe the bug
Upgrading to 1.4.9 from 1.4.8 using docker compose results in blank page and 404 not found.

To Reproduce
Steps to reproduce the behavior:

set VERSION=1.4.9 in .env
docker compose down, docker compose up

Additional context

docker-compose.yml

services:
  mailarchive.example.com-db:
    image: mariadb:12.0.2
    container_name: mailarchive.example.com-db
    restart: unless-stopped
    env_file: .env
    command: --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci
    volumes:
      - ./mariadb_data:/var/lib/mysql
      - ./mariadb_piler.cnf:/etc/mysql/conf.d/piler.cnf:ro
    healthcheck:
      test: ["CMD", "healthcheck.sh", "--connect", "--innodb_initialized"]
      start_period: 10s
      interval: 10s
      timeout: 5s
      retries: 3

  mailarchive.example.com-manticore:
    image: manticoresearch/manticore:14.1.0
    container_name: mailarchive.example.com-manticore
    restart: unless-stopped
    volumes:
      - ./manticore.conf:/etc/manticoresearch/manticore.conf
      - ./manticore_data:/var/lib/manticore

  mailarchive.example.com-memcached:
    image: memcached
    container_name: mailarchive.example.com-memcached
    command: -m 64

  mailarchive.example.com-app:
    image: sutoj/piler:${VERSION:-1.4.8}
    container_name: mailarchive.example.com-app
    init: true
    env_file: .env
    ports:
      - "2525:25"
      - "8025:80"
    volumes:
      - ./piler_data/etc:/etc/piler
      - ./piler_data/store:/var/piler/store
      # - /mnt:/mnt
    healthcheck:
      test: curl -s smtp://localhost/
      interval: "20s"
      timeout: "3s"
      start_period: "15s"
      retries: 3
    depends_on:
      mailarchive.example.com-db:
        condition: service_healthy
      mailarchive.example.com-manticore:
        condition: service_started
      mailarchive.example.com-memcached:
        condition: service_started

.env

ARCHIVE_HOST=mailarchive.example.com
MANTICORE_HOSTNAME=mailarchive.example.com-manticore
MEMCACHED_HOSTNAME=mailarchive.example.com-memcached
MYSQL_DATABASE=piler
MYSQL_DATABASE=piler
MYSQL_HOSTNAME=mailarchive.example.com-db
MYSQL_PASSWORD=piler123
MYSQL_ROOT_PASSWORD=piler123
MYSQL_USER=piler
RT=1
PILER_HOSTNAME=${ARCHIVE_HOST}
VERSION=1.4.9

docker compose logs -f

mailarchive.example.com-app  | $config['RT'] = 1;
mailarchive.example.com-app  | $config['SPHINX_MAIN_INDEX'] = 'piler1';
mailarchive.example.com-app  | $config['SPHINX_HOSTNAME'] = 'mailarchive.example.com-manticore:9306';
mailarchive.example.com-app  | $config['SPHINX_HOSTNAME_READONLY'] = 'mailarchive.example.com-manticore:9307';
mailarchive.example.com-app  | Database
mailarchive.example.com-app  | information_schema
mailarchive.example.com-app  | piler
mailarchive.example.com-app  | DEBUG: mailarchive.example.com-db is ready
mailarchive.example.com-app  | DEBUG: metadata table exists
mailarchive.example.com-app  |  * Starting periodic command scheduler cron
mailarchive.example.com-app  |    ...done.
mailarchive.example.com-app  |  * Starting nginx nginx
mailarchive.example.com-app  |    ...done.
mailarchive.example.com-app  | rsyslogd: imklog: cannot open kernel log (/proc/kmsg): Operation not permitted.
mailarchive.example.com-app  | rsyslogd: activation of module imklog failed [v8.2312.0 try https://www.rsyslog.com/e/2145 ]
mailarchive.example.com-app  | starting piler-smtp . . . 
mailarchive.example.com-app  | starting piler . . .
mailarchive.example.com-manticore  | [Wed Jun 17 14:53:39.028 2026] [1] using config file '/etc/manticoresearch/manticore.conf.sh' (2571 chars)...
mailarchive.example.com-manticore  | starting daemon version '14.1.0 2b53a9307@25110709 (columnar 8.1.0 e1522a2@25100213) (secondary 8.1.0 e1522a2@25100213) (knn 8.1.0 e1522a2@25100213) (embeddings 1.0.1)' ...
mailarchive.example.com-manticore  | listening on all interfaces for sphinx and http(s), port=9312
mailarchive.example.com-manticore  | listening on all interfaces for mysql, port=9306
mailarchive.example.com-manticore  | listening on all interfaces for RO mysql, port=9307
mailarchive.example.com-manticore  | Manticore 14.1.0 2b53a9307@25110709 (columnar 8.1.0 e1522a2@25100213) (secondary 8.1.0 e1522a2@25100213) (knn 8.1.0 e1522a2@25100213) (embeddings 1.0.1)
mailarchive.example.com-manticore  | Copyright (c) 2001-2016, Andrew Aksyonoff
mailarchive.example.com-manticore  | Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
mailarchive.example.com-manticore  | Copyright (c) 2017-2025, Manticore Software LTD (https://manticoresearch.com)
mailarchive.example.com-manticore  | 
mailarchive.example.com-manticore  | precaching table 'piler1'
mailarchive.example.com-manticore  | precaching table 'tag1'
mailarchive.example.com-manticore  | precaching table 'note1'
mailarchive.example.com-manticore  | precaching table 'audit1'
mailarchive.example.com-manticore  | accepting connections
mailarchive.example.com-manticore  | prereading 4 tables
mailarchive.example.com-manticore  | preread 4 tables in 1.586 sec
mailarchive.example.com-manticore  | rt: table piler1: ramchunk saved ok (mode=periodic, last TID=2771681, current TID=2771684, ram=0.098 Mb, time delta=300 sec, took=0.220 sec)
mailarchive.example.com-db         | 2026-06-17 14:53:38+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:12.0.2+maria~ubu2404 started.
mailarchive.example.com-db         | 2026-06-17 14:53:39+00:00 [Warn] [Entrypoint]: /sys/fs/cgroup///memory.pressure not writable, functionality unavailable to MariaDB
mailarchive.example.com-db         | 2026-06-17 14:53:39+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
mailarchive.example.com-db         | 2026-06-17 14:53:39+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:12.0.2+maria~ubu2404 started.
mailarchive.example.com-db         | 2026-06-17 14:53:40+00:00 [Note] [Entrypoint]: MariaDB upgrade not required
mailarchive.example.com-db         | 2026-06-17 14:53:40 0 [Note] Starting MariaDB 12.0.2-MariaDB-ubu2404 source revision aab83aecdca15738d114cf5a2f223f1d12e4e6bd server_uid 23+MOwn0hwc6SR/fTddB4R7owsY= as process 1
mailarchive.example.com-db         | 2026-06-17 14:53:40 0 [Warning] --innodb-file-per-table is deprecated and will be removed in a future release
mailarchive.example.com-db         | 2026-06-17 14:53:40 0 [Note] InnoDB: Compressed tables use zlib 1.3
mailarchive.example.com-db         | 2026-06-17 14:53:40 0 [Note] InnoDB: Number of transaction pools: 1
mailarchive.example.com-db         | 2026-06-17 14:53:40 0 [Note] InnoDB: Using SSE4.2 crc32 instructions
mailarchive.example.com-db         | 2026-06-17 14:53:40 0 [Warning] mariadbd: io_uring_queue_init() failed with EPERM: sysctl kernel.io_uring_disabled has the value 2, or 1 and the user of the process is not a member of sysctl kernel.io_uring_group. (see man 2 io_uring_setup).
mailarchive.example.com-db         | create_uring failed: falling back to libaio
mailarchive.example.com-db         | 2026-06-17 14:53:40 0 [Note] InnoDB: Using Linux native AIO
mailarchive.example.com-db         | 2026-06-17 14:53:40 0 [Note] InnoDB: innodb_buffer_pool_size_max=256m, innodb_buffer_pool_size=256m
mailarchive.example.com-db         | 2026-06-17 14:53:40 0 [Note] InnoDB: Completed initialization of buffer pool
mailarchive.example.com-db         | 2026-06-17 14:53:40 0 [Note] InnoDB: File system buffers for log disabled (block size=512 bytes)
mailarchive.example.com-db         | 2026-06-17 14:53:40 0 [Note] InnoDB: End of log at LSN=8013243954
mailarchive.example.com-db         | 2026-06-17 14:53:40 0 [Note] InnoDB: Opened 3 undo tablespaces
mailarchive.example.com-db         | 2026-06-17 14:53:40 0 [Note] InnoDB: 128 rollback segments in 3 undo tablespaces are active.
mailarchive.example.com-db         | 2026-06-17 14:53:40 0 [Note] InnoDB: Setting file './ibtmp1' size to 12.000MiB. Physically writing the file full; Please wait ...
mailarchive.example.com-db         | 2026-06-17 14:53:40 0 [Note] InnoDB: File './ibtmp1' size is now 12.000MiB.
mailarchive.example.com-db         | 2026-06-17 14:53:40 0 [Note] InnoDB: log sequence number 8013243954; transaction id 19898861
mailarchive.example.com-db         | 2026-06-17 14:53:40 0 [Note] Plugin 'FEEDBACK' is disabled.
mailarchive.example.com-db         | 2026-06-17 14:53:40 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
mailarchive.example.com-db         | 2026-06-17 14:53:40 0 [Note] Plugin 'wsrep-provider' is disabled.
mailarchive.example.com-db         | 2026-06-17 14:53:40 0 [Warning] 'innodb-log-files-in-group' was removed. It does nothing now and exists only for compatibility with old my.cnf files.
mailarchive.example.com-db         | 2026-06-17 14:53:40 0 [Note] InnoDB: Buffer pool(s) load completed at 260617 14:53:40
mailarchive.example.com-db         | 2026-06-17 14:53:44 0 [Note] Server socket created on IP: '0.0.0.0', port: '3306'.
mailarchive.example.com-db         | 2026-06-17 14:53:44 0 [Note] Server socket created on IP: '::', port: '3306'.
mailarchive.example.com-db         | 2026-06-17 14:53:44 0 [Note] mariadbd: Event Scheduler: Loaded 0 events
mailarchive.example.com-db         | 2026-06-17 14:53:44 0 [Note] mariadbd: ready for connections.
mailarchive.example.com-db         | Version: '12.0.2-MariaDB-ubu2404'  socket: '/run/mysqld/mysqld.sock'  port: 3306  mariadb.org binary distribution

[jsuto]

Please show me the nginx error log from the container.

[gabviv73]

# docker compose exec  mailarchive.example.com-app tail -f /var/log/nginx/error.log

2026/06/18 07:36:08 [crit] 55#55: *5 connect() to unix:/run/php/php8.3-fpm.sock failed (2: No such file or directory) while connecting to upstream, client: 172.26.0.1, server: mailarchive.example.com, request: "GET / HTTP/1.1", upstream: "fastcgi://unix:/run/php/php8.3-fpm.sock:", host: "mailarchive.example.com"
2026/06/18 07:36:08 [error] 55#55: *5 open() "/usr/share/nginx/html/50x.html" failed (2: No such file or directory), client: 172.26.0.1, server: mailarchive.example.com, request: "GET / HTTP/1.1", upstream: "fastcgi://unix:/run/php/php8.3-fpm.sock", host: "mailarchive.example.com"

[jsuto]

This must be a wrong image. /etc/piler/piler-nginx.conf* files should point to unix:/run/php/php8.5-fpm.sock
Please pull sutoj/piler:1.4.9 again, and let's see it.

[PaPa1ya]

/etc/piler is preserved across versions in the volume.
I manually updated the FPM version in the config. Were any other configuration files updated between versions?
https://github.com/jsuto/piler/blob/master/docker/docker-compose.yaml

    volumes:
      - piler_etc:/etc/piler
      - piler_store:/var/piler/store

[jsuto]

I don't think so.

[stephdl]

I confirm the 1.4.9 does not start

Jun 22 11:26:40 r3-pve.rocky9-pve3.org systemd[11590]: Started Podman piler.service.
Jun 22 11:26:40 r3-pve.rocky9-pve3.org systemd[11590]: Starting Podman manticore-app.service...
Jun 22 11:26:40 r3-pve.rocky9-pve3.org systemd[11590]: Starting Podman mariadb-app.service...
Jun 22 11:26:40 r3-pve.rocky9-pve3.org systemd[11590]: Starting Podman memcached-app.service...
Jun 22 11:26:40 r3-pve.rocky9-pve3.org systemd[11590]: Started libcrun container.
Jun 22 11:26:40 r3-pve.rocky9-pve3.org piler1[27798]: 7d6dbb1e7872d5cb01baeea070ca7286965638cfef74ce988e956afe076b4a03
Jun 22 11:26:40 r3-pve.rocky9-pve3.org systemd[11590]: Started Podman manticore-app.service.
Jun 22 11:26:40 r3-pve.rocky9-pve3.org manticore-app[27841]: [Mon Jun 22 09:26:40.235 2026] [1] using config file '/etc/manticoresearch/manticore.conf.sh' (2260 chars)...
Jun 22 11:26:40 r3-pve.rocky9-pve3.org manticore-app[27841]: starting daemon version '10.1.0 c63706038@25060608 (columnar 5.0.1 59c7092@25060304) (secondary 5.0.1 59c7092@25060304) (knn 5.0.1 59c7092@25060304) (embeddings 1.0.0)' ...
Jun 22 11:26:40 r3-pve.rocky9-pve3.org manticore-app[27841]: listening on all interfaces for sphinx and http(s), port=9312
Jun 22 11:26:40 r3-pve.rocky9-pve3.org manticore-app[27841]: listening on all interfaces for mysql, port=9306
Jun 22 11:26:40 r3-pve.rocky9-pve3.org manticore-app[27841]: listening on all interfaces for RO mysql, port=9307
Jun 22 11:26:40 r3-pve.rocky9-pve3.org manticore-app[27841]: Manticore 10.1.0 c63706038@25060608 (columnar 5.0.1 59c7092@25060304) (secondary 5.0.1 59c7092@25060304) (knn 5.0.1 59c7092@25060304) (embeddings 1.0.0)
Jun 22 11:26:40 r3-pve.rocky9-pve3.org manticore-app[27841]: Copyright (c) 2001-2016, Andrew Aksyonoff
Jun 22 11:26:40 r3-pve.rocky9-pve3.org manticore-app[27841]: Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
Jun 22 11:26:40 r3-pve.rocky9-pve3.org manticore-app[27841]: Copyright (c) 2017-2025, Manticore Software LTD (https://manticoresearch.com)
Jun 22 11:26:40 r3-pve.rocky9-pve3.org manticore-app[27841]: 
Jun 22 11:26:40 r3-pve.rocky9-pve3.org manticore-app[27841]: precaching table 'piler1'
Jun 22 11:26:40 r3-pve.rocky9-pve3.org manticore-app[27841]: precaching table 'tag1'
Jun 22 11:26:40 r3-pve.rocky9-pve3.org manticore-app[27841]: precaching table 'note1'
Jun 22 11:26:40 r3-pve.rocky9-pve3.org manticore-app[27841]: precaching table 'audit1'
Jun 22 11:26:40 r3-pve.rocky9-pve3.org manticore-app[27841]: prereading 4 tables
Jun 22 11:26:40 r3-pve.rocky9-pve3.org manticore-app[27841]: preread 4 tables in 0.000 sec
Jun 22 11:26:40 r3-pve.rocky9-pve3.org manticore-app[27841]: accepting connections
Jun 22 11:26:40 r3-pve.rocky9-pve3.org systemd[11590]: Started libcrun container.
Jun 22 11:26:40 r3-pve.rocky9-pve3.org piler1[27800]: 0af988e92b562c29b40bb7debb2d8905351d5d51babfff6a24a5d319db2c34b7
Jun 22 11:26:40 r3-pve.rocky9-pve3.org systemd[11590]: Started Podman memcached-app.service.
Jun 22 11:26:40 r3-pve.rocky9-pve3.org systemd[11590]: Started libcrun container.
Jun 22 11:26:40 r3-pve.rocky9-pve3.org mariadb-app[27887]: 2026-06-22 09:26:40+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.11.18+maria~ubu2204 started.
Jun 22 11:26:40 r3-pve.rocky9-pve3.org piler1[27799]: beb72b58866e005c9f698751464673ca36b23aadf9faea60c45bcd8e3daf7bef
Jun 22 11:26:40 r3-pve.rocky9-pve3.org mariadb-app[27887]: 2026-06-22 09:26:40+00:00 [Warn] [Entrypoint]: /sys/fs/cgroup///memory.pressure not writable, functionality unavailable to MariaDB
Jun 22 11:26:40 r3-pve.rocky9-pve3.org mariadb-app[27887]: 2026-06-22 09:26:40+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
Jun 22 11:26:40 r3-pve.rocky9-pve3.org mariadb-app[27887]: 2026-06-22 09:26:40+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.11.18+maria~ubu2204 started.
Jun 22 11:26:40 r3-pve.rocky9-pve3.org systemd[11590]: Started Podman mariadb-app.service.
Jun 22 11:26:40 r3-pve.rocky9-pve3.org systemd[11590]: Starting Podman  piler-app.service...
Jun 22 11:26:40 r3-pve.rocky9-pve3.org mariadb-app[27887]: 2026-06-22 09:26:40+00:00 [Note] [Entrypoint]: MariaDB upgrade not required
Jun 22 11:26:40 r3-pve.rocky9-pve3.org mariadb-app[27887]: 2026-06-22  9:26:40 0 [Note] Starting MariaDB 10.11.18-MariaDB-ubu2204 source revision 197f92bee02d8e836f529f37625be69b83e7acbd server_uid ffACR+9geMTChroIGgAXz7j+2JY= as process 1
Jun 22 11:26:40 r3-pve.rocky9-pve3.org mariadb-app[27887]: 2026-06-22  9:26:40 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
Jun 22 11:26:40 r3-pve.rocky9-pve3.org mariadb-app[27887]: 2026-06-22  9:26:40 0 [Note] InnoDB: Number of transaction pools: 1
Jun 22 11:26:40 r3-pve.rocky9-pve3.org mariadb-app[27887]: 2026-06-22  9:26:40 0 [Note] InnoDB: Using SSE4.2 crc32 instructions
Jun 22 11:26:40 r3-pve.rocky9-pve3.org mariadb-app[27887]: 2026-06-22  9:26:40 0 [Note] mariadbd: O_TMPFILE is not supported on /tmp (disabling future attempts)
Jun 22 11:26:40 r3-pve.rocky9-pve3.org mariadb-app[27887]: 2026-06-22  9:26:40 0 [Warning] mariadbd: io_uring_queue_init() failed with ENOSYS: check seccomp filters, and the kernel version (newer than 5.1 required)
Jun 22 11:26:40 r3-pve.rocky9-pve3.org mariadb-app[27887]: create_uring failed: falling back to libaio
Jun 22 11:26:40 r3-pve.rocky9-pve3.org mariadb-app[27887]: 2026-06-22  9:26:40 0 [Note] InnoDB: Using Linux native AIO
Jun 22 11:26:40 r3-pve.rocky9-pve3.org mariadb-app[27887]: 2026-06-22  9:26:40 0 [Note] InnoDB: innodb_buffer_pool_size_max=8388608m, innodb_buffer_pool_size=256m
Jun 22 11:26:40 r3-pve.rocky9-pve3.org mariadb-app[27887]: 2026-06-22  9:26:40 0 [Note] InnoDB: Completed initialization of buffer pool
Jun 22 11:26:40 r3-pve.rocky9-pve3.org mariadb-app[27887]: 2026-06-22  9:26:40 0 [Note] InnoDB: File system buffers for log disabled (block size=512 bytes)
Jun 22 11:26:40 r3-pve.rocky9-pve3.org mariadb-app[27887]: 2026-06-22  9:26:40 0 [Note] InnoDB: End of log at LSN=44894
Jun 22 11:26:40 r3-pve.rocky9-pve3.org mariadb-app[27887]: 2026-06-22  9:26:40 0 [Note] InnoDB: 128 rollback segments are active.
Jun 22 11:26:40 r3-pve.rocky9-pve3.org mariadb-app[27887]: 2026-06-22  9:26:40 0 [Note] InnoDB: Setting file './ibtmp1' size to 12.000MiB. Physically writing the file full; Please wait ...
Jun 22 11:26:40 r3-pve.rocky9-pve3.org mariadb-app[27887]: 2026-06-22  9:26:40 0 [Note] InnoDB: File './ibtmp1' size is now 12.000MiB.
Jun 22 11:26:40 r3-pve.rocky9-pve3.org mariadb-app[27887]: 2026-06-22  9:26:40 0 [Note] InnoDB: log sequence number 44894; transaction id 14
Jun 22 11:26:40 r3-pve.rocky9-pve3.org mariadb-app[27887]: 2026-06-22  9:26:40 0 [Note] Plugin 'FEEDBACK' is disabled.
Jun 22 11:26:40 r3-pve.rocky9-pve3.org mariadb-app[27887]: 2026-06-22  9:26:40 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
Jun 22 11:26:40 r3-pve.rocky9-pve3.org mariadb-app[27887]: 2026-06-22  9:26:40 0 [Warning] 'innodb-log-files-in-group' was removed. It does nothing now and exists only for compatibility with old my.cnf files.
Jun 22 11:26:40 r3-pve.rocky9-pve3.org mariadb-app[27887]: 2026-06-22  9:26:40 0 [Warning] You need to use --log-bin to make --expire-logs-days or --binlog-expire-logs-seconds work.
Jun 22 11:26:40 r3-pve.rocky9-pve3.org mariadb-app[27887]: 2026-06-22  9:26:40 0 [Note] InnoDB: Buffer pool(s) load completed at 260622  9:26:40
Jun 22 11:26:40 r3-pve.rocky9-pve3.org mariadb-app[27887]: 2026-06-22  9:26:40 0 [Note] Server socket created on IP: '0.0.0.0', port: '3306'.
Jun 22 11:26:40 r3-pve.rocky9-pve3.org mariadb-app[27887]: 2026-06-22  9:26:40 0 [Note] Server socket created on IP: '::', port: '3306'.
Jun 22 11:26:40 r3-pve.rocky9-pve3.org mariadb-app[27887]: 2026-06-22  9:26:40 0 [Note] mariadbd: ready for connections.
Jun 22 11:26:40 r3-pve.rocky9-pve3.org mariadb-app[27887]: Version: '10.11.18-MariaDB-ubu2204'  socket: '/run/mysqld/mysqld.sock'  port: 3306  mariadb.org binary distribution
Jun 22 11:26:41 r3-pve.rocky9-pve3.org systemd[11590]: Started libcrun container.
Jun 22 11:26:41 r3-pve.rocky9-pve3.org piler-app[28001]: DEBUG: Updating /etc/piler/piler.conf
Jun 22 11:26:41 r3-pve.rocky9-pve3.org piler-app[28001]: sed: cannot rename /etc/piler/sed4lPTWd: Device or resource busy
Jun 22 11:26:41 r3-pve.rocky9-pve3.org piler1[27986]: f254a4e3a1a5efabd2182269924d195aacbcc17be7b43eff412a8783d5d6d56b
Jun 22 11:26:41 r3-pve.rocky9-pve3.org systemd[11590]: Started Podman  piler-app.service.
Jun 22 11:26:41 r3-pve.rocky9-pve3.org systemd[11590]: piler-app.service: Main process exited, code=exited, status=4/NOPERMISSION
Jun 22 11:26:41 r3-pve.rocky9-pve3.org piler1[28021]: f254a4e3a1a5efabd2182269924d195aacbcc17be7b43eff412a8783d5d6d56b
Jun 22 11:26:41 r3-pve.rocky9-pve3.org systemd[11590]: piler-app.service: Failed with result 'exit-code'.
Jun 22 11:26:41 r3-pve.rocky9-pve3.org systemd[11590]: piler-app.service: Scheduled restart job, restart counter is at 18.
Jun 22 11:26:41 r3-pve.rocky9-pve3.org systemd[11590]: Stopped Podman  piler-app.service.
Jun 22 11:26:41 r3-pve.rocky9-pve3.org systemd[11590]: Stopping Podman manticore-app.service...
Jun 22 11:26:41 r3-pve.rocky9-pve3.org systemd[11590]: Stopping Podman mariadb-app.service...
Jun 22 11:26:41 r3-pve.rocky9-pve3.org systemd[11590]: Stopping Podman memcached-app.service...
Jun 22 11:26:41 r3-pve.rocky9-pve3.org manticore-app[27841]: caught SIGTERM, shutting down
Jun 22 11:26:41 r3-pve.rocky9-pve3.org manticore-app[27841]: shutdown daemon version '10.1.0 c63706038@25060608 (columnar 5.0.1 59c7092@25060304) (secondary 5.0.1 59c7092@25060304) (knn 5.0.1 59c7092@25060304) (embeddings 1.0.0)' ...
Jun 22 11:26:41 r3-pve.rocky9-pve3.org manticore-app[27841]: shutdown complete
Jun 22 11:26:41 r3-pve.rocky9-pve3.org manticore-app[27841]: precached 4 tables in 0.002 sec
Jun 22 11:26:41 r3-pve.rocky9-pve3.org piler1[28034]: 7d6dbb1e7872d5cb01baeea070ca7286965638cfef74ce988e956afe076b4a03
Jun 22 11:26:42 r3-pve.rocky9-pve3.org mariadb-app[27887]: 2026-06-22  9:26:42 0 [Note] mariadbd (initiated by: unknown): Normal shutdown
Jun 22 11:26:42 r3-pve.rocky9-pve3.org mariadb-app[27887]: 2026-06-22  9:26:42 0 [Note] InnoDB: FTS optimize thread exiting.
Jun 22 11:26:42 r3-pve.rocky9-pve3.org mariadb-app[27887]: 2026-06-22  9:26:42 0 [Note] InnoDB: Starting shutdown...
Jun 22 11:26:42 r3-pve.rocky9-pve3.org mariadb-app[27887]: 2026-06-22  9:26:42 0 [Note] InnoDB: Dumping buffer pool(s) to /var/lib/mysql/ib_buffer_pool
Jun 22 11:26:42 r3-pve.rocky9-pve3.org mariadb-app[27887]: 2026-06-22  9:26:42 0 [Note] InnoDB: Buffer pool(s) dump completed at 260622  9:26:42
Jun 22 11:26:42 r3-pve.rocky9-pve3.org mariadb-app[27887]: 2026-06-22  9:26:42 0 [Note] InnoDB: Removed temporary tablespace data file: "./ibtmp1"
Jun 22 11:26:42 r3-pve.rocky9-pve3.org mariadb-app[27887]: 2026-06-22  9:26:42 0 [Note] Shutdown completed; log sequence number 44894; transaction id 15
Jun 22 11:26:42 r3-pve.rocky9-pve3.org piler1[28036]: 0af988e92b562c29b40bb7debb2d8905351d5d51babfff6a24a5d319db2c34b7
Jun 22 11:26:42 r3-pve.rocky9-pve3.org mariadb-app[27887]: 2026-06-22  9:26:42 0 [Note] mariadbd: Shutdown complete
Jun 22 11:26:42 r3-pve.rocky9-pve3.org piler1[28091]: 7d6dbb1e7872d5cb01baeea070ca7286965638cfef74ce988e956afe076b4a03
Jun 22 11:26:42 r3-pve.rocky9-pve3.org systemd[11590]: Stopped Podman manticore-app.service.
Jun 22 11:26:42 r3-pve.rocky9-pve3.org piler1[28035]: beb72b58866e005c9f698751464673ca36b23aadf9faea60c45bcd8e3daf7bef
Jun 22 11:26:42 r3-pve.rocky9-pve3.org piler1[28122]: 0af988e92b562c29b40bb7debb2d8905351d5d51babfff6a24a5d319db2c34b7
Jun 22 11:26:42 r3-pve.rocky9-pve3.org systemd[11590]: Stopped Podman memcached-app.service.
Jun 22 11:26:42 r3-pve.rocky9-pve3.org piler1[28149]: beb72b58866e005c9f698751464673ca36b23aadf9faea60c45bcd8e3daf7bef
Jun 22 11:26:42 r3-pve.rocky9-pve3.org systemd[11590]: Stopped Podman mariadb-app.service.
Jun 22 11:26:42 r3-pve.rocky9-pve3.org systemd[11590]: Stopping Podman piler.service...
Jun 22 11:26:42 r3-pve.rocky9-pve3.org systemd[11590]: Removed slice cgroup user-libpod_pod_3b13c7edd006c2bac000dcdaee3f24a2406d4171c281d85627efdb24868590fc.slice.
Jun 22 11:26:42 r3-pve.rocky9-pve3.org podman[28163]: piler
Jun 22 11:26:42 r3-pve.rocky9-pve3.org podman[28194]: 3b13c7edd006c2bac000dcdaee3f24a2406d4171c281d85627efdb24868590fc
Jun 22 11:26:42 r3-pve.rocky9-pve3.org systemd[11590]: Stopped Podman piler.service.

what I could do to inspect what is failing, the container piler-app refuses to start.

For what I undertsand my piler.conf configuration that I mount in a volume is not accepted

Jun 22 11:26:41 r3-pve.rocky9-pve3.org piler-app[28001]: DEBUG: Updating /etc/piler/piler.conf
Jun 22 11:26:41 r3-pve.rocky9-pve3.org piler-app[28001]: sed: cannot rename /etc/piler/sed4lPTWd: Device or resource busy
Jun 22 11:26:41 r3-pve.rocky9-pve3.org piler1[27986]: f254a4e3a1a5efabd2182269924d195aacbcc17be7b43eff412a8783d5d6d56b

piler configurations : https://gist.github.com/stephdl/60133392845e5abe376eb8c6b872bc6f

It is fully workable with 1.4.8

[jsuto]

Why did the sed command fail?

[stephdl]

Why did the sed command fail?

I'm not entirely sure why the sed command failed. It could be a permission issue, but I haven't investigated it in detail yet.

Mounting a configuration file into a container is a fairly common pattern and gives administrators flexibility to adjust settings when needed. An alternative approach is to expose configuration through environment variables, which is often more convenient for administrators, but it requires the application developer to explicitly support and provide all the necessary variables.

[stephdl]

Why did the sed command fail?

No real clue, I tested the bind mount outside of systemd and it worked, so :D

For the record this how I did to turn around my issue @jsuto with a pre action

However I found two things probably due to the upgrade to ubuntu 2604, the mysql client wants to talk with tls by default and rc.piler status incorrectly reports "NOT running"

I fixed them in my code but it is a fix and I hope you would accept to fix them or eventually I could offer to fix them by a PR myself