Spec: Cortex.cpp Logs

[freelerobot]

Goal:

• Clear, detailed and configurable (levels, verbosity) logging

Spec:

• Dependency: Trantor with UDS (multi-process logging)
• Log levels: trace, info, debug, warning, error, fatal
• Log format: TBA

Maintenance:

• Log file rotation: when log file exceeds max size, will roll to new file with the same prefix: .00000.log -> .00001.log.
• Max log file size: 20 Mb.
• Log purging: TBA

TBA

• [TBA] verbose mode

Related

• bug: Multiprocess cannot write log to the same file #1056
• feat: CLI stdout logs are clean and readable #1038
• feat: Configurable Log path for main program and dependencies #1062
• feat: how to monitor cortex inbound requests? #1007

Open Questions:

• What logging library are we using? Stdlib or 3rd party and why?
• What log levels are supported?
• Is logging configurable by user?
• How do you handle log rotation and file size limits?
• Are logs formatted consistently? e.g. timestamp, source of issue, message
• How are we passing through upstream errors, e.g. llamacpp errors
• Where are logs stored?
• Is logging threadsafe? Are there multiple processes writing to the same logs?
• Is there a Cortex command to tail logs?

[nguyenhoangthuan99]

Log library

• we are using trantor for logging feature because:
  • It is a Drogon's dependencies, so reuse trantor will save some resources
  • trantor is non-blocking I/O cross-platform, thread safe, async library.
  • Support multi streams logger (can chose to write log to different file, stream, ... )
  • Support log file rotation and set max file size: when log file exceeds max size, will roll to new file with the same prefix: <prefix>.00000.log -> <prefix>.00001.log. Currently max log file size is 20 Mb.
  • Support all needed informations: for example 20240903 06:46:55.974670 UTC 1268127 INFO Request 1727: Inference completed - llama_engine.cc:827, it includes time, timezone, process id, log level, message, source code file and line number
  • Log levels:
    • trace
    • info
    • debug
    • warning
    • error
    • fatal

Configuration
Currently, log files are stored under /path_to_cortex_cpp_binary/logs/ but need to configurable, now we set it as default constant in source code

When this Feature is done, we will read from config to decide where to store log file.

Log upstreams
For upstreams like llama.cpp and tensorrt llm, we have to override log method of those and redirect to right posion as we want, this task is Done for llamacpp and tensorrt llm, refer to this issue

Issues
Trantor is thread safe but it keeps the log file open during the life time of server process. With Linux and MacOS, it works perfectly, but in windows, when a process is holding a file, windows prevents other process to access it. Refer to this Issue

Future features

• Support tail latest logfile

[freelerobot]

1. Log rotation is fine, but what about purging. Previously, we've experienced cases when logs explode to xx Gb for users.

• Is there a global log file max
• Is there a max log expiry date, i.e. is there any use in keeping logs older than x Days/Weeks or Months

4. How do we denote where the log originated, i.e. llamacpp vs cortex.cpp process?
5. Does CLI have verbose / regular logging modes?

[freelerobot]

Also naming wise Im wondering if you should go with - instead of incremental ints

[nguyenhoangthuan99]

1. Trantor provide methods to set max number of log file under log folder, Trantor will delete the oldest log file first when the maximum files exceed, but we need to experiment to confirm how it exactly works.
2. Trantor doesn't support this feature.
3. For now we are using the same log file to log everything (this is configurable), in the log file also contain the file name and line number, -> we can easily find the root cause.
4. CLI has --verbose flag, when this flag is provided, everything will log to terminal instead of files
5. For naming format the full structure of log file is .-.0000.log
   Future change
   When this Pr fix: correct path to work with data folder structure #1132 is finished, will move log folder to the data folder (currently, log folder in the same location with binary)

[freelerobot]

Do we log the underlying API requests, e.g. re obj, response
#1007