Logrotate is a versatile tool for rotating logs. When logrotate is configured to rotate a set of logs using the ‘compress’ command, by default the gzip utility will be used. gzip is a good utility but you may have a need to use something else. My need is:
- Multi-gigabyte files need to rotate hourly or daily
- Rotation and compression was taking several minutes per file
- My CPUs were mostly idle
My solution was to use the pigz compression utility coupled with logrotate. pigz performs multithreaded gzip compression so some of the idle CPUs could be put to work. Here’s how to do it:
- Install pigz. You can either build it from source from here: http://zlib.net/pigz/ or the package manager for your OS may have it available.
- Edit the logrotate.conf configuration to use the different utility. If you want to specify the max number of threads to use for compression, you can use the -p option. By default pigz will use all detected cores, which might not be desirable if you have a busy system or if rotation using all cores would cause lots of IO wait. Here’s an example config file with the new required options highlighted:
/var/log/file_to_rotate.log {
daily
rotate 10
# immediately after rotate, 'postrotate' runs before compression
postrotate
# (if using rsyslog - something similar may be required for other syslog daemons)
# HUP rsyslog to start writing to the new file. You want to HUP before starting to compress so that compression deals with a complete file, rather than one that is still being written to.
/bin/kill -HUP `cat /var/run/rsyslogd.pid 2>/dev/null` 2>/dev/null || true
endscript
# then compress using pigz, maximum of 5 threads
compress
compresscmd /usr/bin/pigz
compressoptions -p5
}
Gotcha! logrotate versions prior to 3.8.1-5 do not support spaces in compressoptions, so if you need to add other options, you’ll need to upgrade logrotate first.
Now compression during log rotation will use multiple threads, resulting in a speed up almost linearly with the number of threads you allow pigz to use.