Category Archives: Tools

Logrotate with alternate compression tool such as pigz

Logrotate is a versatile tool for rotating logs. When logrotate is configured to rotate a set of logs using the ‘compress’ command, by default the gzip utility will be used. gzip is a good utility but you may have a need to use something else. My need is:

  • Multi-gigabyte files need to rotate hourly or daily
  • Rotation and compression was taking several minutes per file
  • My CPUs were mostly idle

My solution was to use the pigz compression utility coupled with logrotate. pigz performs multithreaded gzip compression so some of the idle CPUs could be put to work. Here’s how to do it:

  • Install pigz. You can either build it from source from here: http://zlib.net/pigz/ or the package manager for your OS may have it available.
  • Edit the logrotate.conf configuration to use the different utility. If you want to specify the max number of threads to use for compression, you can use the -p option. By default pigz will use all detected cores, which might not be desirable if you have a busy system or if rotation using all cores would cause lots of IO wait. Here’s an example config file with the new required options highlighted:
/var/log/file_to_rotate.log {
        daily
        rotate 10
        # immediately after rotate, 'postrotate' runs before compression
        postrotate
                # (if using rsyslog - something similar may be required for other syslog daemons)
                # HUP rsyslog to start writing to the new file. You want to HUP before starting to compress so that compression deals with a complete file, rather than one that is still being written to.
                /bin/kill -HUP `cat /var/run/rsyslogd.pid 2>/dev/null` 2>/dev/null || true
        endscript
        # then compress using pigz, maximum of 5 threads
        compress
        compresscmd /usr/bin/pigz
        compressoptions -p5
}

Gotcha! logrotate versions prior to 3.8.1-5 do not support spaces in compressoptions, so if you need to add other options, you’ll need to upgrade logrotate first.

Now compression during log rotation will use multiple threads, resulting in a speed up almost linearly with the number of threads you allow pigz to use.

Automated ESXi Backup Without Dependencies!

I use VMware’s ESXi in my home lab environment. If you’re a technologist that is always messing with operating systems and applications, it is really the only way to go. But along with running any systems comes maintaining good backups!

There are numerous VMware backup products on the market. The problem is they all cost money. For me and other people who just play with this stuff for fun, that is not desirable. At all.

Continue reading Automated ESXi Backup Without Dependencies!

Install Thug on Kali Linux

EDIT: This post was originally written for Kali 1.0 in 2013. It was not tested on any later version.

I started this post with Backtrack, but since Kali Linux came out yesterday, I figured I would update the instructions to use the latest and greatest!

Thug is a low-interaction honeyclient put out by The Honeynet Project. The purpose of a honeyclient is to emulate a web browser when viewing malicious websites without actually visiting the page in a vulnerable browser. It can then follow redirects, pull down malware, and emulate browser plugins to pull down any of the malicious code trying to be served up to victims. You can read more about Thug here and here.

Continue reading Install Thug on Kali Linux