Tag Archives: sysadmin

Logrotate with alternate compression tool such as pigz

Logrotate is a versatile tool for rotating logs. When logrotate is configured to rotate a set of logs using the ‘compress’ command, by default the gzip utility will be used. gzip is a good utility but you may have a need to use something else. My need is:

  • Multi-gigabyte files need to rotate hourly or daily
  • Rotation and compression was taking several minutes per file
  • My CPUs were mostly idle

My solution was to use the pigz compression utility coupled with logrotate. pigz performs multithreaded gzip compression so some of the idle CPUs could be put to work. Here’s how to do it:

  • Install pigz. You can either build it from source from here: http://zlib.net/pigz/ or the package manager for your OS may have it available.
  • Edit the logrotate.conf configuration to use the different utility. If you want to specify the max number of threads to use for compression, you can use the -p option. By default pigz will use all detected cores, which might not be desirable if you have a busy system or if rotation using all cores would cause lots of IO wait. Here’s an example config file with the new required options highlighted:
/var/log/file_to_rotate.log {
        daily
        rotate 10
        # immediately after rotate, 'postrotate' runs before compression
        postrotate
                # (if using rsyslog - something similar may be required for other syslog daemons)
                # HUP rsyslog to start writing to the new file. You want to HUP before starting to compress so that compression deals with a complete file, rather than one that is still being written to.
                /bin/kill -HUP `cat /var/run/rsyslogd.pid 2>/dev/null` 2>/dev/null || true
        endscript
        # then compress using pigz, maximum of 5 threads
        compress
        compresscmd /usr/bin/pigz
        compressoptions -p5
}

Gotcha! logrotate versions prior to 3.8.1-5 do not support spaces in compressoptions, so if you need to add other options, you’ll need to upgrade logrotate first.

Now compression during log rotation will use multiple threads, resulting in a speed up almost linearly with the number of threads you allow pigz to use.

ZFS on Linux with LUKS encrypted disks

To me, encryption of data at rest is just as important as encryption of data in transit. You never know if someone is going to break into your house and steal your computer. With so much personal information like financial data and pictures stored on the computer, it could be a major mess to recover from theft of your computer. (Of course, always keep an off-site backup for the really important stuff!)

I chose to migrate from the Solaris based OpenIndiana to Ubuntu. I had grown to love ZFS on OpenIndiana and didn’t want to lose its features. Luckily ZFS on Linux is now ready for prime-time! Unfortunately, ZFS on Linux is a few versions behind the official Oracle ZFS just like all other third part implementations of ZFS and does not support native encryption through the filesystem.

Continue reading ZFS on Linux with LUKS encrypted disks

SSL Client Authentication Step By Step

SSL’s primary function on the Internet is to facilitate encryption and trust that allows a web browser to validate the authenticity of a web site. However, SSL works the other way around too – client SSL certificates can be used to authenticate a client to the web server. Think SSH public/private key pairs, if that is familiar to you. In this blog post I will outline the steps to create a certificate authority certificate, sign a server certificate and install it in Apache, and create a client cert in a format used by web browsers.

Continue reading SSL Client Authentication Step By Step

Get the Ralink 28xx USB Dongle Running on Raspbian 7 (Raspberry Pi)

There are a variety of extremely cheep USB wifi dongles available on ebay that work wonderfully… once you get them working. I picked one up for $4 for my Raspberry Pi and had trouble getting it to work under Raspbian. Here’s what you need to do for the current Ralink USB adapters out there.

Continue reading Get the Ralink 28xx USB Dongle Running on Raspbian 7 (Raspberry Pi)

Automated ESXi Backup Without Dependencies!

I use VMware’s ESXi in my home lab environment. If you’re a technologist that is always messing with operating systems and applications, it is really the only way to go. But along with running any systems comes maintaining good backups!

There are numerous VMware backup products on the market. The problem is they all cost money. For me and other people who just play with this stuff for fun, that is not desirable. At all.

Continue reading Automated ESXi Backup Without Dependencies!