Logrotate with alternate compression tool such as pigz

Logrotate is a versatile tool for rotating logs. When logrotate is configured to rotate a set of logs using the ‘compress’ command, by default the gzip utility will be used. gzip is a good utility but you may have a need to use something else. My need is:

  • Multi-gigabyte files need to rotate hourly or daily
  • Rotation and compression was taking several minutes per file
  • My CPUs were mostly idle

My solution was to use the pigz compression utility coupled with logrotate. pigz performs multithreaded gzip compression so some of the idle CPUs could be put to work. Here’s how to do it:

  • Install pigz. You can either build it from source from here: http://zlib.net/pigz/ or the package manager for your OS may have it available.
  • Edit the logrotate.conf configuration to use the different utility. If you want to specify the max number of threads to use for compression, you can use the -p option. By default pigz will use all detected cores, which might not be desirable if you have a busy system or if rotation using all cores would cause lots of IO wait. Here’s an example config file with the new required options highlighted:
/var/log/file_to_rotate.log {
        daily
        rotate 10
        # immediately after rotate, 'postrotate' runs before compression
        postrotate
                # (if using rsyslog - something similar may be required for other syslog daemons)
                # HUP rsyslog to start writing to the new file. You want to HUP before starting to compress so that compression deals with a complete file, rather than one that is still being written to.
                /bin/kill -HUP `cat /var/run/rsyslogd.pid 2>/dev/null` 2>/dev/null || true
        endscript
        # then compress using pigz, maximum of 5 threads
        compress
        compresscmd /usr/bin/pigz
        compressoptions -p5
}

Gotcha! logrotate versions prior to 3.8.1-5 do not support spaces in compressoptions, so if you need to add other options, you’ll need to upgrade logrotate first.

Now compression during log rotation will use multiple threads, resulting in a speed up almost linearly with the number of threads you allow pigz to use.

Session Key support for Arduino with RFM69 wireless module

Felix Rusu from lowpowerlab.com has created an amazing library for the Arduino/ATMega platform to use cheap RFM69 wireless modules for wireless transmissions between nodes. For a list of all the features, see his readme: https://github.com/LowPowerLab/RFM69. His Moteino platform and library are perfect for roll-your-own home automation systems, which is just what I am doing!

However, in my mind the library was lacking one critical piece – being able to prevent wireless replay attacks. Using Felix’s library, RFM69 transmissions can be encrypted in hardware, but the receiving node had no way of verifying that the incoming transmission was from a legitimate node and should be processed. If an attacker captured a sensitive packet as it flew through the air, such as one to open your garage door, they could replay it at a time of their choosing and trick your home automation system into opening the door.

TomWS on the lowpowerlab forum and I did some brain storming and he proposed modifications to the base RFM69 library to virtualize some functions, allowing them to be overloaded to create ‘extensions’. Using his modification, I wrote a transmission session key extension.

The exchange between wireless nodes looks like this

  • The Node makes a connection request by setting one of the bits of the control byte built into the RFM69 base library. The control bit indicates a session is requested.
  • The Gateway receives the incoming packet and using the control bit flag, recognizes the packet as a session request. It then generates a random 1 byte session key and stores it. It sets another control bit indicating a key is included in the packet, adds the session key as the first byte of DATA, then sends that all back to the node.
  • The Node receives the incoming packet, parses the control bits and session key, then prepares the data it wants to send. It adds the session key as the first byte of DATA to be sent back, then appends whatever other data needs to be sent after the key. The control bit indicating a session key is included is set. All of that is sent back to the Gateway.
  • The Gateway receives the response and sees the control bit that says a session key is included. It checks to make sure the first byte of DATA where the incoming session key is stored matches the expected key, then processes the rest of the payload if it does. If not, DATALEN is set to 0, indicating to the sketch that there is no data available that should be processed.

Note: when using a session key, the amount of user accessible data that can be sent between nodes is decreased by 1 byte. This byte is used by the session key.

Of course, this is all useless if encryption isn’t enabled 🙂

Configurations that work, configurations that don’t

One of my goals with this extension was to allow session keys to be used for sensitive transactions but also let dumb sensor nodes that only send environment data still ‘fire and forget’ data without a key. Sensor data is likely not critical and therefore does not warrant the overhead of a session key.

Here is a description of each possible combination of gateway and nodes having session keys enabled.

  • Both gateway and node with session key disabled – transmissions occur as they would if using the base RFM69 library only.
  • Both gateway and node with session key enabled – the existing ‘send’ and ‘sendWithRetry’ methods will automatically use session keys protection.
  • Gateway enabled, node disabled – works and allows “fire and forget” messages from nodes that don’t require session keys. However, the gateway cannot initiate a new connection to send data to the node because the gateway will try to initiate a new session, but the node will not be enabled, so it will never handshake a key. This configuration should only be used with nodes that simply send data to the gateway and never need to receive incoming data from the gateway.
  • Gateway disabled, node enabled – does not work, as expected, because the node can’t handshake a key with the gateway.

How to use it

I designed this extension so few changes to the main sketch are needed. It’s as simple as adding my extension library with an ‘include’ statement and adding

radio.useSessionKey(1);

as part of the sketch setup. Any ‘send’ or ‘sendWithRetry’ commands that already exist within your code will now use session keys without any other modifications.

CAUTION: I do not use wireless node programming as is supported by the original RFM69 library, so I did not test if wireless programming still works with session key support enabled.

Step by step:

The below assumes you already have a working gateway/node setup using Felix’s base RFM69 library.

  • Download TomWS’s ‘virtualized’ branch of the RFM69 library that allows overriding functions. Put it in your Arduino Sketchbook Libraries directory. If you already have Felix’s base library, you should remove that and this should replace it. Available from here: https://github.com/dewoodruff/RFM69/tree/virtualized
  • Download my RMF69_SessionKey extension from here: https://github.com/dewoodruff/RFM69_SessionKey. Also place this in the Sketchbook Libraries directory.
  • Edit your existing sketches, both nodes and gateway, as follows.
  • Right before the existing #include <RFM69.h> statement, add this:
    #include <RFM69_SessionKey.h> // enable session key support
  • Where you previously had something like:
    RFM69 radio;

    Change to

    RFM69_SessionKey radio;
  • Lastly, in your setup function, simply add:
    radio.useSessionKey(1);

Session keys are now seamlessly enabled for the rest of the sketch – no need to change anything else.

Pros

  • Provides reasonable protection against replay attacks. Helpful for sensitive transactions such as opening a lock or garage door.

Cons

  • The gateway only maintains a session key for one node at a time. If one node starts a session with a gateway, then a second node starts a new session before the gateway gets data from the first node, the second node’s session will override the first. This would probably only be a problem on a very busy network. Using sendWithRetry will help mitigate this risk.
  • Less efficient than simple ‘fire and forget’ transmissions which can be desirable for battery powered nodes. Instead of 1-2 total packets per session (depending on if an ACK is requested), you’re now looking at 3-4 total, two of which come from the battery powered node. This can be avoided for simple sensor nodes that only gather data rather than control anything important by just not enabling session key support on that node.
  • It’s not perfect. While low risk, an attacker could theoretically still continuously try to ‘guess’ the session key by sending a new connection request, then replaying the stored packet, hoping that the session key in the stored packet ends up matching what was just generated by the gateway.

Conclusions

Felix’s library is great, and this extension adds just a little bit extra security to sensitive transmissions. You should weigh if this security is needed in your environment, as there are some cons to this setup. It’s pretty unlikely that someone who wants to break into your house will try to hack your home automation system – it’s much easier to just throw a rock through the window. But you never know! 😉

References

MCP23017 GPIO Expander python 3 Library – With Interrupts!

There are numerous python libraries out there for the 16 port MCP23017 GPIO expander chip that works with the Raspberry Pi, so why yet another one? None of the ones I could find actually implement interrupts via the chip Polling is not ideal for my home monitoring setup. I was already monitoring via interrupts and the onboard GPIO pins using the awesome RPIO module and wanted to continue using interrupts with the expansion chip. So I built my own module, leveraging Adafruit’s I2C library for the nitty gritty backend interface.

Features:
  • Simple digital input and output via all pins
  • Input interrupts
  • Interrupt port mirroring is configurable – either INTA and INTB can trigger independently for their respective GPIO port banks, or both INTA and INTB can trigger at the same time regardless of what GPIO pin causes the interrupt
  • Configurable interrupt polarity – INT could pull the pin high or push it low
  • Each GPIO pin can be configured for interrupts independently to either compare against the previous value or against a default pin value
  • A utility method cleanupInterrupts that can be called periodically to clear the interrupt if it somehow gets stuck on

You can find the library on my BitBucket.

Stick around after the break for a breakdown of how to use it.

Continue reading MCP23017 GPIO Expander python 3 Library – With Interrupts!

ZFS on Linux with LUKS encrypted disks

To me, encryption of data at rest is just as important as encryption of data in transit. You never know if someone is going to break into your house and steal your computer. With so much personal information like financial data and pictures stored on the computer, it could be a major mess to recover from theft of your computer. (Of course, always keep an off-site backup for the really important stuff!)

I chose to migrate from the Solaris based OpenIndiana to Ubuntu. I had grown to love ZFS on OpenIndiana and didn’t want to lose its features. Luckily ZFS on Linux is now ready for prime-time! Unfortunately, ZFS on Linux is a few versions behind the official Oracle ZFS just like all other third part implementations of ZFS and does not support native encryption through the filesystem.

Continue reading ZFS on Linux with LUKS encrypted disks

SSL Client Authentication Step By Step

SSL’s primary function on the Internet is to facilitate encryption and trust that allows a web browser to validate the authenticity of a web site. However, SSL works the other way around too – client SSL certificates can be used to authenticate a client to the web server. Think SSH public/private key pairs, if that is familiar to you. In this blog post I will outline the steps to create a certificate authority certificate, sign a server certificate and install it in Apache, and create a client cert in a format used by web browsers.

Continue reading SSL Client Authentication Step By Step

Get the Ralink 28xx USB Dongle Running on Raspbian 7 (Raspberry Pi)

There are a variety of extremely cheep USB wifi dongles available on ebay that work wonderfully… once you get them working. I picked one up for $4 for my Raspberry Pi and had trouble getting it to work under Raspbian. Here’s what you need to do for the current Ralink USB adapters out there.

Continue reading Get the Ralink 28xx USB Dongle Running on Raspbian 7 (Raspberry Pi)

Git part 3: Branching, merging and resolving conflicts

At last! The third and final part of our git series has arrived. The first two parts of the series get you started with git and interacting with your project. They are available here:

Git part 1: Intro to git, setup a git server on CentOS and create a new project
Git part 2: Interacting with your project

In the final chapter of this series, we’ll explore a few of the more advanced features.

Continue reading Git part 3: Branching, merging and resolving conflicts

Git Quick Tip – Ignoring a file

Sometimes in a coding project there will be files or directories that should not be included in a git repository. Common examples are temporary folders, log files, or files containing passwords/keys that should be regenerated by each installation of the software and kept secret. Fortunately, it is very easy to ignore these files when committing the git project.

Continue reading Git Quick Tip – Ignoring a file

Automated ESXi Backup Without Dependencies!

I use VMware’s ESXi in my home lab environment. If you’re a technologist that is always messing with operating systems and applications, it is really the only way to go. But along with running any systems comes maintaining good backups!

There are numerous VMware backup products on the market. The problem is they all cost money. For me and other people who just play with this stuff for fun, that is not desirable. At all.

Continue reading Automated ESXi Backup Without Dependencies!

Install Thug on Kali Linux

EDIT: This post was originally written for Kali 1.0 in 2013. It was not tested on any later version.

I started this post with Backtrack, but since Kali Linux came out yesterday, I figured I would update the instructions to use the latest and greatest!

Thug is a low-interaction honeyclient put out by The Honeynet Project. The purpose of a honeyclient is to emulate a web browser when viewing malicious websites without actually visiting the page in a vulnerable browser. It can then follow redirects, pull down malware, and emulate browser plugins to pull down any of the malicious code trying to be served up to victims. You can read more about Thug here and here.

Continue reading Install Thug on Kali Linux