The Security Onion – NSM in an ESXi VM

I’ve been hearing great things about the Security Onion project. It bundles up a whole bunch of great tools that can be used for Network Security Monitoring (NSM) so I thought I would give it a try on my home network. After all, next time the fiancé gets malware, it could be helpful to know what happened…

VM Configuration

16GB OS disk (ext4)
300GB data disk mounted to /nsm (ext4, noatime set). Size this disk according to the usage of your network and how long you want to keep data.
2 NIC – one for management on my standard home VLAN, the other on a specific monitoring VLAN built as specified below.

Network Configuration – DD-WRT port mirroring and ESXi config

The management interface for the VM is on my standard DHCP subnet that serves my house. It will be used for management and to access all the tools on it. Nothing special there.

The special configuration comes into play because we need a port mirror off the router to get a full packet capture of all traffic going through the WAN interface. Now, DD-WRT can’t do a true mirror. A true mirror is a physical port on a switch that is designated to receive a copy of all traffic and the receiver typically does not have an IP address. In DD-WRT, you can’t do that, but you can use some iptables trickery to copy all traffic to a particular IP address. That being said, strictly speaking I did not need a second NIC in the VM. I could have just used one and had iptables forward all traffic to my one IP address. But I chose to have the second NIC to keep the mirrored traffic off my main VLAN. Note that in a production environment where real forensics could be required, this fake mirror would probably not be sufficient and you should invest in a fancy switch.

So to accomplish this configuration, I installed a second NIC in my ESXi server, created another virtual switch, and added a NIC on that switch to my VM:

Next, let’s set up DD-WRT.

DD-WRT Configuration

  1. Choose a port on your router to create a new VLAN. I chose port 4.
  2. In the Setup -> VLANs tab of your router, assign your chosen port to a different VLAN number than the others. The next available for me was 3. Do not assign the port to a bridge. Apply the settings.
  3. Under the Setup -> Networking tab find the Port Setup section and set vlan3 to Unbridged. Set Masquerade/NAT to Disabled (we’re not going to be sending any traffic out to the internet from that VLAN), and choose a subnet for your private monitoring LAN. I am going to go with and give the router interface the IP 254. Apply the settings.
  4. Lastly, choose an IP on that subnet to assign your Security Onion monitoring interface. I’m going to go with Under Administration -> Commands, paste these iptables commands, modified to specify the IP address you chose:
    iptables -A PREROUTING -t mangle -j ROUTE –gw –tee
    iptables -A POSTROUTING -t mangle -j ROUTE –gw –tee
  5. Click the Save Firewall button to apply the iptables rules. You might want to reboot the router for good measure.

Installing the Security Onion

Security Onion on Ubuntu 12.04 is just recently out! I largely followed their official install instructions located here:

  1. Download their ISO image which is based on Xubuntu Server 12.04 64bit and boot from it into Live Mode. When you get to the desktop, double click on the Install SecurityOnion 12.04 icon. You get pretty much the standard Ubuntu installer wizard. Walk through it, selecting Download Updates While Installing so you are fully up to date to start.
  2. When it comes to partitioning, do it manually and specify a 15GB ext4 root partition and a 1GB swap on the OS disk. On the data disk, create a new partition taking up the entire device and set its mount point to /nsm. /nsm is where the Security Onion will store all the data, so we should start with a big mount point before doing the software installation.
  3. Once the OS is installed, update it and install build-essential and linux-headers which will be needed to install VMware Tools:
    sudo apt-get update && sudo apt-get upgrade -y && sudo apt-get dist-upgrade -y && sudo apt-get install build-essential linux-headers-`uname -r` -y && sudo reboot
  4. Install VMware Tools. This is important, if for nothing else, the network card drivers. If you’re running ESXi 5, see my post here on how to successfully install tools on 12.04 if you get an error during compilation.
  5. If you are SSHed into the server, make sure your display is X-forwarded for the next setup steps. Or if not, you can still run the same command from a terminal. To start the setup wizard, run:
    sudo sosetup
  6. Walk through the wizard. Allow it to configure /etc/network/interfaces for you. To determine which interface is management and which is for monitoring, run ifconfig and see which one has a DHCP address. That one will be your management interface. It was eth0 on my VM. Then select eth1 as the monitoring interface and let it do the install.
  7. After reboot, run setup again with sudo sosetup and skip network configuration.
  8. Choose Advanced.
  9. Choose Standalone.
  10. My preference is Snort – try Suricata if you’d like. Select eth1 as the interface to be monitored.
  11. Chose 2 CPUs and 2 Bro instances.
  12. I’m using the Emerging Threats GPL Snort list.
  13. Setup the usernames and password.
  14. I chose to enable ELSA. It is a fantastic tool! This will be the first time I’m monitoring Bro logs, so we’ll see what we get.
  15. Let it work its magic and then you’re installed. It make take a little bit for your VM to settle down and all the processes to initialize.
  16. Now since DD-WRT needs an IP address to mirror the traffic to we need to configure an IP address on the monitoring interface. Edit /etc/network/interfaces and change the line ‘iface eth1 inet manual’ to ‘iface eth1 inet static’ and add these lines immediately below:
    Then comment all the other commands below that except for the last one which disables IPv6. It ends in ‘disable_ipv6’.
  17. Now reboot for good measure. Once you log back in, try pinging If that works, run tcpdump on your monitoring interface and make sure you’re seeing a while bunch of packets that aren’t destinated for your IP. If you see them, you’re capturing!

Next, visit https://your_server_name and you’ll get the Onion’s splash page. It contains links to Squert, Snorby, or ELSA.

Happy network security monitoring!


Security Onion blog

Things from the post install dialog boxes for reference:

Security Onion Setup is now complete!
Setup log can be found here:
You may view IDS alerts using Sguil, Squert, Snorby, or ELSA (if enabled).
Bro logs can be found in ELSA (if enabled) and the following location:
Rules downloaded by Pulledpork are stored in:
Local rules can be added to:
You can have PulledPork modify the downloaded rules
 by modifying the files in:
Rules will be updated every day at 7:01 AM UTC.
 You can manually update them by running:
Sensors can be tuned by modifying the files in:

4 thoughts on “The Security Onion – NSM in an ESXi VM”

  1. Thanks for writing this! I too had the plan of using DD-WRT, ESXi, and Security Onion and you took out a bunch of work for me. Your instructions are clear and easy to follow. Again, thanks for doing this!

    But, I’m having an issue getting and talking to each other, let along ping. Many other people out there have iptables commands similar to yours, but they do not include the extra vlan, port, and IP address setup. Did you ever have to do any other troubleshooting to get the 4th port on your router and your new virtual switch?

    Do you also have a separate physical port for your SecurityOnion monitoring interface? From your screenshot above, it doesn’t seem like you do. I do have another physical interface in my ESXi server that I was using for the monitoring interface. Maybe I need to do some more work with that.

    Thanks again for a great write-up!

    1. I’m glad the article was helpful!

      If you SSH/telnet into your router, does ifconfig show the new VLAN with your IP address? If it shows the VLAN but no IP, try setting it on the command line using ifconfig to see if then it will at least ping. That won’t persist through reboot but will at least point you in a direction. To take any complications introduced by ESXi, you can also just plug that port directly into a laptop, set it to and try pinging that way to rule out VMware problems.

      When I was researching this, I came across posts that some hardware might not like adding the extra vlan, so that is a possibility. I’d also suggest trying another port on the router besides 4.

      Yup, I do have a separate NIC in my ESXi box. I use the onboard NIC for my regular network then a separate PCI NIC that plugs directly into the router set up with the ‘span’ port. I don’t think you’d be able to have both the mirrored network and the regular network on the same port if they’re on their own subnets, unless DDWRT supports tagging, but that’s over my head. If you just set up an IP on your regular network to get the mirrored traffic, then you don’t need to mess with specifying a port and setting up a separate VLAN at all – you just need the iptables rules.

      Hope that helps! Let me know how it goes!

      1. Well, I got it working, but not via vlan 3 or a 10.0.0.x IP address. I did try hooking up a laptop, making it but there was no connectivity. I did try tagging another port as vlan 3 and moving my monitoring cable there, but that didn’t work either. Still no pings between and .254.

        Ultimately, I just gave my second NIC an IP on the same network as my mgmt interface, changed the network settings on the router back to normal, adjusted the iptables commands, saved it, rebooted and we were good to go.

        Of course I had to verify that data was coming to the 2nd NIC. I wasn’t seeing any intrusions… which is a good thing of course. But what I did was just run Wireshark in Security Onion (sudo, of course) and saw all of the traffic being copied.

        Thanks! This will tide me over until I get the rest of my home lab infrastructure up and running. I’ve got some Cisco routers waiting to do some proper port spanning. 🙂

Leave a Reply

Your email address will not be published. Required fields are marked *