One for Hieu: Virtualization explained and compared

Hieu, sorry to single you out, but you provoke some important discussions and I think I can lay things out for you.

Background:  there are 2 principle kinds of Virtualization technologies – hardware and software.  Hardware-assisted virtualization software (which I’ll refer to from now on as hardware virtualization), like VMWare ESXi, is installed like any other distribution and provides direct access to hardware by guests.  Software virtualization programs, like VirtualBox or VMWare Server 2.x, is installed on top of an OS, like Windows or Linux, and runs like any other daemon.  In my experience software virtualization is a little faster than hardware virtualization and I assume, perhaps incorrectly, that this is due to the programming involved.  Software virtualization, as opposed to hardware virtualization, is accomplished using more flexible programming languages.  Hardware virtualization requires programming which is probably more archaic, like assembly, and this limits that flexibility.  The trade off is that software virtualization is only as secure as the virtualization application in conjunction with the security of the host OS.  Hardware virtualization, for whatever it’s worth, is still exploitable – still vulnerable – but when you break out of a guest VM you have a more restricted set of behaviors you can conduct.  This is not to say that one is “more secure” than the other, but be aware that they are different and be aware how they are different.

So, what kinds of technologies are there and how do they stack up?

Hardware:

• VMware ESXi
• 3Leaf DDC
• Citrix Xen Server (Bare metal)

Software:

• VMWare Server
• Sun VirtualBox
• Citrix Xen Server
• Microsoft HyperVisor

This isn’t comprehensive by any means, but we’ll look at each of these and how they are similar and how they are different.  You’ll forgive me, as well, for going light on the Hardware virtualization technologies – I tend to favor software virtualization, though I’ve worked extensively with VMWare ESXi.  I’ve also worked a great deal with Xenserver and if I were to try and compare them, difficult though the task may be, I would say that Xenserver offers a wealth of command-line management and was one of the first packages available with live migration available.  3Leaf’s product is a little different because it is based on an amalgamation of machines and not a standalone server.  This is a cluster-based system, first and foremost, and some of the redundancy it introduces is based on this being a collection of machines functioning as a single system.  3Leaf’s product, as opposed to ESXi, is not free – in fact, it’s about 100,000 dollars to outfit a 42U rack using this system – so it might not be the solution you want.

Software virtualization is really where I wanted to take this.  We’ll be focusing on VMWare Server 2.0, Xen, VirtualBox, and Windows virtualization.

Windows Hyper-V:  this is a proprietary, not open-source, software product that supports 32 and 64 bit hosts and guests with support for many versions of Windows as well as some Linux distributions.  Like other software virtualization products, guests are “jailed” and prevented from directly accessing hardware.  I’ve evaluated this technology using approximately a dozen small Windows 2003 and 2008 server guests on a machine with 4GB of RAM and a pair of Xeon processors at 3.2GHz – I didn’t have any issue with lag or sluggishness and the host was surprisingly responsive as well, despite most resources being allocated to guests.

Sun Microsystems VirtualBox:  licensed under the GPL, VirtualBox can be operated as an installed application, a la VMWare Workstation, or as an installed headless server.  I’ve covered VirtualBox several times and it’s clear I’m a fan.  VirtualBox has some limitations, of course, but with version 3.1 they introduce live migration and snapshot branching.  Perhaps because of the way that VirtualBox presents a console to your guest machines it can be sluggish on older hardware.  It isn’t terribly memory intensive, but I’ve found it to be CPU intensive on non-VT friendly Intel processors.

Citrix Xen Server:  Xen is licensed under the GPL and is kind of remarkable because it boots as a hypervisor and provides direct hardware access, but also allows direct machine manipulation through the Domain U (domU) interface.  Xen Server, although it could also be considered Hardware virtualization, is kind of a hybrid because the primary host kernel is Linux.  In addition Xen has an excellent live migration feature between 2 servers, allowing you to move machines without downtime to ensure a smooth upgrade process.  Xen also works well with pacemaker, heartbeat, and DRBD storage which makes it quite powerful for a free and open-source application.  I evaluated Xen using 4 machines, 2 of which were front-ends running Xen Server, 2 of which were iSCSI hosts storing the actual Guest and using DRBD and heartbeat.  I was able to live migrate a machine very quickly in this way as well as assure I had an absolutely redundant image that would automatically start a guest if the primary went down.

VMWare Server 2.0:  Breaking from VMWare Server 1.0 VMWare does away with the VMWare console.  The console was once the only way to interact with a guest VM, and 2.0 introduces a web console for interaction.  I suppose this is a nice change – it no longer requires you to have installed the console version to match the specific 1.x server version.  This is kind of a bittersweet change, however, because there is only support for Internet Explorer.  The web interface is also slow regardless of your host hardware – on a dual quad core machine with 3.4GHz processors and 8GB of ram this interface still kind of limped along despite being connected to the same subnet not 15 feet away from the server.

That’s a very basic coverage of technologies and isn’t the last word on the subject.  I don’t advocate any one particular technology, but they all have their place, so check them all out.  Can I say that one software is “better” than another on the same hardware?  No.  Learn them all.  Investigate each one and see how it handles specific tasks.  See how each handles SCSI, SATA, iSCSI, and other kinds of devices.  Evaluate the speed of live migration on those softwares that support it.  More importantly, evaluate the management interfaces and see what kinds of monitoring and feedback you can get.

VirtualBox 3.1 released

If you weren’t aware there’s a new version of VirtualBox out that is introducing some new features, most notably live migration (Teleportation) of Guests between VirtualBox implementations.  This feature, already present in other distributions like Xen, is restricted to the command line for the time being, but should make VirtualBox, headless or otherwise, a stronger contender in the Virtualization marketplace.  There’s also support for snapshot forks, allowing you to create many and varied snapshots of a Guest, another feature that makes VirtualBox 3.1 more attractive.

You can download it here.

Supplemental: hardening RouterOS

Primary material here.

After I completed the original write-up I received some feedback and eventually compiled some notes about how to properly secure a wireless device running RouterOS.  This isn’t comprehensive by any means, but demonstrates that the oldest lessons still apply.

You’re going to want to stop unnecessary services first.  Me, I only want web and ssh, so I’ll issue the following:

ip service disable ftp

ip service disable www-ssl

ip service disable api

ip service disable winbox

And it’s a good idea to restrict the scope of those services you want to keep running.  Let us assume that the “safe” network is 10.100.1.0/24 and execute the following commands:

ip service set www address=10.100.1.0/24

ip service set ssh address=10.100.1.0/24

the above command restricts access to any host not on the specified network and, if you are dedicated to your security, you might opt to use a single IP address from which to accept connections at these services.  Due to the ease of SNMP exploits you should also execute a command to restrict access to your SNMP service:

snmp community set address=10.100.1.0/24

You’re probably wondering why I disabled and restricted so many services and I’m delighted to tell you:  I was observing my AP locking up periodically and, when no logging faculty could indicate a hardware failure, I began to notice traffic to the AP on ports that should not have had direct connections.  Some of these ports were for services like ftp, www, ssh, telnet, and winbox, the application that allows you to mass-manage Mikrotik devices.  A remote attacker was clearly trying to gain access and by disabling unused services and restricting the scope of those services I did want to use I was able to completely prevent access from users outside a trusted subnet although the devices still function as bridge without any traffic degradation.

 

Running VirtualBox Headless

VirtualBox, Sun Microsystem’s Virtualization platform, is a fairly solid contender as far as Virtualization softwares go.  For those of you who already know about VirtualBox, you are undoubtedly aware that there is a standalone install that has a console with a GUI, allowing you to simply click in a console window and access your virtual machines.  Why would you want to run this without a monitor?  Well, primarily so that you can access the guests via remote desktop.

Assumptions:

1. you have pre-built a server of some kind
2. you installed a linux-based environment on the pre-built server (I chose Debian linux)
3. you want to run several guests on the server and access them via remote desktop

if any of the above assumptions fail to describe the situation perhaps this isn’t quite the solution you’re looking for.  I’ll continue on with my own assumption:  that this is the solution you’re looking for.

You need to install VirtualBox as the root user, so execute an “su root” at your earliest convenience.  I also like to discourage interaction as the root user unless absolutely necessary, so lets create a new user called vboxadmin:

groupadd vboxadmin && useradd -d /home/vboxadmin -m -g vboxadmin && passwd vboxadmin

You will be prompted to enter a password for vboxadmin.  Pick a good one, for example:  8iJ23Cf&7de.  You will also need to make sure that the VirtualBox repositories are added to your stock Debian repositories, accomplished like so:

cat “deb http://download.virtualbox.org/virtualbox/debian lenny non-free” >> /etc/apt/sources.list.d/virtualbox.list

If you were to try and update the repos and download VirtualBox you’d get an error telling you to go get the public key from Sun Microsystems, but you can pre-empt this by first snagging this key using the wget command:

wget -q http://download.virtualbox.org/virtualbox/debian/sun_vbox.asc -O- | apt-key add -

Now you need to update repositories (aptitude update) and update our server (aptitude upgrade) and now we rock and roll with our installation:

aptitude install linux-headers-$(uname -r) build-essential virtualbox-2.*

The above command will install the linux headers required by VirtualBox as well we installing VirtualBox 2-point-something.  The installer will tell you it is creating a new group called vboxusers, when prompted enter “Ok,” and there will no doubt be some manner of kernel error because you’re using a stock Debian linux kernel – don’t worry, though, because you’ll be prompted to compile a customer VirtualBox-friendly kernel and this you should absolutely agree to do.  Afterwards you need to add your vboxadmin to the vboxusers group:

adduser vboxadmin vboxusers

At this point you can download a virtual machine or create a new one from the command line interface.  You can read about my VirtualBox CLI tips over here.  To start a guest in headless mode execute:

VBoxHeadless -startvm “My Awesome Virtual Machine”

Now just connect via RDP (Remote Desktop Protocol) or VNC (Virtual Network Connection) and you’re good to go.

 

Determining the cause of a high server load

Lets just get right down to business:  you have a server, you’re on the console, and things are sluggish.  You need to know what is causing this behavior and you need to know it a half hour ago.  What to do?  Well, here are some of the things I do in about this order:

df -h     -     I like to make sure my issues aren’t the result of a full harddisk and this command will print out the partitions and how much is in use.  Here is an example:

dkwkstation:~ dk$ df -h /
Filesystem           Size        Used       Avail      Capacity       Mounted on
/dev/disk0s2      186Gi     184Gi     1.3Gi       99%              /

naturally this indicates my / partition is nearly full but is not maxed out.  If you see that your capacity is at or exceeds 100% you need to get in there and delete some files.  Next in line is the top command which provides you with an overview of running processes and system resources.

top -b -i -n 10 >> top.txt     -     this command executes top 10 times in batch mode to capture running processes and outputs that info to a text file called top.txt in the current directory.  The contents of an entry looks like this:

top – 15:52:43 up 160 days, 22:26,  2 users,  load average: 0.08, 0.16, 0.17
Tasks: 141 total,   1 running, 140 sleeping,   0 stopped,   0 zombie
Cpu(s):  3.6%us,  1.9%sy,  0.0%ni, 94.4%id,  0.2%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   2076708k total,  1981852k used,    94856k free,   236104k buffers
Swap:  4049032k total,       44k used,  4048988k free,  1037912k cached

PID     USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
18930 dk            15   0      2300 1160  876  R 0           0.1            0:00.01 top

this output tells you how much CPU and RAM are in use, who is using them, and the command responsible.  You can see your usual top output like uptime, currently logged in users, and the load average.  If your server is sluggish you might have high load averages, averages in double digits are very high.  I have seen load averages over 95 before on badly overutilized machines and I’m sure there are higher load averages that have been documented.  If your machine is sluggish but your load averages are low the issue may be related to the network or disks or, hopefully not, hardware failure.  Diagnosing something like hardware failure is slightly more involved – these two commands should help you rule out whether your machine is out of resources like RAM, CPU, or storage space and tell you what, if any, process has gone rogue.

 

 

Book review: Windows Forensic Analysis DVD Toolkit, 2nd Edition

I don’t review things – first because I feel an endorsement opens a person up to criticism and second because I’m not very good at it.  The reasons I like this book may be the same reasons other folks dislike it.  I suppose I’d better get to it.

Harlan Carvey, the author of this book, is one of the best known forensics and incident response analysts in the world today and he also maintains a blog here that covers a great deal of ground.  This book, Windows Forensic Analysis DVD Toolkit, is one of the most widely enjoyed forensics books available, as well, and I think I understand why.

The first observation I can make about this book is that it attempts to de-mystify the world of digital forensics and it does so by very clearly stating that this is a procedural science and by declaring the imminent demise of so called point-and-click forensic softwares.  I don’t believe this is to say that AccessData or Guardian are going out of business, but rather that the degree of integration of computing devices and the ways that technologies are converging has made it less likely that this kind of analysis suite will do all the things you need it to do.  To give you an example there are a number of GPS devices from which you may want to acquire an image and, by default, your EnCase or FTK installation will not be able to perform this kind of acquisition – from experience I can tell you this sometimes takes some additional drivers, etc.  I found a number of examples where this frank disclosure was reinforced with an open-source utility or a simple parsing script, conveying that, yes, this was an accurate depiction of a solution and here, before me, was how I could repeat the procedure.

I tried to look at how a non-forensics person might interpret this book – if you just want to develop an intimate and practical knowledge of how your Windows operating system does things this is a great way to get started because everything is relative to function.  If you are curious about how malware behaves or how malicious code might affect a machine you can discover that as well.  If you want to write software to better erase the traces of online behavior you can learn that, too.  You will also come away more confident in your knowledge of the function of the registry, that dark and mysterious Windows catacomb organized, frighteningly, into “hives” of eldritch configuration settings.  And let us not omit the file systems.  I am very confident when it comes to file systems and I was surprised to learn a few new things along the way.

Here’s the skinny:  if you work in the industry you should at the very least read this book once.  Me, I’ll be keeping this on the shelf.

MacHeist Free Software Bundle

First and foremost I am in no way affiliated with MacHeist, but there is a free software giveaway over at macheist.com and I encourage you to go get your fully licensed nano bundle.

Productivity: how’s your desktop?

I was talking with Burns today about his desktop configuration and my own and I’ll confess I’m not really ready to adopt the multi-display situation.  I don’t have any issue with multiple machines, or using a machine with multiple desktops that I can tab between, but I think having several displays would be distracting.  My desktop, although slightly cluttered right this minute, consists of a 16 inch CRT which is plugged into my Debian machine under the desk, my MSI Wind U100 netbook, and my Macbook Pro.  The Debian machine has no desktop or window manager – console only, my netbook is running XP Pro, and my Macbook Pro has 4 desktops utilizing Spaces.

I typically use my Debian machine to connect to machines for long-term projects via SSH using Alt-F[1-9] to segregate connections or to validate script functionality.  My netbook is the least frequently used machine – when I need IE, MVLS access, evaluating some particular Windows software, or to connect using the VMWare Console to those VMWare servers still running the older 1.x VMWare Server.  My Mac is my workhorse, though.

Desktop 1 has MacMail, Adium, and PicoPlay, Desktop 2 is split by 2 browsers, Desktop 3 has RDP sessions and iTerm and Desktop 4 is where I do anything else that needs to be done.  I suppose it could be argued that this is not that different from using multiple displays but I don’t have to look at them all at once.  I’ve been playing around with infrastructure virtualization, a technology that would allow one physical machine to paravirtualize several distinctly non-physical machines, but I don’t think I’m ready to operate a half-dozen specialized workstations just yet.  I believe the configuration and management would require more time and energy than it would be worth.  The notable exception, I believe, would be if I sat in the driver’s seat of a NOC and had to monitor, manage, and protect divergent networks.

Outside concern: information overload

Multinode Xen with LiveMigration

I’ve been puttering around with a bunch of machines running Xen, getting more familiar with the LiveMigration functionality which facilitates a copy of a running virtual machine from one node to another, and generally squirreling clutter around the apartment.  There is a project stub for this that I’ve been working on for probably too long, but I wanted to try and find ways to make the system more secure without installing another machine.  Trusted systems, true to their namesake, tend to be insecure.  Well, let’s say they resist security measures.

My mentees are very interested in completing this project, too, and naturally moving on to the next one.

 

Things found on the cutting room floor

This article by Larry Daniel of Guardian outlines the difference between a Computer search and a Forensic examination as it pertains to the recent US Court of Appeals ruling mentioned before.  This is a new way to address what many consider a major upset for law enforcement relative to digital forensics.

This is an application that allows you to forge malicious PDF documents, the purpose of which is to develop better analytical skills and determine how these malicious PDFs are being generated.  This is an educational tool but also allows you to edit and modify PDFs.

A new physical-to-virtual machine converted was released over on the sysinternals site here. This type of program, examples include the VMWare VMConverter, allow you to virtualize a running machine.  The Disk2vhd differs from other kinds of converters because it clones live and uses Volume Snapshot to create the running-state VM.

If you’re looking for something like Tripwire without all the bells, whistles, and price you should check out AIDE.  AIDE, or Advanced Intrusion Detection Environment, is a free and open source solution (FOSS) that you install to a machine.  AIDE maintains a database of file conditions and alerts you when a change has been made.  You can manually update this database when you install something or perform an update and this can function as one component of your overall system security monitor.  Here is the page describing it and here is a brief installation guide over at howtoforge.com.  I’ll probably put something up on the projects page shortly offering a practical application.