11.13.08

VMware Server on Linux (Tuning and Backup)

Posted in Linux at 11:30 am by alfi

(Note: This has been sitting my Drafts folder for a while now… Time to get it out of here.)

A couple months ago I set up a VMware server and I immediately had problems with performance of the VM (especially the Win2K3 servers). They behaved sluggishly and thus I started googling. The folllowing is what I gathered from the VMware documentation and other people’s blog posts. Most of the information in the first part is from fewt’s blog post about Performance Tuning VMWare Server on Ubuntu Linux which he had just written when I was installing that machine.

The reason I’m writing this here now is that I had to install another server like it and found out that I neither had copies of the config files nor the scripts I wrote for the original machine (Yes. At that moment I felt like slamming my head against a wall) — meaning I had to do it all over again.

So this time I’m putting all the stuff here, so I know where to find it the next time around.

Tuning

I used the following settings on a system with 2x Intel Xeon E5335 (Quad-Core, 2Ghz), 10GB RAM and hardware RAID 5 (~3.6TB) running Debian 4.0 (etch) and VMware Server 1.0.6 and they provided a significant performance improvement compared to default settings.

The performance bottleneck is (as so often) disk I/O, so the goal is to reduce it as much as possible.

Step 1) Disable all swap files in the VMs (especially in Windows VMs).

This as well as all the following steps kinda assume that the host system has enough memory for all the VMs. And if we actually run out of real memory, we want the host system to do the swapping and not the VMs.

Step 2) Adjust the VMware parameters

Add the following to /etc/vmware/config to make it use shared memory for its temporary files (instead of putting them on disk) and always use a fixed size memory chunk (instead of resizing it on demand):

# put temp files in RAM...
mainMem.useNamedFile = FALSE
tmpDirectory = "/dev/shm"

# use a fixed size memory chunk, don't return unused memory to host OS...
prefvmx.useRecommendedLockedMemSize = "TRUE"
prefvmx.minVmMemPct = "100"

Make sure VMware Tools are installed in each VM and add the following to each VM configuration file (I also have these lines in /etc/vmware/config, but putting these at the end of each VM configuration prevents them from being overwritten):

# sync time with host OS + and power down the guest OS first when the VM is stopped...
tools.syncTime = "TRUE"
autostop = "softpoweroff"

# don't share memory between VMs...
mem.ShareScanTotal = 0
mem.ShareScanVM = 0
mem.ShareScanThreshold = 4096
sched.mem.maxmemctl = 0
sched.mem.pshare.enable = "FALSE"

# we basically already have this in the /etc/vmware/config,
# but it doesn't hurt to have it here again and make sure
mainMem.useNamedFile = "FALSE"
MemTrimRate = "0"
MemAllowAutoScaleDown = "FALSE"

Step 3) Adjust host OS parameters

Add the following to /etc/sysctl.conf and then activate it by executing ‘sysctl -p’:

vm.swappiness = 0
vm.overcommit_memory = 1
vm.dirty_background_ratio = 5
vm.dirty_ratio = 10
vm.dirty_expire_centisecs = 1000
dev.rtc.max-user-freq = 1024

It should should keep the OS from swapping unless absolutely necessary.

To make sure that there is enough memory at /dev/shm for the temporary files, modify /etc/default/tmpfs to use all available RAM (in this case 10GB):

SHM_SIZE=10G

Backup

This describes how to the backup the complete VMs (from the host) and opposed to backup the data of the VMs (with backup software installed in each VM).

This is a script I wrote to do the backup of each VM. It suspends the VM (unless specified otherwise), deletes the previous snapshot (thereby committing the changes of the previous day to the regular disk files), creates a new snapshot and restarts the VM. Then it creates a list of the snapshot files (which we don’t want to backup) and rsyncs all other files to the backup directory (which in my case is a NAS).

The script has a couple of configuration variables which are set at the beginning:
– VM_LIST contains a space-separated list of VMs to backup. The directory of the VM and the config files need to have the same name for this to work. By default the scripts suspends the VM, but I’ve experienced problems with that on some VMs (some software doesn’t seem to deal with it all that well and doesn’t work anymore after the resume), so I’ve added a way to make the script completely stop the VM instead of just suspending it (see example in script).
– VM_PATH is the path below which all the VMs sit (/var/vm in my case).
– BACKUP_PATH is the destination path for the backup.

The script checks for a file named ‘marker.nas1’ in BACKUP_PATH to make sure the NAS is actually mounted. Wouldn’t want it to dump all the VMs into the root file system (which will be out of space a couple minutes into the backup).

I also experienced some problems with bash’s default open file descriptor limit (which is 1024) during the deletion of the previous snapshot (when it tries to open all the regular disk and the snapshot files). While the system limit of open files on a Debian etch system with an amd64 kernel is close to 1 million, the limit of the shell still needs to be adjusted manually (either in the backup script itself which is what I’m doing or system-wide in /etc/bash.bashrc).

(This is the VMware Server 1.0.6 version of the script.)

#!/bin/sh
#
# backup VMs to NAS
# (suspend/shutdown + delete old snapshot, create new snapshot + restart, backup frozen VM files)
#
# alfi 2008/10/06
#

VM_LIST="FRITZ-FS:stop fritz-srv2k fritz-komm:stop Fritz-von_Hamm karma"
VM_PATH=/var/vm
#BACKUP_PATH=/mnt/nas1/VM_BACKUP
BACKUP_PATH=/mnt/backup/VM_BACKUP

WAIT_TIME=15

if [ ! -e ${BACKUP_PATH}/marker.nas1 ]; then
        echo "${BACKUP_PATH} is not available. Aborting...";
        exit 1
fi

TS="`date +%c`"
echo -e "${TS} Starting backup to '${BACKUP_PATH}'...\n"

# increase file descriptor limit
ulimit -n 100000

for VM_ARGS in $VM_LIST ;
do
        VM="`echo ${VM_ARGS} | cut -d: -f 1`"
        STOP_MODE="`echo ${VM_ARGS} | cut -d: -f 2`"
        if [ ${STOP_MODE} != "stop" ]; then
                STOP_MODE="suspend"
        fi

        echo "VM=${VM}, STOP_MODE=${STOP_MODE}"

        TS="`date +%c`"
        echo "${TS} Starting Backup of VM ${VM}..."

        echo "${TS} Retrieving current state of VM ${VM}..."
        VM_STATE="`vmware-cmd ${VM_PATH}/${VM}/${VM}.vmx getstate | sed -e 's/getstate()\s=\s\(\w*\)/\1/'`"

        TS="`date +%c`"
        echo "${TS} State of VM ${VM} is '${VM_STATE}'."

        if [ "${VM_STATE}" == "on" ]; then
                if [ ${STOP_MODE} != "stop" ]; then
                        # suspend VM
                        echo "${TS} Suspending VM ${VM}..."

                        vmware-cmd ${VM_PATH}/${VM}/${VM}.vmx suspend trysoft
                        RET=$?
                else
                        # shutdown VM
                        echo "${TS} Shutting down VM ${VM}..."

                        vmware-cmd ${VM_PATH}/${VM}/${VM}.vmx stop trysoft
                        RET=$?
                fi

                TS="`date +%c`"
                echo "${TS} vmware-cmd return code was ${RET}."

                if [ ${RET} == 7 ]; then
                        echo "${TS} Timeout while shutting down VM ${VM}? Waiting another ${WAIT_TIME} minutes..."
                        sleep ${WAIT_TIME}m
                fi
        fi

        # wait a little bit...
        sleep 5

        # delete old snapshot
        TS="`date +%c`"
        echo "${TS} Deleting previous snapshot of VM ${VM}..."

        vmrun deleteSnapshot ${VM_PATH}/${VM}/${VM}.vmx
        RET=$?

        TS="`date +%c`"
        echo "${TS} vmrun return code was ${RET}."

        # wait a little bit...
        sleep 5

        # create new snapshot
        TS="`date +%c`"
        echo "${TS} Creating new snapshot of VM ${VM}..."

        vmrun snapshot ${VM_PATH}/${VM}/${VM}.vmx
        RET=$?

        TS="`date +%c`"
        echo "${TS} vmrun return code was ${RET}."

        if [ "${VM_STATE}" == "on" ]; then
                # restart VM
                echo "${TS} Restarting VM ${VM}..."

                vmware-cmd ${VM_PATH}/${VM}/${VM}.vmx start
                RET=$?

                TS="`date +%c`"

                echo "${TS} vmware-cmd return code was ${RET}."
        fi
done

for VM_ARGS in $VM_LIST ;
do
        VM="`echo ${VM_ARGS} | cut -d: -f 1`"

        TS="`date +%c`"
        echo "${TS} Resuming backup of VM ${VM}..."

        # create exclude list for rsync
        echo "${TS} Creating rsync exclude-list for VM ${VM}..."
        find ${VM_PATH}/${VM} -regextype posix-extended -iregex ${VM_PATH}/${VM}/${VM}.\*-[0-9]{6}.\* -o -iname \*Snapshot\* | sed -e "s#$VM_PATH/##g" >/tmp/exclude.${VM}

        TS="`date +%c`"
        echo "${TS} find return code was ${RET}."

        # rsync files
        echo "${TS} Syncing files of VM ${VM}..."

        rsync -vaW --sparse --delete --delete-excluded --stats --exclude-from=/tmp/exclude.${VM} ${VM_PATH}/${VM} ${BACKUP_PATH}
        RET=$?

        TS="`date +%c`"
        echo "${TS} rsync return code was ${RET}."

        echo -e "\n"
done

TS="`date +%c`"
echo "${TS} Backup finished."

(When I upgrade the system to VMware Server 2 (probably around Christmas), I’ll post a VMware Server 2 version of the script. Basically the vmware-cmd (which does not exist anymore) calls need to be replaced. vmrun is now used for all commands. And of course the required parameters for vmrun changed a bit too.)

11.01.08

ECS 945GCT-D – Unknown device 1969:2048 (Attansic L2)

Posted in Linux at 10:27 am by alfi

A new board, a new NIC that’s not recognized. The ECS (ELITEGROUP) 945GCT-D is the Atom 1.6-powered fanless board that I’ll use in my new gateway box and of course it comes with yet another Attansic NIC that is not recognized or supported by the current kernel (and I mean current – I’m using 2.6.26-bpo.1 from etch-backports).

A link to the driver source can be found here (it links to this file).

After you downloaded + extracted the source you will find out that it doesn’t compile out-of-the-box with recent (meaning 2.6.23+) kernels.

Google(tm) turned up a blog post with a patch that’s supposed to make it compile — and it almost does. Still leaves you with unresolved SET_MODULE_OWNER. In the headers of Linux 2.6.26-bpo.1 this macro doesn’t exist anymore. In 2.6.18 it was still defined, even if it didn’t do anything (more information about SET_MODULE_OWNER is available at LWN.net).

After commenting out that line the driver compiled + loaded just fine and, so far at least, it seems to be working.