05.26.15

New Japan map (Re: Wrong transliteration on Garmin OSM maps)

Posted in Misc at 6:54 am by alfi

I’ve created a new map with the latest version of unihandecode and fresh OpenStreetMap data from geofabrik.

gmapsupp_JAPAN_20150525_unihandecode.img.bz2

09.16.14

Wrong transliteration on Garmin OSM maps

Posted in Misc at 6:48 am by alfi

mkgmap uses transliteration tables to convert labels to ASCII unless you specify unicode as your character set of choice (assuming your device supports it). In most cases this works fine, except when you are visiting Japan (or Korea, but less so). Japanese and Chinese use a lot of the same unicode characters, but they are read differently. The mkgmap transliteration table can obviously only contain on of them and it contains the Chinese one. Making the labels on Japanese OSM maps either hard to read or outright useless.

I ran into this problem during my visit to Japan last month and decided to figure out how to fix it before my next stop there.

First I tried generating new transliteration table rows for the unicode blocks containing CJK characters using the correct reading. I wrote a little script in Python that uses unihandecode to generate the files, replaced them in mkgmap.jar and generated a new map click reference. The result (from the couple spots I checked) looked a lot better, but I noticed a couple characters didn’t get transliterated correctly (I put a question mark into the table if unihandecode didn’t return anything for the character).

So I thought about running the whole label through unihandecode instead of processing it character by character with transliteration tables. I hacked together a new Transliterator implementation called ScriptTransliterator that allowed me to call an external script to do the actual transliteration work. Then I wrote another Python script that just ran its arguments through unihandecode in Japanese language mode and built a map of Japan.

My first complete run took almost two days on my old workstation, but since I was saving all the transliteration results in a cache file, the second one was a lot faster. The result looks okay to me (within the limitations of unihandecode), but obviously I didn’t do more than spot checking. I guess I’ll find out how good it actually is the next time I’m in Japan.

I’ve attached the generated map to this post, so feel free to try it out (and maybe give me some feedback, if you notice any problems). It was generated from the geofabrik extract I downloaded on the morning of 2014/09/13.

gmapsupp_JAPAN_20140912_unihandecode.img.bz2

06.12.14

Gtk-Sharp 2.99.3 for Windows (32bit)

Posted in .NET, Windows at 1:48 pm by alfi

Finally managed to build Gtk# 2.99.3 on/for Windows today (from master branch zip, downloaded 2014/06/11 07:39 UTC).

Steps:

  1. Set up clean MinGW/MSYS build environment (as described here – up to Installing a static pkg-config binary)
  2. Fix pkg-config installation (add the missing pkg.m4 which the gtk-sharp configure script needs)
  3. Extract Gtk+ 3.6.4 Windows bundle to /local32
  4. Fix PATH so the configure script can find gacutil.exe, csc.exe et al – Windows SDK Directory (for example: /c/Program Files/Microsoft SDKs/Windows/v7.1/bin) and .NET Framework Directory (for example: /c/Windows/Microsoft.NET/Framework/v4.0.3031)
  5. If you want to build the MSI then you also have to add the WiX Toolset to the PATH (for example: /c/Program Files (x86)/WiX Toolset v3.8/bin)
  6. Generate configure script (autoreconf -fi) – you will run into problems here if the pkg.m4 can’t be found (I just ended up just copying it to the m4 directory and running aclocal -I m4)
  7. Run configure script – you will run into problems here if the PATH isn’t set correctly and it can’t find all the build tools
  8. Run make

The MSI is available here.

 

08.17.13

Importing photos into YAPB

Posted in Linux at 6:43 pm by alfi

It’s been a while since my last post here, mostly due to lack of time (tho I’ve accumulated a couple half-finished pieces in my drafts folder). Lately I’ve been playing around with the Yet Another Photo Blog (YAPB) plugin for WordPress and while trying to make it do and look like I wanted it to (by modifying the ReflectionMod theme), I hacked together a little script (to be run on the server, but it would be simple enough to extend it to work from anywhere) to create posts from a directory full of photos and associate them with a category (“albums”) and tags. The photo blog in question is still not working exactly as I want it to, but I thought I would post the script since it might be useful to someone else out there.

The script yapb-post-file.php accepts only a category, tags and a filename as command line parameters. All the other necessities are hard-defined in the configuration file yapb-post-file.config.php. The script then simply executes the following steps:

  1. read EXIF/IPTC info from the image file
  2. create a new WordPress post (using the XML-RPC interface), setting the filename (dash and underscore characters are substituted with spaces) as post title, the capture date/time of the image as post date and the image comment as post content.
  3. copy the image file directly into the configured upload path (since YAPB circumvents the WordPress media handling – this is the only part than does not work over the network at the moment, tho it could be modified to upload the image using rsync/scp/ftp)
  4. create a new row in the wp_yapbimage database table associating the newly created post with the image file (that is what it needs the MySQL login info for)
  5. trigger an update of the post (again using the XML-RPC interface) to give YAPB and all the YAPB plugins an opportunity to do their thing

To import a whole directory of photos, one can simple run something like

for f in `ls /tmp/many-photos/*.jpg` ; do php yapb-post-file.php $f ; done

and be done with it.

The files are available here and all need to use it is PHP >= 5.3 with the XML-RPC module – and, of course, a WordPress installation with YAPB to use it with. After extracting the files you need to adjust the parameters in yapb-post-file.config.php to fit your installation and you’re good to go.

07.15.10

Automating OpenOffice.org using COM and C (Part 2…kinda)

Posted in Windows at 5:40 pm by alfi

I never got around to writing the second part (about OOo Calc), but today someone posted a comment asking how to do stuff with Calc documents and so I thought I would at least post a couple code snippets…

Creating a new sheet

I never had to create a new sheet because I always opened a blank document which already contained an empty sheet. But, if you believe the API documentation/examples in the wiki, you just have to call the “insertNewByName” method on the Sheets object (see the snippets below on how to get that one).

Manipulating cells

This little function sets the value (a number) <Value> into cell <X>/<Y> on sheet <Sheet>. The same principle applies for strings and formulas (just set the “String” or the “Formula” property instead).

long SCalc_SetCellValue( IDispatch* pDoc, short Sheet, long X, long Y, double Value )
{
 VARIANT     resultMe;
 IDispatch* pMe         = NULL;
 IDispatch* pSheet      = NULL;
 IDispatch* pCell       = NULL;
 VARIANT     parm[2];
 HRESULT     hr;

 VariantInit( &resultMe );
 hr = AutoWrap( DISPATCH_METHOD, &resultMe, pDoc, L"getSheets", 0 );
 pMe = resultMe.pdispVal;

 if( FAILED( hr ) )
   return FALSE;

 VariantInit( &resultMe );
 VariantInit( &parm[0] );
 parm[0].vt = VT_I4;
 parm[0].lVal = Sheet;
 hr = AutoWrap( DISPATCH_METHOD, &resultMe, pMe, L"getByIndex", 1, parm[0] );
 pSheet = resultMe.pdispVal;

 if( FAILED( hr ) )
   return FALSE;

 VariantInit( &resultMe );
 VariantInit( &parm[1] );
 parm[1].vt = VT_I4;
 parm[1].lVal = Y;
 VariantInit( &parm[0] );
 parm[0].vt = VT_I4;
 parm[0].lVal = X;
 hr = AutoWrap( DISPATCH_METHOD, &resultMe, pSheet, L"getCellByPosition", 2, parm[1], parm[0] );
 pCell = resultMe.pdispVal;

 if( FAILED( hr ) )
    return FALSE;

 VariantInit( &resultMe );
 VariantInit( &parm[0] );
 parm[0].vt = VT_R8;
 parm[0].dblVal = Value;
 hr = AutoWrap( DISPATCH_PROPERTYPUT, &resultMe, pCell, L"Value", 1, parm[0] );
 VariantClear( &parm[0] );

 return !FAILED( hr );
} // SCalc_SetCellValue

To retrieve the content of a cell, just use PROPERTY_GET for the relevant property (Value, String or Formula) of the cell, like this:

 VariantInit( &resultMe );
 hr = AutoWrap( DISPATCH_PROPERTYGET, &resultMe, pCell, L"String", 0 );
 bstrVal = resultMe.bstrVal;
*Text = ConvBSTR2A( bstrVal );
 VariantClear( &parm[0] );
 VariantClear( &resultMe );

Formatting cells

This little function sets the background color of a cell.

long SCalc_SetCellBgC( IDispatch* pDoc, short Sheet, long X, long Y, long RGB )
{
 VARIANT     resultMe;
 IDispatch* pMe         = NULL;
 IDispatch* pSheet      = NULL;
 IDispatch* pCell       = NULL;
 VARIANT     parm[2];
 HRESULT     hr;

 VariantInit( &resultMe );
 hr = AutoWrap( DISPATCH_METHOD, &resultMe, pDoc, L"getSheets", 0 );
 pMe = resultMe.pdispVal;

 if( FAILED( hr ) )
    return FALSE;

 VariantInit( &resultMe );
 VariantInit( &parm[0] );
 parm[0].vt = VT_I4;
 parm[0].lVal = Sheet;
 hr = AutoWrap( DISPATCH_METHOD, &resultMe, pMe, L"getByIndex", 1, parm[0] );
 pSheet = resultMe.pdispVal;

 if( FAILED( hr ) )
    return FALSE;

 VariantInit( &resultMe );
 VariantInit( &parm[1] );
 parm[1].vt = VT_I4;
 parm[1].lVal = Y;
 VariantInit( &parm[0] );
 parm[0].vt = VT_I4;
 parm[0].lVal = X;
 hr = AutoWrap( DISPATCH_METHOD, &resultMe, pSheet, L"getCellByPosition", 2, parm[1], parm[0] );
 pCell = resultMe.pdispVal;

 if( FAILED( hr ) )
    return FALSE;

 VariantInit( &resultMe );
 VariantInit( &parm[1] );
 parm[1].vt = VT_I4;
 parm[1].lVal = RGB;
 VariantInit( &parm[0] );
 parm[0].vt = VT_BSTR;
 parm[0].bstrVal = SysAllocString( L"CellBackColor" );
 hr = AutoWrap( DISPATCH_METHOD, &resultMe, pCell, L"setPropertyValue", 2, parm[1], parm[0] );
 VariantClear( &parm[0] );

 return !FAILED( hr );
} // SCalc_SetCellBgC

Setting the border color of cells is a bit more complicated, because you need create BorderLine object to do it:

void MakeBorderLine( IDispatch* pStardesktop, long Color, short InnerLineWidth, short OuterLineWidth, short LineDistance, IDispatch** BL )
{
 VARIANT     resultMe;
 VARIANT     parm[1];
 IDispatch* pdispValueObj;

 // create a property value object...
 VariantInit( &parm[0] );
 parm[0] kamagra gel australia.vt = VT_BSTR;
 parm[0].bstrVal = SysAllocString( L"com.sun.star.table.BorderLine" );
 AutoWrap( DISPATCH_METHOD, &resultMe, pStardesktop, L"Bridge_GetStruct", 1, parm[0] );
 pdispValueObj = resultMe.pdispVal;
 VariantClear( &parm[0] );

 // Color
 VariantInit( &resultMe );
 VariantInit( &parm[0] );
 parm[0].vt = VT_I4;
 parm[0].lVal = Color;
 AutoWrap( DISPATCH_PROPERTYPUT, &resultMe, pdispValueObj, L"Color", 1, parm[0] );

 // InnerLineWidth
 VariantInit( &resultMe );
 VariantInit( &parm[0] );
 parm[0].vt = VT_I2;
 parm[0].iVal = InnerLineWidth;
 AutoWrap( DISPATCH_PROPERTYPUT, &resultMe, pdispValueObj, L"InnerLineWidth", 1, parm[0] );

 // OuterLineWidth
 VariantInit( &resultMe );
 VariantInit( &parm[0] );
 parm[0].vt = VT_I2;
 parm[0].iVal = OuterLineWidth;
 AutoWrap( DISPATCH_PROPERTYPUT, &resultMe, pdispValueObj, L"OuterLineWidth", 1, parm[0] );

 // LineDistance
 VariantInit( &resultMe );
 VariantInit( &parm[0] );
 parm[0].vt = VT_I2;
 parm[0].iVal = LineDistance;
 AutoWrap( DISPATCH_PROPERTYPUT, &resultMe, pdispValueObj, L"LineDistance", 1, parm[0] );

 *BL = pdispValueObj;
} // MakeBorderLine

long SCalc_SetCellBorder( IDispatch* pDoc, short Sheet, long X, long Y, long Color, short Width, short Pos )
{
 IDispatch* pStardesktop         = NULL;
 VARIANT     resultMe;
 IDispatch* pMe                  = NULL;
 IDispatch* pSheet               = NULL;
 IDispatch* pCell                = NULL;
 VARIANT     parm[2];
 IDispatch* BLtop                = NULL;
 IDispatch* BLbottom             = NULL;
 IDispatch* BLleft               = NULL;
 IDispatch* BLright              = NULL;
 HRESULT     hr;

 // Get desktop and its assoc. IDispatch... 
 GetDesktop( pDoc, &pStardesktop );

 /// GET SHEET
 VariantInit( &resultMe );
 hr = AutoWrap( DISPATCH_METHOD, &resultMe, pDoc, L"getSheets", 0 );
 pMe = resultMe.pdispVal;

 if( FAILED( hr ) )
    return FALSE;

 VariantInit( &resultMe );
 VariantInit( &parm[0] );
 parm[0].vt = VT_I4;
 parm[0].lVal = Sheet;
 hr = AutoWrap( DISPATCH_METHOD, &resultMe, pMe, L"getByIndex", 1, parm[0] );
 pSheet = resultMe.pdispVal;

 if( FAILED( hr ) )
    return FALSE;
 ///

 /// GET CELL
 VariantInit( &resultMe );
 VariantInit( &parm[1] );
 parm[1].vt = VT_I4;
 parm[1].lVal = Y;
 VariantInit( &parm[0] );

 parm[0].vt = VT_I4;
 parm[0].lVal = X;
 hr = AutoWrap( DISPATCH_METHOD, &resultMe, pSheet, L"getCellByPosition", 2, parm[1], parm[0] );
 pCell = resultMe.pdispVal;

 if( FAILED( hr ) )
    return FALSE;
 ///

 if( Pos & SCALC_CELLBORDER_TOP )
 {
    MakeBorderLine( pStardesktop, Color, 0, Width, 0, &BLtop ); 

    // TopBorder
    VariantInit( &resultMe );
    VariantInit( &parm[0] );
    parm[0].vt = VT_DISPATCH;
    parm[0].pdispVal = BLtop;
    hr = AutoWrap( DISPATCH_PROPERTYPUT, &resultMe, pCell, L"TopBorder", 1, parm[0] );

    if( FAILED( hr ) )
       return FALSE;
  }
  if( Pos & SCALC_CELLBORDER_BOTTOM )
  {
     MakeBorderLine( pStardesktop, Color, 0, Width, 0, &BLbottom ); 

     // BottomBorder
     VariantInit( &resultMe );
     VariantInit( &parm[0] );
     parm[0].vt = VT_DISPATCH;
     parm[0].pdispVal = BLbottom;
     hr = AutoWrap( DISPATCH_PROPERTYPUT, &resultMe, pCell, L"BottomBorder", 1, parm[0] );

     if( FAILED( hr ) )
        return FALSE;
  }
  if( Pos & SCALC_CELLBORDER_LEFT )
  {
     MakeBorderLine( pStardesktop, Color, 0, Width, 0, &BLleft ); 

     // LeftBorder
     VariantInit( &resultMe );
     VariantInit( &parm[0] );
     parm[0].vt = VT_DISPATCH;
     parm[0].pdispVal = BLleft;
     hr = AutoWrap( DISPATCH_PROPERTYPUT, &resultMe, pCell, L"LeftBorder", 1, parm[0] );

     if( FAILED( hr ) )
        return FALSE;
  }
  if( Pos & SCALC_CELLBORDER_RIGHT )
  {
     MakeBorderLine( pStardesktop, Color, 0, Width, 0, &BLright ); 

     // RightBorder
     VariantInit( &resultMe );
     VariantInit( &parm[0] );
     parm[0].vt = VT_DISPATCH;
     parm[0].pdispVal = BLright;
     hr = AutoWrap( DISPATCH_PROPERTYPUT, &resultMe, pCell, L"RightBorder", 1, parm[0] );

     if( FAILED( hr ) )
        return FALSE;
  }

  return TRUE;
} // SCalc_SetCellBorder

Anyway, I hope this helps and sorry about the somewhat screwed up formatting of the code (it seems like I lost some TABs when I pasted from WebSVN).

01.09.10

Serial cable for Metrologic ScanPal2

Posted in Misc at 2:11 pm by alfi

12pin Lumberg connector to 9pin serial connector

Pin Lumberg12 Bezeichnung Pin Serial9
1 GND <-> GND 5
2 T1out (RS-232 out) <-> RX 2
3 R1in (RS-232 in) <-> TX 3

05.11.09

lftp 3.7.6 for Win32

Posted in Windows at 10:43 am by alfi

Had to do some scripted FTP transfers on Win32 and the standard Windows ftp client just wasn’t cutting it (transfers failed and I had no way of recognizing it). So naturally I thought of using lftp, but couldn’t find a native Win32 executable. So it’s back to Cygwin. But I didn’t want to install the whole Cygwin environment  just so I could use lftp. So I spent some time extracting DLLs from the packages on the Cygwin FTP server till I had all the required files together (download).

I used files from the following archives:

http://gd.tuwien.ac.at/gnu/gnu-win32/release/lftp/lftp-3.7.6-3.tar.bz2
ftp://ftp.cygwin.com/pub/cygwin/release/expat/libexpat1/libexpat1-2.0.1-1.tar.bz2
ftp://ftp.cygwin.com/pub/cygwin/release/libiconv/libiconv2/libiconv2-1.12-1.tar.bz2
ftp://ftp.cygwin levitra uk.com/pub/cygwin/release/gettext/libintl8/libintl8-0.17-3.tar.bz2
ftp://ftp.cygwin.com/pub/cygwin/release/readline/libreadline6/libreadline6-5.2.13-11.tar.bz2
ftp://ftp.cygwin.com/pub/cygwin/release/minires/minires-1.02-1.tar.bz2
ftp://ftp.cygwin.com/pub/cygwin/release/openssl/openssl-0.9.8k-1.tar.bz2
ftp://ftp.cygwin.com/pub/cygwin/release/cygwin/cygwin-1.5.25-15.tar.bz2

01.30.09

Samba 4 as Active Directory Server

Posted in Linux at 2:18 pm by alfi

Samba 4.0.0alpha6 was released on 2009/01/19 and while the final release (probably) is still a far way off, I thought it was time to find out how to setup Samba as an ADS.

My test system (or rather my test VM) was a clean Debian etch (4.0r6) AMD64 installation, which I dist-upgrade’d to lenny before beginning my quest.

1. Building Samba 4

There are Samba 4 packages in experimental, but I went for building my own fresh packages instead:

To be able to build Samba 4, some additional packages are required:

apt-get install build-essential svn-buildpackage git-core quilt autoconf fakeroot debhelper libparse-yapp-perl docbook-xsl docbook-xml xsltproc po-debconf libgnutls-dev libreadline5-dev libpam0g-dev libblkid-dev libattr1-dev flex bison tdb-dev libtalloc-dev libtalloc1 python2.5-dev libpopt-dev

Time to build the packages (as described here):

svn co svn://svn.debian.org/pkg-samba/trunk/samba4 samba4/debian
cd samba4/debian
./debian/rules get-orig-source
mkdir ../tarballs && mv *.orig.tar.gz ../tarballs
svn-buildpackage -rfakeroot

2. Installing Samba 4

Assuming the packages were built successfully (I had this fail when I tried it on a different machine a day after my first installation — but what do you expect building from GIT), we can now proceed to installing them… right after we fixed all the dependencies:

apt-get install libglib2.0-0 libglib2.0-data pkg-config python-tdb

cd ../build-area
dpkg -i *.deb

For me this went without a hitch..and without any configuration dialog asking for workgroup/domain as one has come to expect from installing Debian Samba packages.

So it came as not much of a surprise that the generated /etc/samba/smb.conf was pretty much useless. I just deleted it (provision will then create a new one).
The Samba 4 init script (at /etc/init.d/samba4) was equally useless. It still checked for the existence and tried to start smbd (the executable has been renamed to samba ). This is easily fixed by just replacing all occurences of smbd with samba.

3. Configuring Samba 4 as ADS

First make sure that the server’s entry in /etc/hosts points a real IP (and not just 127.0.0.1 as the installer sets it) or the DNS zone files generated by the provision script will be pretty useless.

If you want provision to create a smb.conf for you (which you probably do at this point), make sure that it doesn’t already exist.

/usr/share/samba/setup/provision --realm=SAMBA4.MYDOMAIN.ORG --domain=SAMBA4DOM --adminpass=samba4 --server-role='domain controller'

Now you can fire up Samba with

/etc/init.d/samba4 restart

and you’re almost done xylocaine sans ordonnance.

What’s left is setting up DNS by installing bind9 (if necessary) and integrating the files generated by the provision script:

apt-get install bind9
cp /var/lib/samba/private/samba4.mydomain.org.zone /etc/bind/
cat /var/lib/samba/private/named.conf >> /etc/bind/named.conf.local

Now edit /etc/bind/named.conf.local and fix the path to the zone file (from /var/lib/samba/private to /etc) and restart BIND:

/etc/init.d/bind9 restart

Check /etc/resolv.conf and make sure that the local DNS server is the first one listed there. If the machine gets its IP/DNS via DHCP (as mine does) then edit /etc/dhcp3/dhclient.conf (if you’re using dhcp3-client) and make sure there is a line like

prepend domain-name-servers 127.0.0.1;

in there (it’s commented by default) or you will just lose the change on the next renewal of the lease.

There is more stuff you can do (read /var/lib/samba/private/named.txt and/or Samba 4 HOWTO), but this enough to get things running.

At this point you join machines to the new domain (don’t forget to add the new server as the first DNS server or joining the domain might fail).

4. Administrating the Samba 4 ADS

Having set up a Samba 4 ADS is nice and all, but it’s pretty much useless without adding users and such. This is where it gets a bit tricky, because of the things that Samba 4 is still lacking is administrative tools/frontends.

First of all, you need UNIX users (same as you did when running Samba 3 as PDC) for all the users you want to create in the AD.
New AD users can then be added with the newuser script:

/usr/share/samba/setup/newuser --unixname my_unix_user my_samba_user my_pass

The unixname option can be omitted if the AD username is identical to the UNIX username and if you don’t specify the password the script will ask for it.

From here on out it’s a lot easier to just install the Windows 2003 Administration Pack on a Windows XP Pro box (or VM) and do the administrative stuff from there. For more information see the Samba 4 HOWTO.

You can log on to the domain with users created this way. You can add those users to groups (using dsa.msc from the Admin Pack). You can set file/directory permissions with those users/groups.

This is were my little experiment ended. I’ll probably try something like this again in another alpha or two and I really hope they add some way to properly administer this thing till then (if that doesn’t happen I will have to figure out how to change things using ldbedit).

11.13.08

VMware Server on Linux (Tuning and Backup)

Posted in Linux at 11:30 am by alfi

(Note: This has been sitting my Drafts folder for a while now… Time to get it out of here.)

A couple months ago I set up a VMware server and I immediately had problems with performance of the VM (especially the Win2K3 servers). They behaved sluggishly and thus I started googling. The folllowing is what I gathered from the VMware documentation and other people’s blog posts. Most of the information in the first part is from fewt’s blog post about Performance Tuning VMWare Server on Ubuntu Linux which he had just written when I was installing that machine.

The reason I’m writing this here now is that I had to install another server like it and found out that I neither had copies of the config files nor the scripts I wrote for the original machine (Yes. At that moment I felt like slamming my head against a wall) — meaning I had to do it all over again.

So this time I’m putting all the stuff here, so I know where to find it the next time around.

Tuning

I used the following settings on a system with 2x Intel Xeon E5335 (Quad-Core, 2Ghz), 10GB RAM and hardware RAID 5 (~3.6TB) running Debian 4.0 (etch) and VMware Server 1.0.6 and they provided a significant performance improvement compared to default settings.

The performance bottleneck is (as so often) disk I/O, so the goal is to reduce it as much as possible.

Step 1) Disable all swap files in the VMs (especially in Windows VMs).

This as well as all the following steps kinda assume that the host system has enough memory for all the VMs. And if we actually run out of real memory, we want the host system to do the swapping and not the VMs.

Step 2) Adjust the VMware parameters

Add the following to /etc/vmware/config to make it use shared memory for its temporary files (instead of putting them on disk) and always use a fixed size memory chunk (instead of resizing it on demand):

# put temp files in RAM...
mainMem.useNamedFile = FALSE
tmpDirectory = "/dev/shm"

# use a fixed size memory chunk, don't return unused memory to host OS...
prefvmx.useRecommendedLockedMemSize = "TRUE"
prefvmx.minVmMemPct = "100"

Make sure VMware Tools are installed in each VM and add the following to each VM configuration file (I also have these lines in /etc/vmware/config, but putting these at the end of each VM configuration prevents them from being overwritten):

# sync time with host OS + and power down the guest OS first when the VM is stopped...
tools.syncTime = "TRUE"
autostop = "softpoweroff"

# don't share memory between VMs...
mem.ShareScanTotal = 0
mem.ShareScanVM = 0
mem.ShareScanThreshold = 4096
sched.mem.maxmemctl = 0
sched.mem.pshare.enable = "FALSE"

# we basically already have this in the /etc/vmware/config,
# but it doesn't hurt to have it here again and make sure
mainMem.useNamedFile = "FALSE"
MemTrimRate = "0"
MemAllowAutoScaleDown = "FALSE"

Step 3) Adjust host OS parameters

Add the following to /etc/sysctl.conf and then activate it by executing ‘sysctl -p’:

vm.swappiness = 0
vm.overcommit_memory = 1
vm.dirty_background_ratio = 5
vm.dirty_ratio = 10
vm.dirty_expire_centisecs = 1000
dev.rtc.max-user-freq = 1024

It should should keep the OS from swapping unless absolutely necessary.

To make sure that there is enough memory at /dev/shm for the temporary files, modify /etc/default/tmpfs to use all available RAM (in this case 10GB):

SHM_SIZE=10G

Backup

This describes how to the backup the complete VMs (from the host) and opposed to backup the data of the VMs (with backup software installed in each VM).

This is a script I wrote to do the backup of each VM. It suspends the VM (unless specified otherwise), deletes the previous snapshot (thereby committing the changes of the previous day to the regular disk files), creates a new snapshot and restarts the VM. Then it creates a list of the snapshot files (which we don’t want to backup) and rsyncs all other files to the backup directory (which in my case is a NAS).

The script has a couple of configuration variables which are set at the beginning:
– VM_LIST contains a space-separated list of VMs to backup. The directory of the VM and the config files need to have the same name for this to work. By default the scripts suspends the VM, but I’ve experienced problems with that on some VMs (some software doesn’t seem to deal with it all that well and doesn’t work anymore after the resume), so I’ve added a way to make the script completely stop the VM instead of just suspending it (see example in script).
– VM_PATH is the path below which all the VMs sit (/var/vm in my case).
– BACKUP_PATH is the destination path for the backup.

The script checks for a file named ‘marker.nas1’ in BACKUP_PATH to make sure the NAS is actually mounted. Wouldn’t want it to dump all the VMs into the root file system (which will be out of space a couple minutes into the backup).

I also experienced some problems with bash’s default open file descriptor limit (which is 1024) during the deletion of the previous snapshot (when it tries to open all the regular disk and the snapshot files). While the system limit of open files on a Debian etch system with an amd64 kernel is close to 1 million, the limit of the shell still needs to be adjusted manually (either in the backup script itself which is what I’m doing or system-wide in /etc/bash.bashrc).

(This is the VMware Server 1.0.6 version of the script.)

#!/bin/sh
#
# backup VMs to NAS
# (suspend/shutdown + delete old snapshot, create new snapshot + restart, backup frozen VM files)
#
# alfi 2008/10/06
#

VM_LIST="FRITZ-FS:stop fritz-srv2k fritz-komm:stop Fritz-von_Hamm karma"
VM_PATH=/var/vm
#BACKUP_PATH=/mnt/nas1/VM_BACKUP
BACKUP_PATH=/mnt/backup/VM_BACKUP

WAIT_TIME=15

if [ ! -e ${BACKUP_PATH}/marker.nas1 ]; then
        echo "${BACKUP_PATH} is not available. Aborting...";
        exit 1
fi

TS="`date +%c`"
echo -e "${TS} Starting backup to '${BACKUP_PATH}'...\n"

# increase file descriptor limit
ulimit -n 100000

for VM_ARGS in $VM_LIST ;
do
        VM="`echo ${VM_ARGS} | cut -d: -f 1`"
        STOP_MODE="`echo ${VM_ARGS} | cut -d: -f 2`"
        if [ ${STOP_MODE} != "stop" ]; then
                STOP_MODE="suspend"
        fi

        echo "VM=${VM}, STOP_MODE=${STOP_MODE}"

        TS="`date +%c`"
        echo "${TS} Starting Backup of VM ${VM}..."

        echo "${TS} Retrieving current state of VM ${VM}..."
        VM_STATE="`vmware-cmd ${VM_PATH}/${VM}/${VM}.vmx getstate | sed -e 's/getstate()\s=\s\(\w*\)/\1/'`"

        TS="`date +%c`"
        echo "${TS} State of VM ${VM} is '${VM_STATE}'."

        if [ "${VM_STATE}" == "on" ]; then
                if [ ${STOP_MODE} != "stop" ]; then
                        # suspend VM
                        echo "${TS} Suspending VM ${VM}..."

                        vmware-cmd ${VM_PATH}/${VM}/${VM}.vmx suspend trysoft
                        RET=$?
                else
                        # shutdown VM
                        echo "${TS} Shutting down VM ${VM}..."

                        vmware-cmd ${VM_PATH}/${VM}/${VM}.vmx stop trysoft
                        RET=$?
                fi

                TS="`date +%c`"
                echo "${TS} vmware-cmd return code was ${RET}."

                if [ ${RET} == 7 ]; then
                        echo "${TS} Timeout while shutting down VM ${VM}? Waiting another ${WAIT_TIME} minutes..."
                        sleep ${WAIT_TIME}m
                fi
        fi

        # wait a little bit...
        sleep 5

        # delete old snapshot
        TS="`date +%c`"
        echo "${TS} Deleting previous snapshot of VM ${VM}..."

        vmrun deleteSnapshot ${VM_PATH}/${VM}/${VM}.vmx
        RET=$?

        TS="`date +%c`"
        echo "${TS} vmrun return code was ${RET}."

        # wait a little bit...
        sleep 5

        # create new snapshot
        TS="`date +%c`"
        echo "${TS} Creating new snapshot of VM ${VM}..."

        vmrun snapshot ${VM_PATH}/${VM}/${VM}.vmx
        RET=$?

        TS="`date +%c`"
        echo "${TS} vmrun return code was ${RET}."

        if [ "${VM_STATE}" == "on" ]; then
                # restart VM
                echo "${TS} Restarting VM ${VM}..."

                vmware-cmd ${VM_PATH}/${VM}/${VM}.vmx start
                RET=$?

                TS="`date +%c`"

                echo "${TS} vmware-cmd return code was ${RET}."
        fi
done

for VM_ARGS in $VM_LIST ;
do
        VM="`echo ${VM_ARGS} | cut -d: -f 1`"

        TS="`date +%c`"
        echo "${TS} Resuming backup of VM ${VM}..."

        # create exclude list for rsync
        echo "${TS} Creating rsync exclude-list for VM ${VM}..."
        find ${VM_PATH}/${VM} -regextype posix-extended -iregex ${VM_PATH}/${VM}/${VM}.\*-[0-9]{6}.\* -o -iname \*Snapshot\* | sed -e "s#$VM_PATH/##g" >/tmp/exclude.${VM}

        TS="`date +%c`"
        echo "${TS} find return code was ${RET}."

        # rsync files
        echo "${TS} Syncing files of VM ${VM}..."

        rsync -vaW --sparse --delete --delete-excluded --stats --exclude-from=/tmp/exclude.${VM} ${VM_PATH}/${VM} ${BACKUP_PATH}
        RET=$?

        TS="`date +%c`"
        echo "${TS} rsync return code was ${RET}."

        echo -e "\n"
done

TS="`date +%c`"
echo "${TS} Backup finished."

(When I upgrade the system to VMware Server 2 (probably around Christmas), I’ll post a VMware Server 2 version of the script. Basically the vmware-cmd (which does not exist anymore) calls need to be replaced. vmrun is now used for all commands. And of course the required parameters for vmrun changed a bit too.)

11.01.08

ECS 945GCT-D – Unknown device 1969:2048 (Attansic L2)

Posted in Linux at 10:27 am by alfi

A new board, a new NIC that’s not recognized. The ECS (ELITEGROUP) 945GCT-D is the Atom 1.6-powered fanless board that I’ll use in my new gateway box and of course it comes with yet another Attansic NIC that is not recognized or supported by the current kernel (and I mean current – I’m using 2.6.26-bpo.1 from etch-backports).

A link to the driver source can be found here (it links to this file).

After you downloaded + extracted the source you will find out that it doesn’t compile out-of-the-box with recent (meaning 2.6.23+) kernels.

Google(tm) turned up a blog post with a patch that’s supposed to make it compile — and it almost does. Still leaves you with unresolved SET_MODULE_OWNER. In the headers of Linux 2.6.26-bpo.1 this macro doesn’t exist anymore. In 2.6.18 it was still defined, even if it didn’t do anything (more information about SET_MODULE_OWNER is available at LWN.net).

After commenting out that line the driver compiled + loaded just fine and, so far at least, it seems to be working.

« Previous entries Next Page » Next Page »