openMosix:
Edit: Minor Fixes and Typos
Apr 30, 2005 |
Two years later, the cluster returns!
What am I talking about? In the spring of 2003, I set out to setup a cluster. After successfully creating the cluster, I wrote an article for the site. For background information on my cluster experience, you may wish to read it here. It is a complete howto on setting up a Mosix cluster on x86 architecture.
Fast Forward to spring 2005, and I'm wanting to setup the cluster again. But the tools are aging and don't have enough functionality anymore. I decided to look into openMosix. OpenMosix is based on Mosix, but I switched to it for the following reasons:
- Developement stopped for Mosix. Any developement is for openMosix
- Support for openMosix has blossomed in the last couple years
- Certain tools are only available for openmosix, like autodiscd
This article I am writing today will hopefully get you from nothing to having a cluster in a decent amount of time.
Initial Setup
I covered initial setup in my previous howto, so I will not repeat that. You can view the initial setup
here
Also, there is a very good howto to setup a basic system
here. You only need to go as far as the Base System setup. There is no need to setup a web, mail or mysql server on the cluster. Also, when it comes time to run tasksel, don't run it, the remainder of this howto will cover all necessary software and packages.
Node Setup
First, I should explain the concept/topology of my setup. I use a head node, which has 2 network cards in it. This machine is accessible from the outside, and has a network card to access the machines on the inside. On the internal network, there are the dummy nodes. These machines do not get accessed directly. They are there simply for processes to migrate to. You do not have to setup your cluster in this fashion, but I think it makes it easier to manage.
I will assume at this point that you have a barebones system up and running. I do all my cluster setup at a console, and as root.
- #apt-get install wget
This is your basic web get app. You will use this to get your kernel source, and the openmosix source
- #wget http://internap.dl.sourceforge.net/sourceforge/openmosix/openMosix-2.4.26-1.bz2
Visit the openmosix site on sourceforge.net to find other mirrors. If you are getting different version of this patch, just make sure it matches the version of your kernel, otherwise your system may not work
- #wget http://www.kernel.org/pub/linux/kernel/v2.4/linux-2.4.26.tar.bz2
Again, make sure this is the same version as your patch. This is your vanilla kernel. It not recommended you try patching a kernel from your distro, as there might be compatibility issues. If you try apt-getting a kernel source, it might not work. I know as of this writing, the 2.4.26 debian kernel source does not work with the patch
- #wget http://internap.dl.sourceforge.net/sourceforge/openmosix/openmosix-tools-0.3.6-2.i386.rpm
This is the openmosix tools. Check the sourceforge site, if you are getting a different version for kernel version compatibility and other mirrors.
- #apt-get install kernel-package gcc libc6-dev make ssh bzip2 libncurses5-dev alien
Yes, that's all one line, one command. Just do it! It will install all the nexessary software to compile and install your openMosix enabled kernel.
- #bunzip2 openMosix-2.4.26-1.bz2
- #tar -vxjf linux-2.4.26.tar.bz2
- #cp openMosix-2.4.26-1 linux-2.4.26
You have just unzipped the kernel and the patch, and copied the patch into your kernel directory
- #cd linux-2.4.26
- #patch -p1 < openMosix-2.4.26-1
You now have a patched kernel source ready for configuration and compilation.
- #make menuconfig
This step will likely be the most challenging. There are certain things you must make sure you do, and some things that you will need to figure out on your own.
First, make sure the following features are enabled / set:
- Processor types and features --> Processor Family
[Pick your processor type]
enable MTRR
- Networking options --> Socket Filtering
[For DHCP client]
- Network Device Support --> [your Network Card]
[You're on your own. Find your network cards device driver. Use online support, ie: google, if you don't know your driver]
- File System --> Ext3 journalling file system support
[Not critical, but if you have any Ext3 partitions, you'll be glad you enabled this, because you won't be able to mount them otherwise]
For your head node, if you wish to have NAT and masquerading for your machines on the cluster subnet, you must enable the following things. Do not worry about these for dummy nodes
- Networking options --> Network packet filtering Y
- Networking options --> IP: Netfilter Configuration --> Connection Tracking Y
- Networking options --> IP: Netfilter Configuration --> IP Tables Support Y
- Networking options --> IP: Netfilter Configuration --> Full NAT Y
- Networking options --> IP: Netfilter Configuration --> Masquerade target support Y
- #make-kpkg kernel_image
This step may take a while depending on your machine. On a piece of junk, this could take up to an hour. On decent machines, this could be done in 5-10 minutes. If anything goes wrong in this step, you may have to go back through the config and verify everything is correct. You can also try the online community (ie: google, or debian mailling lists)
- #cd ..
- #dpkg -i kernel-image-2.4.26-om1_10.00.Custom_i386.deb
This installs your new kernel. It will also edit lilo so you can boot into the new kernel.
- #alien openmosix-tools-0.3.6-2.i386.rpm
Alien converts .rpm's to .deb's
- #dpkg -i openmosix-tools_0.3.6-2_i386.deb
- #update-rc.d openmosix start 90 2 3 4 5 . stop 80 0 6 .
Note: the periods [.] are critical in this command, do not forget them, and make sure there's a space on either side of each period, except the one at the end. This installs the tools needed to run the openMosix enabled kernel, and includes tools such as openMosix Auto Discovery Daemon.
- #init 6
Reboot your system to bring up the openMosix enabled kernel and it's tools
You should now have a fully functional openMosix node now.
Notes on the head node:
- You may want to do some extra configuration on the head node. This may include things such as installing Xserver and a window manager so you can actually run your apps. That is beyond the scope of this howto.
- You should also run some sort of DHCP server on the network with all the dummy nodes. Thanks to the openMosix autodiscovery daemon, all you need is to have the machine boot up and get network information from the DHCP server, and it can find and configure itself into the cluster on its own! DHCP server setup is quite simple, but again, not in the scope of this howto.
- Next, if you want your head node to have NAT and masquerading for the dummy nodes, you must do the following. In your favourite text editor (vi for console editing) create a file /etc/init.d/masquerade with the following 2 lines of text:
- iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
This assumes that eth0 is attached to the outside, while other interfaces are internal network connections to the cluster. Change ethX as appropriate
- echo 1 > /proc/sys/net/ipv4/ip_forward
Enable IP Forwarding
Once you have that file, you must enable this script to be run at boot time.
- #chmod 755 /etc/init.d/masquerade
Allow the file to be executed
- #update-rc.d masquerade start 90 S .
Once again, the period [.] is critical. The command will not work without it.
- Lastly, on your head node, the openMosix autodiscovery daemon defaults to running on eth0. If eth0 is not attached to the network with the rest of your dummy nodes, you will need to change with interface the daemon runs on. To change this, you need to edit "/etc/init.d/openmosix" with your favourite text editor. You will need to change the line [OMDISCD_OPTIONS="-i eth0"] to use the interface attached to your cluster network. (ie: eth0 -> eth1)
Any questions? Email me at Doug. Try the debian mail lists. There is a debian-beowulf list, as well as a general help list. They are both very friendly, and usually quite quick with responses. Good Luck!
