Bear in mind that there are some applications that migrate very nicely over an openMosix cluster, for example 'make' when you're compiling can speed up your compile times significantly. If you do a little research on the net, you'll find examples of applications that will migrate well and which won't yield much speed increase. If you are a developer looking to take advantage of openMosix, applications that fork() child processes will migrate wonderfully, whereas multithreaded applications at present, do not seem to migrate threads.

Anyway, here are a couple of cool uses for your cluster:

Distributed Password Cracking

If your role is in security role or if you work as a penetration tester, you'll probably encounter the need to crack passwords at some point or other. We regularly use l0phtcrack for Windows passwords, but were interested by the opportunity to use our nice new 10 system cluster to significantly speed things up. After briefly hunting around the net, we discovered 'Cisilia', which is a Linux based Windows LM / NTLM password cracker designed specifically to take advantage of openMosix style clustering!

You can get a copy of cisilia by visiting the following site and clicking on the R&D Projects menu on the left: http://www.citefa.gov.ar/SitioSI6_EN/si6.htm

There you'll find two files, 'cisilia' which is the actual command line based password cracking engine and 'xisilia' which is an X based GUI for the same. We didn't install the X based GUI, since we were working with our cluster completely over SSH.

Once you download the RPMs, you can install them by typing:

rpm -ivh *isilia*.rpm

If you installed from the tarball sources like we did, it is just as simple:

1) Unzip the tarball

tar xvzf cisilia*.tar.gz

2) Enter the directory and configure the compilation process for your system:

./configure

3) Finally, start the compilation process:

make

Now you need to get a Windows password file to crack. For this you'll want to use pwdump to grab the encrypted password hashes. This is available at the following link:

https://packetstormsecurity.com/files/13790/pwdump2.zip.html

Unzip it and run it on the Windows box which has the passwords you want to crack. You will want to save the results to a file, so do the following:

pwdump2 > passwdfile

Now copy the file 'passwdfile' across to a node in your cluster. Fire up cisilia using the following command:

cisilia -l crack_file -n 20 <path to the passwdfile you copied>

•  -l   tells cisilia to save the results to a file called crack_file

•  -n  tells cisilia how many processes it should spawn. We started 20, since we wanted 2 processes to go to each node in the cluster.

We were pleasantly surprised by how quickly it started running through 6-7 character alphanumeric passwords. Enjoy !

Encoding MP3s

Do you get annoyed by how long it takes to convert a CD to MP3? Or to convert any kind of media file?

This is one of the places where a cluster excels. When you convert your rips to MP3, you only process one WAV file at a time, how about you run it on your cluster and let it simultaneously encode all your MP3s?

Someone has already taken this to the absolute extreme, check out http://www.rimboy.com/cluster/ for what he's got setup.

To quickly rip a CD and convert it to digital audio, you will need 2 programs:

A digital audio extractor, and an audio encoder.

For the digital audio extractor we recommend Cdparanoia. For the audio encoder, we're going to do things a bit differently:

In the spirit of the free open source movement, we suggest you check out the OGG Vorbis encoder. This is a free, open audio compression standard that will compress your WAV files much better than MP3, and still have a higher quality!

They also play perfectly in Winamp and other media players. Sounds too good to be true? Check out their website at the link below. Of course if you still aren't convinced that OGG is better than MP3, you can replace the OGG encoder with any MP3 encoder for this tutorial.

Get and install both the cdparanoia ripper and the oggenc encoder from the following URLs:

CDparanoia - http://www.xiph.org/paranoia/

OGG Vorbis Encoder - https://xiph.org/vorbis/

Now we just need to rip and encode on our cluster. Put the CD you want to convert in the drive on one node, and just run the following:

cdparanoia -B

for i in `ls *.wav`;

do oggenc $i &

done;

This encodes your WAV files to OGG format at the default quality level of 3, which produces an OGG file of a smaller size and significantly better sound quality than an MP3 at 128kbps. You can experiment with the OGG encoder options to figure out the best audio quality for your requirements.

This just about completes the openmosix tutorial we've prepaired for you.

We surely hope it has been an enlighting tutorial and will help most of you make some good use of those old 'mini super computers' you never knew you had :)

Back to the Linux/Unix Section or OpenMosix Section.

This is just a quick walk-through on how to do that, for a much more detailed explanation on public/private key SSH authentication, see our tutorial in the GNU/Linux Section.

First, generate your SSH public/private key-pair:

ssh-keygen -t dsa

Second, copy the public key into the authorized keys file. Since your home directory is shared between nodes, you only need to do this on one node:

cat ~/.ssh/*.pub >>~/.ssh/authorized_keys

However, for root, you will have to do this manually for each node (replace Node# with each node individually):

cat ~/.ssh/*.pub >>/mfs/Node#/root/.ssh/authorized_keys

After this, you have to start SSH-agent, to cache your password so you only need to type it once. Add the following to your .bash_profile or .profile:

ssh-agent $SHELL

Now each time after you login, just type ‘ ssh-add' and supply your password once. By following this you will be able to login passwordless to any of the nodes, and the sliders in openMosixview should work perfectly for you. Next: Interesting Ideals: Distributed Password Cracking & Encoding MP3s

For example, if we run a process that needs to read some data from a file /etc/test.conf on our local machine, if this process migrates to another node, how will openMosix read that file ? The answer is in the openMosix File System, or OMFS.

OMFS does several things. Firstly, it shares your disk between all the nodes in the cluster, allowing them to read and write to the relevant files. It also uses what is known as Direct File System Access (DFSA), which allows a migrated process to run many system calls locally, rather than wasting time executing them on the home node. It works somewhat like NFS, but has features that are required for clustering.

If you installed openMosix from the RPMs, the omfs should already be created and automatically mounted. Have a look in /mfs, and you will see a subdirectory for every node in the cluster, named after the node ID. These directories will contain the shared disks of that particular node.

You will also see some symlinks like the following:

here -> maps to the current node where your process runs

home -> maps to your home node

If the /mfs directory has not been created, you can mount it manually with the following:

mkdir /mfs

mount /mfs /mfs -t mfs

If you want it to be automatically mounted at boot time, you can create the following entry in your /etc/fstab

mfs_mnt /mfs mfs dfsa=1 0 0

Bear in mind that this entry has to be on all the nodes in the cluster. Lastly, you can turn the openMosix file system off using the command:

mosctl nomfs

Now that we've got that all covered, it's time to take a look on how you can make the ssh login process less time consuming, allowing you to take control of all your cluster nodes any time you require, but also help the cluster system execute special functions. Next topic covers using SSH keys with openMosix instead of passwords.

From the command line, the main monitoring and control tools are:

• mosmon – which shows you the load on each of the nodes, their speed, memory usage, etc. Pressing 'h' will bring up the help with the different options;

• mosctl - which is a very powerful command that allows you to control how your system behaves in the cluster, some of the interesting options are:
  • mosctl block – this stops other people's processes being run on your system (a bit selfish don't you think ;))
  • mosctl -block – the opposite of the above
  • mosctl lstay – this stops your local processes migrating to other nodes for processing
  • mosctl nolstay – the opposite of the above
  • mosctl setspeed <number> - which sets the max processing speed to contribute. 10000 is a benchmark of a Pentium 3 1Ghz.
  • mosctl whois <node number> - this tells you the IP address of a particular node
  • mosctl expel – this expels any current remote processes and blocks new ones from coming in
  • mosctl bring – this brings back any of your own local processes that have migrated to other nodes
  • mosctl status <node number> - which shows you whether the node is up and whether it is 'blocking' processes, 'staying' them, etc.
  • mosrun - allows you to run a process controlling which nodes it should run on
  • mps - this is just like 'ps' to show you the process listing, but it also shows which node a process is running on
  • migrate - this command allows you to manually migrate a process to any node you like, the syntax for using it is 'migrate <pid> <node #>'. You can also use 'migrate <pid> balance' to load balance a process automatically.
  • dsh - Distributed Shell. This allows you to run a command on all the nodes simultaneously. For example ‘ dsh -a reboot ' will reboot all the nodes.

From the GUI, you can just start 'openmosixview'. This allows you to view and manage all the nodes in your cluster. It also shows you the load balancing efficiency of the cluster in near-real-time. You can also see what is the total speed and RAM that your cluster is providing you:

We should note that all cluster nodes that are online are represented with the green colour, while all offline cluster with the red colour.

One of the neatest things about 'openmosixview' is the GUI for controlling process migration.

It depicts your current node at the center, and other nodes in the cluster around it. The ring around your node represents the processes running on your local box. If you hover over any of them you can see the process name and PID. Whenever one of your processes migrates to another node, you will see it detach and appear on a new node with a line linking it to your system!

You can also manually control the migration. You can drag and drop your processes onto other nodes, even selecting multiple processes and then dragging them to another node is easy. If you double click on a process running on a remote node, it will come back home and execute locally.

You can also open the openMosix process monitor which shows you which process is running on which node.

There is also a history analyzer to show you the load over a period of time. This allows you to see how your cluster was being used at any given point in time:

As you can see, the GUI tools are very powerful, they provide you with a large amount of the functionality that the command line tools do. If, however, you want to make your own scripts, the command line tools are much more versatile. Managing a cluster can be a lot of fun, modify the options and play around with the GUI to tweak and optimize your raw processing power!! Our next article covers The openMosix File System.

Type 'mosmon' and press enter. You should see a screen similar to the screenshot below:

Run mosmon in one VTY (press ctrl+alt+f1), then switch to another VTY (ctrl+alt+f2)

Let's run a simple awk command to run a nested loop and use up some processing power. If everything went well, we should see the load in mosmon jump up on one node, and then migrate to the other nodes.

The command you need to run is:

awk 'BEGIN {for(i=0;i<10000;i++)for(j=0;j<10000;j++);}'

If you choose to, you can start multiple awk processes by backgrounding them. Just append an ‘&' to the command line and run it a few times.

Go back to mosmon by pressing ctrl+alt+f1, you should see the load rising on your current node, and then slowly distributing to the other machines in the cluster like in the picture below:

Congratulations! You are now taking advantage of multi system clustering!

If you want you can time the process running locally, turn off openmosix by entering the command:

/etc/init.d/openmosix stop

Then run the following script:

#!/bin/sh

date

awk 'BEGIN {for(i=0;i<10000;i++)for(j=0;j<10000;j++);}'

date

This will tell you how long it took to perform the task. You can modify the loop values to make it last longer. Now restart openmosix, using the command:

/etc/init.d/openmosix start

Re-run the script to see how long it takes to process. Remember that your network is a bottleneck for performance. If your process finishes really quickly, it won't have time to migrate to the other nodes over the network. This is where tweaking and optimizing your cluster becomes fun.

Next up we'll take a look on how you can control a OpenMosix cluster.

OpenMosix has two ways of doing this:

1. Auto-discovery of Cluster Nodes

OpenMosix includes a daemon called 'omdiscd' which identifies other openMosix nodes on the network by using multicast packets (for more on multicasting, please see our multicast page). This means that you don't have to bother manually configuring the nodes. This is a simple way to get your cluster going as you just need to boot a machine and ensure it's on the network. When this stage is complete, it should then discover the existing cluster and add itself automatically!

Make sure you set up your network properly. As an example, if you are assigning an IP address of 192.168.1.10 to your first ethernet interface and your default gateway is 192.168.1.1 you would do something like this:

ifconfig eth0 192.168.1.1 netmask 255.255.255.0 broadcast 192.168.1.255 up (configure's your system's ethernet interface)

route add default gw 192.168.1.1 (adds the default gateway)

The auto-discovery daemon might have started automatically on bootup, check using:

ps aux | grep 'omdiscd'

The above command should reveal the 'omdiscd' process running on your system.

If it hasn't, you can manually start by typing 'omdiscd'. If you want to see the nodes getting added, you can choose to run omdiscd in the foreground by typing 'omdiscd -n'. This will help you troubleshoot the auto-discovery.

2. The /etc/openmosix.map File Configuration

If you don't want to use autodiscovery, you can manually manage your nodes using the openmosix.map file in the /etc directory. This file basically contains a list of the nodes on your cluster, and has to be the same across all the nodes in your cluster.

The syntax is very simple, it is a tab delimited list of the nodes in your cluster. There are 3 fields:

Node ID, IP Address and Number.

•  Node ID is the unique number for the node.

•  IP address is the IP address of the node.

•  Number specifies how many nodes in the range after the IP.

As an example, if you have nodes

192.168.1.10

192.168.1.11

192.168.1.12

192.168.1.50

your file would look like this:

1 192.168.1.10 2

2 192.168.1.50 1

We could have manually specified the IP's 192.168.1.11 and 192.168.1.12, but by using the 'number' field, openmosix counts up the last octet of the IP, and saves you the trouble of making individual entries.

Once you've done your configuration, you can control openMosix using the init.d script that should have been installed. If they were not, you can find it in the scripts directory of the userland tools you downloaded, make it executable and copy it to the init.d directory like this:

mv ./openmosix /etc/init.d

chmod 755 /etc/init.d/openmosix

You can now start, stop and restart openMosix with the following commands:

/etc/init.d/openmosix start

/etc/init.d/openmosix stop

/etc/init.d/openmosix restart

Next up we'll take a look on how you can test your new openMosix cluster!

The best thing about Cluster Knoppix, is that you can just boot a system with the CD and it will automatically add itself to the cluster. You don't even need to install the O/S to your hard disk. This makes it a very useful way to setup a cluster in a hurry using pre-existing systems.

Another really nice feature is that you don't need to burn 20 copies of the CD to make a 20 system cluster. Just boot one system with the CD, and then run the command

knoppix-terminalopenmosixserver

This will let you setup a clustering-enabled terminal-server. Now if you have any systems that can boot from their network card (PXE compliant booting), they will automatically download a kernel image and run Cluster Knoppix!

It's awesome to see this at work, especially since we were working with 2 systems that didn't have a CD-ROM drive or a hard-disk. They just became diskless clients and contributed their resources to the cause! Next page covers starting up your openMosix Cluster.

We will require the following:

1. Two or more machines (we need to cluster something!), the configuration doesn't matter even if they are lower end. They will require network cards and need to be connected to each other over a network. Obviously, the more systems you have, the more powerful your cluster will be. Don't worry if you don't have many machines, we'll show you how to temporarily use resources from systems and schedule when they can contribute their processing power (this works very well in an office when you might want some systems to join the cluster only after office hours).

2. A Cluster Knoppix LiveCD for the second part of this tutorial. While this is not strictly necessary, we want to show you some of the advantages of using the LiveCD for clustering. It also makes setting up the cluster extremely easy. You can get a fully working cluster up in the amount of time it takes you to boot a system ! You can get Cluster Knoppix from the following link: https://distrowatch.com/table.php?distribution=clusterknoppix

Getting & Installing openMosix

OpenMosix consists of two parts, the first is the kernel patch which does the actual clustering and the second is the userland tools that allow you to monitor and control your cluster.

There are a variety of ways to install openMosix, we've chosen to show three of them:

1. Patching the kernel and installing from the source

2. Installing from RPM's

3. Installing in Debian

1. Installing from source

The latest version of openMosix at the time of this writing works with the kernel version 2.4.24. If you want to do this the proper way, get the plain kernel sources for 2.4.24 from https://www.kernel.org/ and the openMosix patch for the same version of the kernel from https://sourceforge.net/projects/openmosix/

At the time of writing this, the direct kernel source link is

http://www.kernel.org/pub/linux/kernel/v2.4/linux-2.4.24.tar.bz2

Once you've got the kernel sources, unpack them to your kernel source directory, in this case that should be:

/usr/src/linux-2.4.24

Now move the openMosix patch to the kernel source directory and apply it, like so:

mv /root/openMosix-2.4.24.gz /usr/src/linux-2.4.24

cd /usr/src/linux-2.4.24

zcat openMosix-2.4.24.gz | patch -Np1

NOTE: If you downloaded a bzip zipped file, you might need to use 'bzcat' rather than 'zcat' in the last line.

Now your kernel sources are patched with openMosix.

Now you have to configure your kernel sources, using one of the following commands:

make config

make menuconfig (uses an ncurses interface)

make xconfig (uses a TCL/TK GUI interface)

If you use X and have a recent distribution, 'make xconfig' is your best bet. Once you get the kernel configuration screens, enable the following openMosix options in the kernel configuration:

CONFIG_MOSIX=y

# CONFIG_MOSIX_TOPOLOGY is not set

CONFIG_MOSIX_UDB=y

# CONFIG_MOSIX_DEBUG is not set

# CONFIG_MOSIX_CHEAT_MIGSELF is not set

CONFIG_MOSIX_WEEEEEEEEE=y

CONFIG_MOSIX_DIAG=y

CONFIG_MOSIX_SECUREPORTS=y

CONFIG_MOSIX_DISCLOSURE=3

CONFIG_QKERNEL_EXT=y

CONFIG_MOSIX_DFSA=y

CONFIG_MOSIX_FS=y

CONFIG_MOSIX_PIPE_EXCEPTIONS=y

CONFIG_QOS_JID=y

Feel free to tweak your other kernel settings based on your hardware and requirements just as you would when installing a new kernel.

Finally, finish it all off by compiling the kernel with:

make dep bzImage modules modules_install

Now install your new kernel in your bootloader. For example, if you use LILO, edit your /etc/lilo.conf and create a new entry for your openMosix enhanced kernel. If you simply copy the entry for your regular kernel and change the kernel image to point to your new kernel, this should be enough. Don't forget to run 'lilo' when you finish editing the file.

After you have completed this, reboot, and if all went well, you should be able to select the openMosix kernel you just installed and boot with it. If something didn't work right, you can still select your regular kernel and boot normally to troubleshoot.

2. Installing from RPM

If you have an RPM based distribution, you can directly get a pre-compiled kernel image with openMosix enabled from the openMosix site (https://sourceforge.net/projects/openmosix/).

This is a fairly easy way to install openMosix as you just need to install 2 RPMs. This should work with Red Hat, SUSE etc. Get the two latest RPMs for the

a) openmosix-kernel

b) openmosix-tools

Now you can simply install both of these by using the command:

rpm -Uvh openmosix*.rpm

If you are using GRUB, the RPM's will even make the entry in your GRUB config so you can just reboot and select the new kernel. If you use LILO you will have to manually make the entry in /etc/lilo.conf. Simply copying the entry for your regular kernel and changing the kernel image to point to your new kernel should be enough. Don't forget to run 'lilo' when you finish editing the file.

That should be all you need to do for the RPM based installation. Just reboot and choose the openMosix kernel when you are given the choice.

3. Installing in Debian

You can install the RPM's in Debian as well as using Alien, but it is better to use apt-get to install the kernel sources and openmosix kernel patch. You can also use the 'apt-get' command to install openmosixview, which will give you a GUI to manage the cluster.

This is the basic procedure needed to follow for installing openMosix under Debian.

First, get the packages:

cd /usr/src

apt-get install kernel-source-2.4.24 kernel-package \

openmosix kernel-patch-openmosix

Untar them and create the links:

tar vxjf kernel-source-2.4.24.tar.bz2

ln -s /usr/src/kernel-source-2.4.24 /usr/src/linux

Apply the patch:

cd /usr/src/linux

../kernel-patches/i386/apply/openmosix

Install the kernel:

make menuconfig

make-kpkg kernel_image modules_image

cd ..

dpkg -i kernel-image-*-openmosix-*.deb

After this you can use 'apt-get' to install the openmosixview GUI utility for managing your cluster using the following command:

apt-get install openmosixview

Assuming you've successfully installed ClusterKnoppix, your ready to start using it - which also happens to be the topic of the next section:  Using ClusterKnoppix

It allows a cluster of computers to behave like one big multi-processor computer. For example, if you run 2 processes on a single machine, each process will only get 50% of the CPU time. However, if you run both these processes over a 2 node cluster, each process will get 100% CPU time since there are two processors available. In essence, this behavior is very similar to SMP (Symmetric Multi-Processor) systems.

Diving Deeper

What openMosix does is balance the processing load over the systems in the cluster, taking into account the speed of the systems and the load they already have. Note however, that it does not parallelize the processing. Each individual process only runs on one computer at a time.

To quote the openMosix website example :

This simultaneous processing has a lot of uses, as there are many tasks that adapt extremely well to being used on a cluster. In the later sections, we'll show you some practical and fun uses for an openMosix based GNU/Linux cluster. Next: Building An openMosix Cluster