Condor: A Tool For Creating a Desktop Grid

Condor was developed as a means to utilise idle computing time on desktop machines. Two examples used in the eMinerals minigrid are groups of teaching computers and the desktop computers in a research group. A Condor setup has one machine acting as the master node, and all others acting as clients, thereby defining a Condor pool. The master node handles control of jobs submitted to the condor pool, which includes the tasks of job scheduling and resource brokering, job monitoring, and data transfer. Any number of the machines within a pool can be configured to allow job submission. In a pool composed of teaching computers, it may be most sensible to have only one submit node, but in a pool based on owned desktop machines it is likely to be more transparent to users to allow all machines to be capable of submitting jobs.

Condor has a number of key grid facilities built in. These include:

• Ability to handle a wide range of computing platforms and operating systems. The Condor resource brokering facilities can be used to specify a particular platform that a job will run on if executable images are only available for a limited set of platforms or operating systems.
• Fault tolerance, namely that if one machine fails the job will be migrated onto another member of the Condor pool. In certain system configurations Condor allows for checkpointing of jobs, so that with machine failure the job will restart close to the point it had reached at failure. At the time of writing, it is not possible to use checkpointing in some critical parts of the eMinerals minigrid, namely for Windows computers and for the use of the Intel Fortran compiler on Linux machines.
• Respect for the owners of contributed resources. For example, jobs on the Condor pool can run with low priority so that their use as a desktop machine is not compromised. It is also possible to establish rules such that Condor jobs will only run outside of office hours, or after a fixed period of inactivity.
• Condor jobs and data generated by Condor jobs are secure from the view of the desktop machine on which they are running. Similarly, the desktop machine is secure against potentially hazardous commands run through the Condor system. For example, it is not possible to run a Condor command that manipulates (views, deletes, alters) data on the desktop.
• Condor only requires that the client software be installed on the desktop, and from that point onwards nothing more needs to be done on the client computers. The important point in this respect is that it is not necessary to create user accounts on any of the desktop computers: all jobs handled by Condor are run under a generic account.
• Condor is relatively easy to install and administer, and there is now a sufficiently large user base from which help can be obtained.

It may be questioned whether, by some definitions, Condor is a true Grid tool, in that it operates on locally-controlled resources. It is in fact possible to join separate Condor pools together in a process known as “flocking” to form what would be called a true grid. Unfortunately this process does not yet handle well the problems caused by firewalls or the use of private IP addresses, and hence has not been used extensively.

General references

Papers that describe the eMinerals science areas are:

---