 |
Environment from the Molecular Level A NERC eScience testbed project |
 |
Computational Grids
Introduction
Traditionally escience is best known for work on grid computing, and we perceive that the eMinerals project is better known for its work in this area. However, we note from the outset that computational grids cannot be independent of data and collaborative grids; thus we have put a commensurate effort into developing and integrating our data and collaborative grids into our compute grid infrastructure.
Basic approach
Our approach in developing a compute grid for the eMinerals project has been to use core middleware tools such as Globus and Condor. We have successfully used these tools to link together a number of shared compute resources, including clusters, parallel computers and Condor pools.
Access to our compute grid
Being based on Globus, user access to the resources of our compute grid is handled by the use of X.509 digital certificates; each of our science users has their own certificate. And since we are an escience testbed project, we aim to not allow anyone to cut corners. Thus we forbid scientists to logon to resources to submit their jobs direction. In fact this discipline has not been onerous, and our grid team has risen to the challenge of providing usable job submission tools.
Our main job submission tool is called "my_condor_submit" (MCS), for which the users provide a simple Condor-like script containing relevant job information. MCS is now quite advanced, providing integration with our data grid (including automatic metadata collection as well as data management), and metascheduling between the resources in our compute grid and to external resources to which we have access (including the National Grid Service).
The main drawback with MCS is that users need to have Condor and Globus installed on the computers from which they submit their jobs. Some users are able to install these tools on their desktops, but not all (particularly users of Windows PCs). Thus we are developing two tools to enable jobs to be submitted from any user's desktop computer. One is a web services wrapping of MCS, called RMCS, and the other is a Gridsam tool designed to have the same functionality of MCS.
Integration with our data grid
Our data grid work is based around the use of the Storage Resource Broker (SRB). Our approach now is to use the SRB to provide a complete archive of all files associated with a single job. In brief, the user first uploads all input files to the SRB. The job then downloads the files, runs the simulation, and uploads all the generated output files to the SRB. The user and any collaborators are then able to view the output files directly from the SRB. To aid in this work, we have developed a number of tools (including TobysSRB and ccViz) to help deliver the information content of output files to the user and collaborators.
The integrated compute and data grids within our project are collectively managed as the "eMinerals minigrid".
General references
Representative papers that describe the eMinerals work on computer grids, which can be downloaded as pdf files, are:
