This page is maintained by Karolina Sarnowska-Upton from the University of Virginia.


UNICORE 6 on FutureGrid
User Manual

Author: Karolina Sarnowska-Upton, University of Virginia
Version: 1.0
Today's Date: 2010-12-21
Last Revision: 2011-11-28


UNICORE 6 has been deployed at various FutureGrid sites. This tutorial explains how to connect to the existing FutureGrid UNICORE endpoints from other grid middleware platforms or a UNICORE Commandline Client (UCC) as well as how runs jobs on UNICORE sites and how to deploy a new UNICORE grid.

What is UNICORE?

UNICORE (Uniform Interface to Computing Resources) is a Grid middleware system. Listed below are the key principles of the UNICORE design. More information about UNICORE can be found at http://www.unicore.eu.

  • Open source under BSD license.

  • Standards-based, conforming to the latest standards from the Open Grid Forum (OGF), W3C, OASIS, and IETF, in particular the Open Grid Services Architecture (OGSA) and the Web Services Resource Framework (WS-RF 1.2).
  • Open and extensible realized with a modern Service- Oriented Architecture (SOA), which allows easily replacement of particular components with others.
  • Interoperable with other Grid technologies to enable a coupling of Grid infrastructures or the users needs
  • Seamless, secure, and intuitive following a vertical, end-to-end approach and offering components at all levels of a modern Grid architecture from intuitive user interfaces down to the resource level. Like previous versions UNICORE 6 seamlessly integrates in existing environments.
  • Mature security mechanisms adequate for the use in supercomputing environments and Grid infrastructures. X.509 certificates form the basis for authentication and authorization, enhanced with a support for proxy certificates and virtual organizations (VO) based access control.
  • Workflow support tightly integrated into the stack while being extensible in order to use different workflow languages and engines for domain-specific usage.
  • Application integration mechanisms on the client, services and resource level for a tight integration of various types of applications from the scientific and industrial domain.
  • Different clients serving the needs of various scientific communities, e.g. graphical clients to define complex workflows, command line tool, web based access.
  • Quick and simple to install and configure to address requirements from operational teams and to lower the barrier of adopting Grid technologies. Similar the configuration of the various services and components is easy to handle.
  • Various operating and batch systems are supported on all layers, i.e. clients, services and systems; Windows, MacOS, Linux, and Unix systems as well as different batch systems are supported such as LoadLeveler, Torque, SLURM, LSF, OpenCCS, etc.
  • Implemented in Java to achieve platform independence.

Connecting to the UNICORE BES Endpoints From Other Grid Middleware Clients

Two UNICORE BES endpoints have been deployed on FutureGrid for interoperability testing. One endpoint is located on Sierra and the other is located on India. This section contains the information needed for other grid middleware platforms to connect to the UNICORE BES endpoints.

India Endpoint Info <currently unavailable>

  • Security: configured for username/password authentication; Email for username token and CA cert.
  • OS: Red Hat Enterprise Linux Server release 5.5
  • Arch: x86_64
  • Cores: 8 (Jobs submitted directly to machine - i.e. not through PBS queue)

Sierra Endpoint Info

  • Security: configured for X-509 based mutual client authentication; Email with X-509 cert and for CA cert.
  • OS: Red Hat Enterprise Linux Server release 5.5
  • Arch: x86_64
  • Cores: 320 (Jobs submitted to PBS queue)

Connecting to the UNICORE BES Endpoints Using a UNICORE Commandline Client

A UNICORE client can be used to connect to the FutureGrid UNICORE6 endpoints. This section describes how to install a UCC ( UNICORE Commandline Client), configure it to connect to a FutureGrid U6 endpoint via X-509 based mutual client authentication, and then submit jobs via BES.

Installing the UNICORE6 Commandline Client (UCC)

Acquire Client Bundle

  1. Navigate to the UNICORE website: http://www.unicore.eu/
  2. Select “Download” link in left hand tool bar
  3. Under Clients section, select “Download” link for Commandline Client
  4. Click on folder for desired version  (i.e. 6.4.1)
  5. Click on desired distribution bundle to download  (i.e. ucc-6.4.1-all.tar.gz)

Unpack UCC

  • Unpack files from downloaded distribution bundle  (i.e. tar -xvzf ucc-6.4.1-all.tar.gz)
  • On completion, there should be a directory containing the UNICORE6 commandline client  (i.e. ucc-6.4.1)
  • You can add the bin directory to your path for easier client execution
 export PATH=$PATH:<UCC_HOME>/bin,  where UCC_HOME is the directory you installed UCC

Examine UCC Files

Directory Structure

  • bin – contains executable “ucc”
  • certs
  • conf – contains preferences file to be configured with security and registry settings
  • doc
  • extras
  • lib
  • samples

Run Commandline



Run ucc to get list and description of available commands


Usage: ucc <command> [OPTIONS] <args>
The following commands are available:
 ls                     - list a storage
 rm                    - remove a remote file or directory
 copy-file-status      - check status of a copy-file
 get-file              - get remote files
 find                 - find files on storages
 resolve                - resolve remote location
 mkdir                  - create a directory remotely
 copy-file             - copy remote files
 put-file               - puts a local file to a remote server

 create-storage       - create a storage service instance
 connect               - connect to UNICORE
 list-storages         - list the available remote storages
 list-applications    - lists applications on target systems
 list-jobs             - list your jobs
 list-sites             - list remote sites
 system-info            - checks the availability of services

 run                    - run a job through UNICORE 6
 get-status             - get job status
 abort-job             - abort a job
 batch                  - run ucc on a set of files
 get-output            - get output files

 bes-list-att           - provides information about U6 BES Interface
 bes-terminate-job      - terminate bes activity
 bes-submit-job         - run a job through UNICORE 6 BES Interface
 bes-list-jobs         - list jobs running on BES.
 bes-job-status         - get bes activity status

 shell                  - starts an interactive UCC session
 issue-delegation      - allows to issue a trust delegation assertion
 connect-to-testgrid   - get credentials for the public testgrid
 wsrf                   - perform a WSRF operation
 cip-query              - query a CIS Infoprovider at a UNICORE site
 run-groovy            - run a Groovy script

 workflow-trace        - trace info on a workflow in Chemomentum
 workflow-control      - offers workflow control functions
 workflow-submit       - submit a workflow
 workflow-info         - lists info on workflows.
 broker-run             - submit work assignment to service orchestrator

Enter 'ucc <command> -h' for help on a particular

Installation Conclusion

  • At this point, the UNICORE Commandline Client has been installed
  • However, the client is currently not connected to any UNICORE sites
  • The next section will explain how to configure the client so that the client will connect to one of the FutureGrid U6 endpoints

Configuring Client to Connect to FutureGrid U6 Endpoints

Configuration Overview

  • To connect to a FutureGrid U6 endpoint, you need to
    • setup security information so that your client will trust the FutureGrid U6 endpoint and vice-versa
    • specify the connection address that the client should use (aka registry address)
  • This configuration process consists of
    • setting up keystore and truststore files with security info
    • specifying a registry address for the FutureGrid endpoint
  • This information is stored in a “preferences” file (starter at $UCC_HOME/conf/preferences)

Setting Up Security in UNICORE

To setup security, you will need to inform the UNICORE software of your identity and who you trust via keystore and truststore files:

  • Keystore - a file from which UNICORE software reads your identity, i.e. your private key and your certificate. As your private key is very sensitive, the keystore is encrypted and you will need a password to "unlock” it before usage.
  • Truststore - a file from which UNICORE software reads certificates of the Certificate Authorities you trust. It is not as sensitive as a keystore, but it is also encrypted.

For an overview of the security mechanism found in the UNICORE grid middleware, please consult the Users' UNICORE Security Guide | UCC version
This guide also discusses common security configuration problems and details how to create keystores/truststores

Setting Up a


Assumption: you want to use a preexisting X.509 certificate for security validation

  • If your key and certificate are in a keystore (in PKCS12 or JKS format), you can directly use this keystore
  • If you have PEM files, you will need to wrap your key and certificate files into a PKCS12 keystore:
openssl pkcs12 -export -in myCert.pem -inkey myKey.pem -out keystore.p12 -name mykey 

Creating a Truststore

Assumption: you have certificates for CAs that you trust and want to put into a truststore file. This should include:

CA cert for CA that issued your certificate
CA cert for CA that issued FutureGrid U6 endpoint certs (Email uvacse@virginia.edu for cert)

To create a truststore with keytool

keytool -import -keystore truststore.jks -file CAcert.pem -alias 

Repeat command for every CA certificate file (set a unique alias for each certificate)

Acquiring the Registry


The UNICORE Registry server provides information about available services to clients and other services

Registry address for FutureGrid U6 Endpoint on Sierra (as of 08/2011)

Connecting to Endpoints w/o Registries

A Registry URL is expected to be provided in the preferences file. If a BES endpoint is not advertised via a UNICORE Registry, the configuration options can be modified to allow this behavior.

  1. In the preferences file, set the contact-registry to false:
  1. Instead, provide at least one BESFactory URL using the following format.

If the "contact-registry" option is set to false and no OGSA-BES URL is specified, UCC will report an error.

To use an XML endpoint reference (EPR) read from a file for contacting a BESFactory service, the contents of a EPR file must validate against the WS-Addressing’s endpoint reference schema.

Preferences File Modifications

Once you have keystore and truststore files, configure the client preferences file to use this security information


Keystore Settings
  • Set keystore path to the full path to where your keystore file is saved locally
  • Provide the password that “unlocks” your keystore file
  • If you do not want specify the password, you will be asked for it on the commandline
  • Provide keystore’s alias
  • Provide keystore’s storetype (PKCS12 or JKS)
Truststore Settings
  • Set truststore path to the full path to where your truststore file is saved locally
  • Provide the password that “unlocks” your truststore file
Registry Address or BESFactory URL Information
  • Provide registry address/BESFactory URL of FutureGrid U6 endpoint

Example Preferences File



# The address(es) of the registries to contact (space separated list)

# ... other properties can follow

Locating the Preferences


By default, UCC checks for the existence of a file at $USER_HOME/.ucc/preferences and reads default settings from there

Copy your preferences file to this location or specify it’s a location every time you issue a UCC command

  -c $UCC_HOME/conf/preferences

Validate Client


To check access to FutureGrid endpoints, try a command while specifying the path to your preferences file

$UCC_HOME/bin/ucc system-info -v -l -c $UCC_HOME/conf/preferences
$UCC-HOME/bin/ucc list-sites -c $UCC_HOME/conf/

If everything is alright you should see information about the FutureGrid endpoint.



Your UCC has now been configured to access the FutureGrid U6 endpoint via X-509 based mutual client authentication
Consult the next section to learn about getting started and submit jobs

Submitting Jobs to FutureGrid U6 Endpoints



Make sure you have access to some target system

$UCC_HOME/bin/ucc connect -c $UCC_HOME/conf/preferences

List the sites available to you using

  $UCC_HOME/bin/ucc list-sites -c $UCC_HOME/conf/

Run a sample job

  $UCC_HOME/bin/ucc run -v $UCC_HOME/samples/date.u -c $UCC_HOME/conf/preferences

Submit Jobs to


To send a job read from a JSDL file

  • to a site listed in the preferences file:
        $UCC_HOME/bin/ucc bes-submit-job -j hellompi.xml -s bes.3 -v
  • using a BESFactory URL:
  ucc bes-submit-job -j hellompi.xml -s https://example.com/services/BESFactory -v
  • using an endpoint reference file path:
  ucc bes-submit-job -j hellompi.xml -s file:///tmp/bes.xml -v

Other BES Related


Check job status

  ucc bes-job-status jobid.job

Terminate job

ucc bes-terminate-job jobid.job

List user’s jobs on a BESFactory

ucc bes-list-job-s bes.1

List BESFactory properties:

ucc bes-list-att -s bes.1

*Descriptor (.job) file is automatically generated after a successful execution of "bes-submit-job" command

Job Submission Conclusion

You should now be able to submit jobs to a BES service on a FutureGrid U6 endpoint




The information provided in this section has been extracted from the following sources. Each contains much more detail about the various topics discussed here.


Please email uvacse@virginia.edu

Running Jobs on UNICORE Sites

This section provides a general overview of running jobs on UNICORE sites.

First, you can check whether there is an available target system for execution:

 $ucc connect -c <path_to_ucc.preferences> 

You can also list the available target sites:

 $ucc list-sites

To run a job, you can specify the job using the UNICORE job description format.

 $ucc run <job_description_file>

Or you can specify a job to run using a Job Submission Description Language (JSDL) file.

 $ucc run -j <JSDL_file>

For example, a simple job to run the date, date.u, could be described using the UNICORE description format as follows:

# simple job: run Date

 ApplicationName: Date,
 ApplicationVersion: 1.0,

This job can then be run with the command:

$ucc run date.u

Alternately, this job could be described using a JSDL file, date.jsdl, as follows:

 <?xml version="1.0" encoding="UTF-8"?>
<jsdl:JobDefinition xmlns="http://www.example.org/"

                       <jsdl:JobName>Test Job</jsdl:JobName>

                       <jsdl:ApplicationName>Date 1.0</jsdl:ApplicationName>

This JSDL job can then be run with the command:

 $ucc run -j date.jsdl.

Deploying a New UNICORE 6 Grid

This section details how to install and setup a UNICORE 6 grid. This will create a new grid infrastructure separate from the existing deployments mentioned above.


Prerequisites listed in UNICORE 6 README.txt file:

  • Java 5 JRE or SDK (or later). We recommend using SUN Java 6.

You can check what version of java is installed by running:

 $java -version

If the java version is too old or the command is not found, download the latest JRE version. After unpacking, update JAVA_HOME and PATH.

  • Perl 5 (only for the classic TSI)

You will need Perl if you will be running jobs via a queue management system. By default, jobs run directly on the local machine. Extra configuration is needed to setup a TSI and run jobs via a queue management system.

Installing the Core Server Bundle

The UNICORE 6 Core Server Bundle can be downloaded from the UNICORE website. This bundle will provide you with the minimal set of UNICORE components (Gateway, UNICORE/X, TSI, XUUDB) needed to get a UNICORE 6 grid up and running quickly.

From the README.txt:
The basic configuration and installation is as follows
1) edit configure.properties
2) ./configure.py <unicore_user> [<hostname>]
3) run ./install.py to install UNICORE in the selected directory

To start the configuration process, do (as non-root user)
./configure.py <login> [<hostname>]
where <login> is the Unix account that the installation shall use, and the   
optional <hostname> indicates which hostname shall be used. If not given,
a hostname lookup will be performed.

If you want to customize this process, edit the "configure.properties" file,
which contains all the ports and other settings (except security settings 
such as keystore locations).

To finally copy the files into the installation directory, run

More documentation at http://www.unicore.eu/documentation.

Starting/Stopping the UNICORE Servers

The servers are started with the start.sh script and stopped with the stop.sh script. Both of these scripts are located under the UNICORE install directory.

You can confirm that the UNICORE servers are running with the ps command. 
You should see three processes running: XUUDB, Gateway, and UNICOREX. 
  $ps aux| grep unicore