Simple API for Grid Applications

For other uses, see Saga (disambiguation).

Simple API for Grid Applications

Developer(s)	Center for Computation and Technology at LSU, RADICAL Group at Rutgers University, IN2P3 (France), and Vrije Universiteit (Amsterdam, the Netherlands)
Written in	C++, Python, Java
Platform	Cross-platform
Type	Grid computing, distributed computing library
License	Boost Software License, GPL
Website	saga-project.org

The Simple API for Grid Applications (SAGA) is a family of related standards specified by the Open Grid Forum to define an application programming interface (API) for common distributed computing functionality.

Overview

The SAGA specification for distributed computing originally consisted of a single document, GFD.90, which was released in 2009.

The SAGA API does not strive to replace Globus or similar grid computing middleware systems, and does not target middleware developers, but application developers with no background on grid computing. Such developers typically wish to devote their time to their own goals and minimize the time spent coding infrastructure functionality. The API insulates application developers from middleware.

The specification of services, and the protocols to interact with them, is out of the scope of SAGA. Rather, the API seeks to hide the detail of any service infrastructures that may or may not be used to implement the functionality that the application developer needs. The API aligns, however, with all middleware standards within Open Grid Forum (OGF).

The SAGA API defined a mechanism to specify additional API packages which expand its scope. The SAGA Core API itself defines a number of packages: job management, file management, replica management, remote procedure calls, and streams. SAGA covers the most important and frequently used distributed functionality and is supported and available on every major grid systems - Extreme Science and Engineering Discovery Environment (XSEDE), EGI and FutureGrid. SAGA not only supports a wide range of distributed programming and coordination models but is also easily extensible to support new and emerging middleware.

Standardization

The SAGA API is standardised in the SAGA Working Group the Open Grid Forum. Based on a set of use cases , the SAGA Core API specification defines a set of general API principles (the 'SAGA Look and Feel', and a set of API packages which render commonly used Grid programming patterns (job management, file management and access, replica management etc.) The SAGA Core specification also defines how additional API packages are to be defined, and how they relate to the Core API, and to its 'Look and Feel'. Based on that, a number of API extensions have been defined, and are in various states of the standardisation process.

All SAGA specifications are defined in (a flavor of) IDL, and thus object oriented, but language neutral. Different language bindings exist (Java, C++, Python), but are, at this point, not standardised. Nevertheless, different implementations of these language bindings have a relatively coherent API definition (in particular, the different Java implementations share the same abstract API classes).

The 'Look and Feel' part of the SAGA Core API specification covers the following areas:

• security and session management
• permission management
• asynchronous operations
• monitoring
• asynchronous notifications
• attribute management
• I/O buffer management

Architecture

The SAGA C++/Python architecture: a light-weight runtime system dispatches API calls from the application to the middleware through a set of plug-ins or adaptors.

SAGA is designed as an object oriented interface. It encapsulates related functionality in a set of objects, that are grouped in functional namespaces, which are called packages in SAGA. The SAGA core implementation defines the following packages:

• saga::advert - interface for Advert Service access
• saga::filesystem - interface for file and directory access
• saga::job - interface for job definition, management and control
• saga::namespace - abstract interface (used by advert, filesystem and replica interfaces)
• saga::replica - interface for replica management
• saga::rpc - interface for remote procedure calls client and servers
• saga::sd- interface for service discovery in distributed environments
• saga::stream - interface for data stream client and servers

The overall architecture of SAGA follows the adaptor pattern, a software design pattern which is used for translating one interface into another. In SAGA it translates the calls from the API packages to the interfaces of the underlying middleware. The SAGA run-time system uses late binding to decide at run-time which plug-in (middleware adaptor) to load and bind.

Supported middleware

The following table lists the distributed middleware systems that are currently supported by SAGA. The column labeled Adaptor Suite names the collection (release package) of the (set of) middleware adaptors that provides support for the middleware system.

Middleware System	SAGA Adaptor Suite	SAGA API Namespace
Amazon EC2	saga-adaptors-aws	saga::job
Condor	saga-adaptors-condor	saga::job
Eucalyptus	saga-adaptors-aws	saga::job
Globus GRAM (2 and 5)	saga-adaptors-globus	saga::job
Globus GridFTP	saga-adaptors-globus	saga::filesystem
Globus RLS	saga-adaptors-globus	saga::replica
HDFS	saga-adaptors-hdfs	saga::file
Local file system	part of saga-core	saga::file
Local fork	part of saga-core	saga::job
Nimbus	saga-adaptors-aws	saga::job
PBS (Pro)	saga-adaptors-pbs	saga::job
Platform LSF	saga-adaptors-lsf	saga::job
SQL Advert Service	part of saga-core	saga::advert
SQL Replica Service	part of saga-core	saga::replica
SSHFS	saga-adaptors-ssh	saga::file
SSH	saga-adaptors-ssh	saga::job
TORQUE	saga-adaptors-torque	saga::job

Implementations

Since the SAGA interface definitions are not bound to any specific programming language, several implementations of the SAGA standards exist in different programming languages. Apart from the implementation language, they differ from each other in their completeness in terms of standard coverage, as well as in their support for distributed middleware.

SAGA C++

SAGA C++ was the first complete implementation of the SAGA Core specification, written in C++. Currently the C++ implementation is not under active development.

RADICAL-SAGA(Python)

RADICAL-SAGA is a light-weight Python package that implements parts of the OGF GFD.90 interface specification and provides plug-ins for different distributed middleware systems and services. RADICAL-SAGA implements the most commonly used features of GFD.90 based upon extensive use-case analysis, and focuses on usability and simple deployment in real-world heterogeneous distributed computing environments and application scenarios. RADICAL-SAGA currently implements the job and the file management core APIs as well as the resource management API extension. RADICAL-SAGA provides plug-ins for different distributed middleware systems and services, including support for the PBS, Sun Grid Engine, SSH, SFTP and others. RADICAL-SAGA can be used to develop distributed applications and frameworks that run on distributed cyber-infrastructure including XSEDE, LONI and FutureGrid, other clouds and local clusters.

JavaSAGA

JavaSAGA is a Java implementation of SAGA. This status of JavaSAGA remains uncertain.

import java.util.io.*

int main (int argc, char** argv)
{
    namespace sa    = saga::attributes;
    namespace sja = saga::job::attributes;

    try 
    {
        saga::job::description jd;

        jd.set_attribute (sja::description_executable, "/home/user/hello-mpi");
        jd.set_attribute (sja::description_output, "/home/user/hello.out");
        jd.set_attribute (sja::description_error, "/home/user/hello.err");

        // Declare this as an MPI-style job
        jd.set_attribute (sja::description_spmd_variation, "mpi");

        // Name of the queue we want to use 
        jd.set_attribute (sja::description_queue, "checkpt");
        jd.set_attribute (sja::description_spmd_variation, "mpi");
        // Number of processors to request 
        jd.set_attribute (sja::description_number_of_processes, "32");

        saga::job::service js("gram://my.globus.host/jobmanager-pbs");
        saga::job::job j = js.create_job(jd);

        j.run()
    } 
    catch(saga::exception const & e) 
    {
        std::cerr << "SAGA exception caught: " << e.what() << std::endl;
    }
}

jSAGA

jSAGA is another Java implementation of the SAGA Core specification. jSAGA is currently under active development.

DESHL

The DESHL (DEISA Services for Heterogeneous management Layer), provides functionality for submission and management of computational jobs within DEISA. DESHL is implemented as a set of command-line tools on-top of a SAGA-inspired API implemented in Java. On the back-end, it interfaces with HiLA, a generic grid access client library, which is part of the UNICORE system.

Examples

Job submission

A typical task in a distributed application is to submit a job to a local or remote distributed resource manager. SAGA provides a high-level API called the job package for this. The following two simple examples show how the SAGA job package API can be used to submit a Message Passing Interface (MPI) job to a remote Globus GRAM resource manager.

C++

#include <saga/saga.hpp>

int main (int argc, char** argv)
{
  namespace sa  = saga::attributes;
  namespace sja = saga::job::attributes;

  try 
  {
    saga::job::description jd;

    jd.set_attribute (sja::description_executable, "/home/user/hello-mpi");
    jd.set_attribute (sja::description_output, "/home/user/hello.out");
    jd.set_attribute (sja::description_error, "/home/user/hello.err");

    // Declare this as an MPI-style job
    jd.set_attribute (sja::description_spmd_variation, "mpi");

    // Name of the queue we want to use 
    jd.set_attribute (sja::description_queue, "checkpt");
    jd.set_attribute (sja::description_spmd_variation, "mpi");
    // Number of processors to request 
    jd.set_attribute (sja::description_number_of_processes, "32");

    saga::job::service js("gram://my.globus.host/jobmanager-pbs");
    saga::job::job j = js.create_job(jd);

    j.run()
  } 
  catch(saga::exception const & e) 
  {
    std::cerr << "SAGA exception caught: " << e.what() << std::endl;
  }
}

Python

#!/usr/bin/env python3
import sys
import time
import bliss.saga as saga

def main(jobno: int, session, jobservice) -> None:
    bfast_base_dir = saga.Url("sftp://india.futuregrid.org/N/u/oweidner/software/bfast/")

    try:
        workdir = "%s/tmp/run/%s" % (bfast_base_dir.path, str(int(time.time())))
        basedir = saga.filesystem.Directory(bfast_base_dir, session=session)
        basedir.make_dir(workdir)

        jd = saga.job.Description()
        jd.wall_time_limit   = 5  # wall-time in minutes
        jd.total_cpu_count   = 1
        jd.environment       = {"BFAST_DIR": bfast_base_dir.path}
        jd.working_directory = workdir
        jd.executable        = "$BFAST_DIR/bin/bfast"
        jd.arguments         = ["match", "-A 1",
                                "-r $BFAST_DIR/data/small/reads_5K/reads.10.fastq",
                                "-f $BFAST_DIR/data/small/reference/hg_2122.fa"]

        myjob = js.create_job(jd)
        myjob.run()

        print("Job #%s started with ID '%s' and working directory: '%s'"
          % (jobno, myjob.jobid, workdir))

        myjob.wait()

        print("Job #%s with ID '%s' finished (RC: %s). Output available in: '%s'"
          % (jobno, myjob.jobid, myjob.exitcode, workdir))

        basedir.close()

    except saga.Exception, ex:
        print(f"An error occurred during job execution: {ex}")
        sys.exit(-1)

if __name__ == "__main__":
    execution_host = saga.Url("pbs+ssh://india.futuregrid.org")
    ctx = saga.Context()
    ctx.type = saga.Context.SSH
    ctx.userid  = "oweidner"  # like 'ssh username@host ...'
    ctx.userkey = "/Users/oweidner/.ssh/rsa_work"  # like ssh -i ...'

    session = saga.Session()
    session.contexts.append(ctx)

    js = saga.job.Service(execution_host, session)

    for i in range(0, 4):
        main(i, session, js)

Grants

The work related to the SAGA Project is funded by the following grants: NSF-CHE 1125332 (CDI), NSF-EPS 1003897 (LaSIGMA), NSF-OCI 1007115 (ExTENCI). Previous grants include: NSF-OCI 0710874 (HPCOPS), NIH grant number P20RR016456 and UK EPSRC grant number GR/D0766171/1 via OMII-UK

External links

• SAGA-Bliss - A Python implementation of SAGA
• jSAGA - A Java implementation of SAGA
• SAGA C++ - A C++ implementation of SAGA
• SAGA-GLib - A Vala implementation of SAGA for GLib
• SAGA PROJECT
• POSIX