Adaptive workflow processing and execution in Pegasus

K. Lee; N.W. Paton; R. Sakellariou; E. Deelman; A.A.A. Fernandes; G. Mehta

doi:10.1109/GPC.WORKSHOPS.2008.30

Back

Adaptive workflow processing and execution in Pegasus

Conference paper

Open access

Adaptive workflow processing and execution in Pegasus

K. Lee, N.W. Paton, R. Sakellariou, E. Deelman, A.A.A. Fernandes and G. Mehta

2008 The 3rd International Conference on Grid and Pervasive Computing - Workshops, pp.99-106

IEEE

3rd International Workshop on Workflow Management and Applications in Grid Environments (WaGe08) (Kunming, China, 25/05/2008–28/05/2008)

2008

DOI: https://doi.org/10.1109/GPC.WORKSHOPS.2008.30

Files and links (2)

pdf

adaptive_workflow_processing.pdfDownload View

Author’s Version Open Access

url

Link to Published Version *Subscription may be requiredView

Abstract

Workflows are widely used in applications that require coordinated use of computational resources. Workflow definition languages typically abstract over some aspects of the way in which a workflow is to be executed, such as the level of parallelism to be used or the physical resources to be deployed. As a result, a workflow management system has responsibility for establishing how best to execute a workflow given the available resources. The Pegasus workflow management system compiles abstract workflows into concrete execution plans, and has been widely used in large-scale e-Science applications. This paper describes an extension to Pegasus whereby resource allocation decisions are revised during workflow evaluation, in the light of feedback on the performance of jobs at runtime. The contributions of this paper include: (i) a description of how adaptive processing has been retrofitted to an existing workflow management system; (ii) a scheduling algorithm that allocates resources based on runtime performance; and (Hi) an experimental evaluation of the resulting infrastructure using grid middleware over clusters.

Details

Title: Adaptive workflow processing and execution in Pegasus
Authors/Creators: K. Lee (Author/Creator) - University of Manchester
N.W. Paton (Author/Creator) - University of Manchester
R. Sakellariou (Author/Creator) - University of Manchester
E. Deelman (Author/Creator) - University of Southern California
A.A.A. Fernandes (Author/Creator) - University of Manchester
G. Mehta (Author/Creator) - University of Southern California
Publication Details: 2008 The 3rd International Conference on Grid and Pervasive Computing - Workshops, pp.99-106
Conference: 3rd International Workshop on Workflow Management and Applications in Grid Environments (WaGe08) (Kunming, China, 25/05/2008–28/05/2008)
Publisher: IEEE
Identifiers: 991005543764807891
Murdoch Affiliation: Murdoch University
Language: English
Resource Type: Conference paper

Metrics

455 File views/ downloads

74 Record Views