Journal article
Adaptive workflow processing and execution in Pegasus
Concurrency and Computation: Practice and Experience, Vol.21(16), pp.1965-1981
2009
Abstract
Workflows are widely used in applications that require coordinated use of computational resources. Workflow definition languages typically abstract over some aspects of the way in which a workflow is to be executed, such as the level of parallelism to be used or the physical resources to be deployed. As a result, a workflow management system has the responsibility of establishing how best to execute a workflow given the available resources. The Pegasus workflow management system compiles abstract workflows into concrete execution plans, and has been widely used in large-scale e-Science applications. This paper describes an extension to Pegasus whereby resource allocation decisions are revised during workflow evaluation, in the light of feedback on the performance of jobs at runtime. The contributions of this paper include: (i) a description of how adaptive processing has been retrofitted to an existing workflow management system; (ii) a scheduling algorithm that allocates resources based on runtime performance; and (iii) an experimental evaluation of the resulting infrastructure using grid middleware over clusters.
Details
- Title
- Adaptive workflow processing and execution in Pegasus
- Authors/Creators
- K. Lee (Author/Creator)N.W. Paton (Author/Creator)R. Sakellariou (Author/Creator)E. Deelman (Author/Creator)A.A.A. Fernandes (Author/Creator)G. Mehta (Author/Creator)
- Publication Details
- Concurrency and Computation: Practice and Experience, Vol.21(16), pp.1965-1981
- Publisher
- John Wiley & Sons
- Identifiers
- 991005543797607891
- Copyright
- © 2009 John Wiley & Sons, Ltd
- Murdoch Affiliation
- Murdoch University
- Language
- English
- Resource Type
- Journal article
UN Sustainable Development Goals (SDGs)
This output has contributed to the advancement of the following goals:
Source: InCites
Metrics
133 File views/ downloads
77 Record Views
InCites Highlights
These are selected metrics from InCites Benchmarking & Analytics tool, related to this output
- Collaboration types
- Domestic collaboration
- International collaboration
- Citation topics
- 4 Electrical Engineering, Electronics & Computer Science
- 4.46 Distributed & Real Time Computing
- 4.46.85 Cloud Resource Scheduling
- Web Of Science research areas
- Computer Science, Software Engineering
- Computer Science, Theory & Methods
- ESI research areas
- Computer Science