Agenda

• Pegasus as a job submission framework for Virgo. Slides: Pegasus as possible job scheduler for Virgo
  

Participants

Cristiano, Mab, Pia, Florent, Luca, Elena, EGO control room, John,

Minutes

Gergely presents Pegasus.

Pegasus can be interesting for us as a job submission framework.
It is able to handle jobs interconnections. Is very feasible. Compatible with
different execution environments. Slide 4 shows the steps to submit workflowes.
Jobs and executable are different, each job should have at least 1 executable,
clearly.

Slide 5: an example with 2 parents and 2 children.

Slide 6: example of a script. It is python here but can be other, like Java,Perl..

The sintax is simple. Gergely reads the comments in the green lines
(import, create DAX,define input and output files, define the executables...).

Slide 7: define the jobs

Slide 8: define the dependencies among the jobs.
Note --sites local which means the submission site is local.

Sites can be: local,creamce,condor (Slide 9).

Slide 10: Pegasus can be even installed on a laptop, but then clearly
it has to be on while the job runs...so use a fixed computer..

Slide 11: Final considerations.
Pegasus can be interfaced almost with all...
Note points 2 and 3: to be more symmetric with LIGO, by running LIGO pipelines on Virgo clusters,

this seems to be a good solution.
Note that DT is decoupled, Pegasus can use any catalog.
The support is good. the error report is not so clear.
Need to clarify the long term support to CreamCE.

Cristiano: it is important that it can interface different system. The fact that
Pegasus doesn't have a WEB interface, like Dirac,
might discourage the average user.
Question: do you have to specify the physical or the logical file name,
through a catalog ?

Gergely: yes, it might not be so attractive for the average user.
logical names can be given as an argument,
then the job translates the name using the catalog.

Cristiano: and what about authentication ?

Gergely: it depends on remote configuration. To submit to a CreamCE you need a
voms-proxy.

Cristiano: what about the long term maintenance of CreamCE ?

Gergely: in principle you can use it without CreamCE, by using the local batch resources behind it,

but clearly it is not so convenient.

Pia: (if )CreamCE will not be supported is because there is some alternative ?

Gergely: In principle creamCE will be supported, but we need a written statement from the Pegasus team

about it.

Florent: I have have seen a simpler example of submission framework.
This is complex, how simple it is to submit a typical job ?

Gergely: yes, will prepare an example for next Monday.

Florents: what about job monitoring ?

Gergely: very easy. Run pegasus status and will have a summary.

It is not a colored web page, but it's a good summary.
One step further compared to what you have in grid.

I think that pegasus is the most straigthforward solution for us,
based on the experience of the porting of TIGER CBC pipeline. But if Dirac can
do the same it can be a good solution. Florent can you give a presentation ?

Florent: yes, next Monday I can give an update on what done so far.
We need to make a decision.

Gergely: yes, need to start comparing functionalities.

MAB: is nice to have presentations but we will need to define tests.
So you can define a list of tests. Test different configurations of different
pipelines.

Gergely: yes.
Another item: I have investigated file catalogs available in next years.
Lfc will be retired, but the migration will be solved. So, we are safe from
this point of view.

Pia: what about compatibility with LIGO pipelins using Dirac ?
Florent: Pegasus is more convenient for this, it is possible with Dirac but
a middleware is needed. But we can't say that Pegasus is all what LSC has adopted.
There are many LSC pipelines, burst detchar, not using Pegasus.

Next week: example by Gergely on a simpler submission framework. Presentation
on Dirac by Florent.