Agenda

1.     Pegasus tutorial , Part III -" A typical CW workflow - mass submission" ('15) (Gergely) Pages: http://wwwcascina.virgo.infn.it/computing/docs/
   
2.     Using Amazon EC2 as possible cheap extension - an Introduction ('10) (Gergely) Slides: Amazon EC2 Spot Instances
   
3.     Summary of Virgo Week's "usage policy" discussion and decisions ('15) Loic
4.     Pipeline use cases in Lyon. Current situation and future improvements ('10) Mab
   
5.     File catalog requirements for the CCs - discussion, IMPORTANT: whoever planning to run various pipelines at the CCs please join !
6.     AOB

Minutes

About file catalog:

We continued the discussion of the mailing list. Marie-Anne prepared some slide about use-case, she could not present it today, but she will present it on one fo the next meeting. Benoit gave a summary of the current situation and we discussed a bit of requirements. Since there is a very limited number of file types (streams) in Virgo we decided that we do not need a tool which says which channel is in which stream, this is anyway quite obvious. The channel database will give some (static) metadata on channel types and that is sufficient. Only the following steps found to be necessary:

a.) There is a need for some mechanisms that ensures that all the file has been copied over to CCs and the ffl list is updated, so there should be no need for manual post-checking. We will try to organize a discussion on this with Livio and others involved in data transfer.

b.) In order to allow LIGO - like datafind queries we will have a simple script which mimics the behaviour of ligo_data_find by simply translating and filtering the ffl file list to a format compatible with the output of ligo_data_find. The very same script can be used also in other CCs using the corresponding ffl list.
     b.1) The script have to be written up, committed to revsision control system and packaged, installed.
     b.2) It has to maintened and checked periodically that it iw working OK on the site.

Volunteers needed for b1, b2 !!! (If no one else Gergely can do b1, but b2 should be done by somebody related to the CC.

c.) We need to improve the channel database, check whether channel descriptions are enough detailed and up-to-date. Check what improvements are presen in CIS and see whether we want to incorporate, further develop them or not.

Amazon Elastic Compute Cloud 4 Virgo

Amazon can offer computing (not storage!) for *half price* compared what we pay now for the CC. Carefule benchmarking has to be performed. Proof-of-principle studies has been performed by Gergely. Amazon cloud instances has been connected to Pegasus condor pool. Monitoring, shutting down and restarting of instances is very easy. Ideal for compute intensive analysis such as CW. One pay only for what is used up, no need to pay for unused resources.

 Pegasus:

Pegasus example 03 mimic the behaviout of a typical CW pipeline (i.e. thousand job processing the same peakmap file in different frequency band) has been written up, documented and tested. A stress test with 1221 job submitted to remote CreamCE resources at CNAF has been successfully performed. No more Pegasus tutorial next weeks, only if there will be some feedback or interest. Lisa is working on Pegasus fo cwBurst pipeline. There was a problem with downloading the example, Gergely fixed that after the telecon.

 Usage policy, virgo staging:

Loic presented the outcome of the discussion during Virgo Week and the new usage policy (about accounts and software installation) and freezing period of VirgoDev area. Now explicit act from VDASC is required, but people are welcome to take part, test the new installation methods and give feedbacks.

No other issues has been brought up

Participants:

Elena, Lisa, Luca, Loic, Benoit, Gergely, Dider

  Highlights of Next week:

- Guide for online software developers - (Loic)
- GPU tutorial 01 - Jump in the middle : The Hough search (Gergely)
- Benchmarking