Agenda

• Current issues:
• Data transfer/data access at CNAF and CCIN2P3
• data stored in CC bookkeeping document
• update on VSR2/VSR3 pre-staging at CNAF
• Any other issues? Lyon data access
  
• Software situation
  
• LIGO/Virgo software management recent issues (pss, Fr, ...): status of integration
  

• Specific topics:
  
• S6e/VSR4 run preparation: list of software/tasks required to be ready
  
• CMT/autoconf interoperability: present situation/problems, how to improve the situation
  
• Adv Detector software developments: workflow
  

Minutes

Attendance: Alberto Colla, Nicolas Leroy, Florent Robinet, Benoit Mours, Franco Carbognani, Livio Salconi, Didier Verkindt

• Current issues:
• Data transfer/data access at CNAF and CCIN2P3
• data stored in CC bookkeeping document

MAB: Virgo data storage at CNAF has changed recently: data are now automatically stored on tapes and the most recently accessed data are on disk. The disk capacity is 384TB. When the disk is 90% full, the less recently accessed data are deleted from disk.
During the move CASTOR --> GEMSS, data path have been reorganized such that the path names follow a certain logic and the same logic applied at CCIN2P3. The document explains the logic and provides other metadata such that data set volumes. Future data transfers should follow conventions described in the document.
Livio: will check data transfer tool configuration for the next run.
Benoit: is there a convention for h(t) data streams?
MAB: yes. This is described in the document.

• update on VSR2/VSR3 pre-staging at CNAF

MAB: VSR1/2/3 pre-staging done by CNAF is not finished because we reached the disk is full at 90%
gpfs_virgo3 disk space (block-size=1000): 15.1811 / 32.0058 48%
gpfs_virgo4 disk space (block-size=1000): 345.229 / 384.07 90%
/storage/gpfs_virgo4/virgo/virgoD0T1/ size: 344.516

Total volumes of the requested dataset:
VSR1: 76 TB
VSR2: 166 TB
VSR3:  68 TB
Total VSR1+VSR2+VSR3: 310 TB

VA3:   70 TB is using too much space. VA3 files must be deleted.

Alberto: last phone call with Karen : CNAF will delete VA3 files. Virgo could do it, but CNAF prefers to do it itself. There are also other old dataset occupying some 10s of TB that could be deleted.

To be deleted:
5.657e+00  /storage/gpfs_virgo4/virgo/virgoD0T1/Run/C6
9.493e+00  /storage/gpfs_virgo4/virgo/virgoD0T1/Run/VA1
7.486e+01  /storage/gpfs_virgo4/virgo/virgoD0T1/Run/VA3

--> Action item for Alberto

MAB: we should also obtain that newer VA3 files are automatically deleted when they come.
Alberto: this is something more complicated for CNAF. Would require more modification for them

MAB: what will be the VSR4 data stream size:
Benoit/Livio: it's around 14 MB/s right now. Should be reduced to ~ 11 MB/s like during VSR3

• Any other issues? Lyon data access

Since more than few months, users have problem to access data at Lyon. The staging is too long and pipelines failed at reading data (timeout reached). Lyon has been contacted few times. Answer is "we had a problem it should work better". The situation does not seem to improve over time. Users are not happy with the situation.

Franco: dataSender at Lyon is also not working because of this staging issues. DataSender from CNAF is now the privileged solution. Will try to encourage Virgo users to use it and then we can see how the Lyon situation has evolved.

Action item for MAB: perform tests and contact Lyon to see how to improve the data access situation which has degraded since few months/1 year.

• Software situation
  
• LIGO/Virgo software management recent issues (pss, Fr, ...): status of integration
  

Franco: pss is managed with autoconf. LIGO keeps using an updated version of pss. Problem fixed. But there is a new package pss_Fr that still requires an installation on LSC cluster. Work with Adam about this. The situation about Fr is similar. Fr is manageable with autoconf. New version v8r14 released last week.

• Specific topics:
  
• S6e/VSR4 run preparation: list of software/tasks required to be ready

Franco: would like to align all (basic) software before the run starts: Cm, Fr, Frv, Fd ....
MAB: which version of Fr is going to be used? v8r14?
Franco: v8r14 is not yet used in the online.
Benoit: no real reason to use v8r14. Changes concern LIGO/GEO.Alignment (especially Fd) is a priori transparent. Will try to do it within 2 weeks.
MAB: would be good to switch anyway to this version before the run starts. Detchar tools need to be aligned also.

MAB: concerning online/detchar computing and disk space need, is there any need? We have lots of free olnodes.

Florent: Omega is fine (machines + disk space)
Didier: for DQ production, things are OK
Nicolas: for KW triggers: things are OK
Alberto: NOEMI needs 500 GB. Negotiation with Antonella. + final decision of what is run online at Cascina and offline at CNAF.
Didier: bufferv22 has something like 700 GB free.

Action item for Alberto: update VDAS once he gets more info from Antonella.

• CMT/autoconf interoperability: present situation/problems, how to improve the situation

MAB: Patrick Brady has clearly expressed that Virgo and LIGO have both the right to use the tools they are used and/or want to use as long as basic management rules are followed both for software configuration, archive system and bug tracking system.

Franco: concerning bug tracking, Virgo is using SPR because I thought this was simple enough for us, but could be acceptable to move towards a more modern tools. Will follow LIGO discussions on the matter.

Software management and installation:
Franco explains the work he has done (see slides) with the pss software to let LIGO team to install this package on LSC clusters and permit a use within lalapps.
- CMT fragment to permit to use local installation of extremal packages: Fr, FFTW amd root
- generate a file that tells autoconf where to find library compiled by CMT.
Modulo few difficulties this scheme should work. But it supposes that CMT is installed on LSC clusters and used. That does not correspond to the reality. LIGO wants now rpms of Virgo packages.

MAB: that would maybe solve the problem of stable release on LSC clusters but that does not provide any easy solution for users who wants to participate to Virgo software.
Franco: it seems that LIGO wants to be able to use scripts they developed on top of autoconf.
MAB: autoconf is one of the worse tools one can imagine.
Franco: autoconf is used everywhere.
Benoit: autoconf can be used for Fr but this is not as convenient as CMT. Why LIGO does not want to use CMT? Is CMT still used by HEP?

MAB: if Virgo has to provide rpms we will have to do all the work for all supported LIGO plateforms. It means additional work for Virgo. And that does not provide a solution for software developers. If we want LSC develops Virgo packages we will have to move all Virgo packages to autoconf which would mean a real backward regression for us. We can't go in that direction.
 
Action item for MAB: investigate other experiments (HEP and others) present solution for software management and distribution.

MAB: we need to find a solution that encourage cross-development. Software installation is 1 part of the problem. This is critical to think now as new solutions as we have to decide how Virgo software is going to evolve for Adv Virgo. We have online software but we also favor/encourage LSC colleagues to develop Virgo software. Why Virgo users have been able to do development inside LSC

Benoit: mbta has not been a real success in that respect.

Didier: Virgo software development is more or less condemned. All LSC pipelines are developed around condor, etc ...We are obliged to use LSC software.

MAB: this is a very pessimistic opinion that I don't share. Virgo DA software is smaller than LSC software but this is a long term choice that Virgo needs to take now. If we give up, that should be said, but there will be consequences. And I don't see groups like the pulsar group abandoning their developments. We have a few projects that we want to promote. Let's try to do it. And we need to provide opportunities to new Virgo comers. In the meantime we need to all Virgo users to do physics and not only battle with tools related issues.

Benoit: what's your concrete proposal?

MAB: I would like first to make a kind of survey of the existing software that each DA group wants to maintain/develop for Adv Virgo. The situation is rather inhomogeneous. If there is plan for new Virgo developments, if groups want to go in the direction of doing developments only within LSC framework and so on. Then we would have concrete examples to see how we can have a friendly Virgo/LSC environment any LSC or Virgo user. Will try to explain this during the Virgo week.

Meeting adjourned because we reached 2 hours!
Next meeting in 2 weeks. 
 

• Adv Detector software developments: workflow