Personal tools
You are here: Home Virgo Data Analysis VDAS vdas_20111117
Navigation
Log in


Forgot your username or password?
 
Document Actions

vdas_20111117

by Marie Anne Bizouard last modified 2011-11-17 21:08

Agenda:

1. Announcements:

2. Current issues:
    • GE and data access in Lyon (MAB/Didier)
    • Cascina circular buffer size (Nicolas)
    • Data transfer (Livio/MAB)
    • VDB developments (Gary/MAB)
    • CNAF users' disk increase (+40TB) for 2012 (MAB)

3. Software Engineering Run 1 preparation: projects + organisation Software Engineering Run 1 (Nicolas)

4. Special topic:

  • VDB short and long term roadmap : postponed to a next meeting (Gary could not make it)
  • CC use for advanced detectors data analysis (MAB)

5. AOB:


Attendees:

Nicolas L., Salvatore V., Cristiano P., Livio S., MAB (minutes), Elena C., NIKHEF, Gegerly D., Chris V.D.B., Florent R., Didier V., Stefano C.


Minutes:

  • GE and data access at Lyon: MAB explains Didier had problems recently to access VSR4 rawdata. The reason is that a large fraction of the files were not staged on disk. Solution: pre-staging files before a large production. Didier had some scripts to do this (just open file without reading too much data to avoid server overload).
--> Action item for MAB: get Didier's scripts to do pre-staging on demand in the future.

  • Cascina circular buffer size: Nicolas explains that the rawdata flow has been reduced by a factor ~ 2 (detection bench PD readout switched off). The circular buffer size is now ~100 days long. Commissioning team happy with that. Want to keep VA4 rawdata (locked & unlocked). Use of dataDisplay with data @ Bologna is also OK for them. Re-discuss situation in a few months, especially when EIB-SAS commissionning starts. Nicolas also mentions the pb to do huge processing on the computing farm because of file server @ Cascina. Elena would like this to be fixed. Stefano C explains that there can't be an easy "universal configuration" that would fullfil all users' requests. MAB reminds that offline reprocessing must be done at CCs. Cascina's priority remains the "online" Elena mentions NOEMI that might have problem in the future to run with many channels including LIGO's ones. MAB agrees that NOEMI must be able to run. Will need to address this problem when we discuss NOEMI plans for advanced detectors.
  •  Data transfer to CCs: Livio informs us that VA4 rawdata transfer to Bologna is complete. Files in 100 folder have been moved into 1000, 1001, 1002, 1003 and 1004 folders. Lyon transfer is not finished. Once it's done, files with GPS times higher than 100100000 that are in 1000 folder will have to be transfered in 1001 and 1002 folders.
    Still to be done: VSR4 RDS files transfered to CCs. Livio asked where are the file and final destination.
Action item for MAB: files with GPS times higher than 100100000 that are in 1000 folder will have to be transfered in 1001 and 1002 folders at Lyon + .ffl file generated + check no files are missing.

Action item for MAB: send an email to Livio of the VSR4 RDS location and final destination paths

  • CNAF users' disk increase: we agree (CW group and CNAF) that 40 TB of the 385 TB virgo_gpfs4 disk will be reserved for users. No tape migration is needed. A quota will be installed to prevent pb with data access.
Action item for MAB: update the document "computing needs for 2012"

3. ER1 preparation: Nicolas explains that Gaussian colored noise will be simulated. No glitch. No auxiliary channels. Segments will be produced: locked and unlocked time. DQ generated randomly: 1% deadtime. Signal injection: blind and unblind. Injections done like Hardware injections (generation of a scheduled injections file + preparation of the waveforms). What still needs to be done: define signals, and rate. Need to know which pipeline is going to run. Pulsar: Virgo group has shown limited interest, but will discuss this with LIGO colleagues. Want to run NOEMI. Will help Nicolas to prepare pulsar injections. Nicolas has contacted NIKHEF group to participate to this ERs.

NIKHEF/Chris: interest to participate in data production (ET MDC experience).
 
Nicolas: Trento has also shown interests. Will continue to work with people on this. A priori signal generation should be ready in december.

MAB: data transfer to CC for ER1? We will receive LIGO h(t) frames. Do we want to keep them?
Nicolas: yes. we should keep these h(t).
MAB: will use Livio tools to do these transfers.

Elena/Cristiano: want to run NOEMI on LIGO data: get RDS or not? No RDS produced during ER1. can run @ CIT alos. Elena mentions this is still under evaluation. Running is fine but then results must be transfered in mysql database at Cascina. Security issues.

We agree that NOEMI for advanced LIGO/Virgo requires a full discussion during a next VDAS call. MAB asks for long term plans.

4. Special topic: advanced Virgo computing model
MAB explains that there are few services that require upgrade and or maintenance: among them there is VDB and data transfer tools. Gary is now in charge of VDB. He is proposing to write a document that explains the long term strategy about developments for VDB. The other service that will require an upgrade is data transfer. Need an homogenuous solution between Bologna and Lyon. Maintenance is difficult. Livio's tools require upgrade anyway as bbftppro is no more supported by CNAF. A development based on gridftp has been done by Cristiano, Alberto, Leone and Giancarlo. Tests have been positive. Relation with VDB for file path catalogue. Could be a good solution, but before going further, we need to understand what will be the future of users at CCs. For the moment, both standard way and GRID enabling tools are used to run jobs, at least in Bologna. Cristinao mentions that at Bologna both GRID and non GRID access to files transfered through gridftp ias provided. Stefano C. mentions this is just a workaround solution not satisfatory for CNAF. In Lyon we got the strong recommendation not to use GRID for job submission in the past (no support provided), because they were facing too many problems with GRID for LHC. Situation seems more satisfactory. Need to understand if this is the right moment to switch to GRID. MAB will contact Lyon to start the discussions.
Didier volunteers to follow this for Lyon. Cristiano, Alberto want to join this effort. Antonella mentions someone at EGO will also follow up these discussions with Lyon and Bologna.
Didier asks if LDR is based on gridftp? Cristiano: yes. MAB: but files are still read in a traditional way thanks to layer built on top of gridftp. LDR strategy is OK when you have the control on the storage system which is not the case of Virgo at CCs. We need to follow CNAF and Lyon infrastructure.

--> Action item for MAB + people volunteering (Didier, Cristiano, Alberto, EGO, ...): start defining our computing model for advanced Virgo. List of services we need to upgrade.

End of the call!