1. Announcements:

2. Current issues:

• Data transfer/data storage @ Cascina
• Software Engineering Run 1 [Nicolas]
  

3. Special topic: VDB upgrade:

- VDB status report document prepared by Gary
- Some first comments about Gary's document
- VDB web interface questions/requests on VDQ telecon of Oct 6th 2011
- Some VDB requests on February 25th 2011

- Work organization: list of preamble questions:

1. Define the scope of VDB (segments db vs segments+data transfer bookkeeping db) --> other experimental data
   

2. Clarify the dichotomy between on-line and off-line.

3. Access issues from outside Cascina: firewall issues ...
4. The web interface: clean-up + new functions
5. The VDBtk_segment toolkit

6. Inter-application relationships + link with segdb.
   

7. Documentation.

8. Database structure and back-up procedure.

9. Changes in tables (remove "stable" property for  segments list, add frequency band...)
10. Scimon DQ and Scimon DQ web interface

... Work organization: short term & long term

Mail from Didier:
- Until now, all DQ segments lists of a science run have been uploaded in VDB and in segdb.
- Until now, the data analysis pipelines have mainly used LIGO and Virgo DQ segments from segdb only.
- A full synchronization between VDB and segdb has not yet been set up.
- Most of the science run reprocessed DQ segments have been uploaded "manually" in VDB using
the VDBtk_segment tool and in segdb using the LIGO tools
- SegOnline produces online DQ segments and upload them in segdb through XML files
and in VDB thanks to VDB library APIs
- Uploads in VDB occurs mostly in a few days short time period (no more DQ segments lists
uploaded just for tests)
- Downloads from VDB occurs regularly, using VDBtk_segments, especially for the monitoring of the online DQ flags
- A specific upload is done from time to time when a Scimon DQ segment is produced from control room
- Florent may confirm, but I guess that now most of the tests on DQ segments lists are done using ascii files
- A distinction has been made between online DQ and offline DQ (not managed by the same tables in VDB)
and I must confess that I never understood the reason for this.

Attendees:

Gary, Alberto, Elena, Didier, Stefano, MAB (minutes), Benoit, Livio                                                           

Minutes:

1. ER1 update: Benoit : ER1 start has been hectic, mainly because of LSC software organization. Data from H1 and L1 and V1 are sent to CIT. Data from H1 and L1 are also received at Cascina. MBTA is running at Cascina: 2 x 12 cores (1 GB / core) for H1 and L1. 8 cores for V1. Issues with not enough memory. Need to use the 2cores/8GB memory nodes. Ask permission to "assigned users" to proceed, but these nodes were not use right now. Cores and memory allocation distribution seems not optimal

Action item: rediscuss the memory allocation on each node + discuss more generally the needs for adV and if the current online farm fits with thes needs at a next VDAS call.

2. VDB discussion resume:

- 4 : web interface: agreement that broken links, simplification, and maybe new functionalities are needed. Agreement that the work on the web interface is a lower priority. Gary mentions the interface should be changed in order to make maintenance easier than it is now.

- 5: VDBtk_segments tool: very useful in the past for upload and download + functionalities like segments combination. VDBtk and the web interface should provide more or less the same functionalities. Discussion about segdb functionalities. Python interface that allows search pipelines to query the segdb database. Should VDBtk offer the same functionalities. Same interface? --> raise the question of VDB / segdb synchronisation. Will require some investigation with segdb experts.  

- 6: see above. A priori we would like a large interoperability. We need quickly to understand what is reasonable/feasible.

- 7: documentation entry point sin VDB web interface: should be cleared up. Experience has shown that DQ documentation is always changing thing. Didier is mentioning html pages are enough for our purposes. Gary says that connection database which should be favoured at some point does not support html. Open issues

- 8: database structure: Gary argues that there is little relation between tables which is strange for such a relational database. MAB asks whether query performance is not affected when too many relations between tables are set up. Gary argues that it is not directly connected. Queries performance depends on how the query is formulated. Query performance is an issue.

Gary mentions that there are still tables/fields he does not understand. To avoid to break everything he will make changes in a copy and set up performance tests to be sure nothing is broken. But some tables need to be modified. Segment table should have a unique integer identifier that other tables refer to.

Information that VDQ should provide about advanced Virgo era: order of magnitude about:

# segments nb

# segment list maximal size

# sub-second list

such that Gary can see if the extrapolation of VDB to AdV era will work fine.

Back-up procedure: so far a weekly copy of VDB (1.2 GB) was perfromed. We filled the 255 GB disk. Brute force back up strategy. Proposal: keep 1 copy each N months and a copy of the last weeks, but delete all other copies. Keep the automatic procedure. Stefano asks if tools could not be set up to check that all tables are not corrupted.

 

9: discussion about table changes. Ackownledge the fact that we do not understand all fields/ tables. Need to be caution. Work on a copy of the database and check that nothing is broken with tests.

10: SCIMON DQ: Didier reminds us that the version is the string SCI. BTW version field is a string in VDB. Is that judicious? We agree to suppress online vs offline lists. SCIMON lists could be flagged differently than with a version flag.

version should be reserved to what is made for. MAB reminds that SCIMON lists had always problem when different segments were generated. Overlapping segments and chronology were not working fine.

 

Conclusion:

- write users requriments (VDQ group - Didier)

- write software requirements (description + back-up+validation procedure) - detailed working plan (Gary)

- VDB becomes a recurrent item of VDAS call.

 

Next meeting Feb 9th.