Personal tools
You are here: Home → Virgo → Data Analysis → Grid (LCG) Data Management
Navigation
Log in


Forgot your username or password?
 
Document Actions

Grid (LCG) Data Management

by Alberto Colla — last modified 2013-11-23 18:25

Examples how to transfer and retrieve data using LCG tools


1. Legenda:


  • GUID: Grid Unique ID
  • LFC: Logical File Catalogue
  • LFN: Logical File Name
  • SFN: Site File Name

2. Obtaining a proxy, setting environment


  • First of all, initialize the voms proxy, e.g.:
    voms-proxy-init --voms virgo
  • Set catalogue endpoint into LFC_HOST:
    export LFC_HOST=`lcg-infosites --vo virgo lfc`
  • Set VO name into LCG_GFAL_VO. With this variable set, you won't have to type "--vo voname" each time you issue a command:
    export LCG_GFAL_VO=virgo


3. Summary of the catalogue, file and replica management commands:


  • Catalogue management

    • lfc-chmod Change access mode of a LFC file/directory
    • lfc-chown Change owner and group of a LFC file/directory
    • lfc-delcomment Delete the comment associated with a file/directory
    • lfc-getacl Get file/directory access control lists
    • lfc-ln Make a symbolic link to a file/directory
    • lfc-ls List file/directory entries in a directory
      Note: set variable LFC_HOME to your lfc home! (e.g. /grid/virgo)
    • lfc-mkdir Create directory
    • lfc-rename Rename a file/directory
    • lfc-rm Remove a file/directory
    • lfc-setacl Set file/directory access control lists
    • lfc-setcomment Add/replace a comment
    • lfc-entergrpmap Defines a new group entry in the Virtual ID table
    • lfc-enterusrmap Defines a new user entry in Virtual ID table
    • lfc-modifygrpmap Modifies a group entry corresponding to a given virtual gid
    • lfc-modifyusrmap Modifies a user entry corresponding to a given virtual uid
    • lfc-rmgrpmap Suppresses group entry corresponding to a given virtual gid or group name
    • lfc-rmusrmap Suppresses user entry corresponding to a given virtual uid or user name.
  • Replica Management
    • lcg-cp Copies a Grid file to a local destination (download)
    • lcg-cr Copies a file to a SE and registers the file in the catalogue (upload)
    • lcg-del Deletes one file (either one replica or all replicas)
    • lcg-rep Copies a file from one SE to another SE and registers it in the catalogue (replicate)
    • lcg-gt Gets the TURL for a given SURL and transfer protocol
    • lcg-sd Sets file status to \u201dDone\u201d for a given SURL in an SRM\u2019s request
  • File Catalogue Interaction
    • lcg-aa Adds an alias in the catalogue for a given GUID
    • lcg-ra Removes an alias in the catalogue for a given GUID
    • lcg-rf Registers in the catalogue a file residing on an SE
    • lcg-uf Unregisters in the the catalogue a file residing on an SE
    • lcg-la Lists the aliases for a given LFN, GUID or SURL
    • lcg-lg Gets the GUID for a given LFN or SURL
    • lcg-lr Lists the replicas for a given LFN, GUID or SURL
    • lcg-get-checksum check the checksum of file

4. Some examples of the most common commands:



Choose a SE where to register your first file (e.g. CNAF): 


lcg-infosites --vo virgo se |grep cnaf

44546443359 165556640  SRM storm-fe-archive.cr.cnaf.infn.it 


Copy a file on the grid and register it on the LFN:


1. Simple mode

lcg-cr -d storm-fe-archive.cr.cnaf.infn.it -l lfn:/grid/virgo/helloDir/helloWorld.txt file:`pwd`/helloWorld.txt

where:
  • -d indicates the destination SE;
  • -l specifies the logical file name;


2. Without invoking BDII service, that is -b option (BDII retains the low-level information about the storage server), in two steps: 

lcg-cp -b -v -U srmv2 file://`pwd`/helloWorld.txt  srm://storm-fe-archive.cr.cnaf.infn.it:8444/srm/managerv2?SFN=/virgo3/scratch/helloWorld2.txt  

lcg-rf -v -b -T srmv2 -l /grid/virgo/helloDir/helloWorld2.txt srm://storm-fe-archive.cr.cnaf.infn.it:8444/srm/managerv2?SFN=/virgo3/scratch/helloWorld2.txt


where:

  • -b: do not invoke BDII
  • -v: verbose
  • -U (-T): remte storage SRM version (typically srmv2)


List the LFC 


lfc-ls /grid/virgo/helloDir/ 

helloWorld.txt
helloWord2.txt


Read the SFN of the file you've just uploaded:

lcg-lr lfn:/grid/virgo/helloDir/helloWorld2.txt  

srm://storm-fe-archive.cr.cnaf.infn.it/virgo3/scratch/helloWorld2.txt


Read file's GUID:

 
lcg-lg lfn:/grid/virgo/helloDir/helloWorld2.txt 
guid:ea4231fc-666e-4b08-90f4-b68c44e0fa91


Replicate file (e.g. from Cnaf to Lyon, ccsrm02.in2p3.fr):

lcg-rep -d ccsrm02.in2p3.fr lfn:/grid/virgo/helloDir/helloWorld.txt


Replica between two remote storages (third party replica) without invoking BDII and LFC

lcg-cp -v -b -T srmv2 -U srmv2 srm://storm-fe-archive.cr.cnaf.infn.it:8444/srm/managerv2?SFN=/virgo3/helloDir/helloWorld2.txt srm://ccsrm02.in2p3.fr:8443/srm/managerv2?SFN=/pnfs/in2p3.fr/data/virgo/tape/helloDir/helloWorld2.txt

where:

  • -T is the destination srm version

  • -U is the source srm version


New! Third party copy from a GSIFTP endpoint to SRM:

lcg-cp -v -b --src-protocols gsiftp -U srmv2
        gsiftp://atlas1.atlas.aei.uni-hannover.de/home/collaalb/test
        srm://storm-fe-archive.cr.cnaf.infn.it:8444/srm/managerv2?SFN=/virgo4/virgo/test

Folders behind GSIFTP servers can be listed with uberftp:

uberftp -dir gsiftp://atlas3.atlas.aei.uni-hannover.de//atlas/user/atlas3/



Check file checksum: 

lcg-get-checksum [--checksum-type MD5|ADLER32|...] srm://file


Download file to local disk: 


1. Simple mode

lcg-cp lfn:/grid/virgo/helloDir/helloWorld3.txt file:/`pwd`/helloAgain.txt

Note: Local file path must be a full (not relative) path!

2. Without invoking the LFC:

lcg-cp srm://storm-fe-archive.cr.cnaf.infn.it/virgo3/scratch/helloWorld2.txt file:/`pwd`/helloAgain.txt


3. Withouth invoking LFC and BDII:

lcg-cp -b -U srmv2 srm:storm-fe-archive.cr.cnaf.infn.it:8444/srm/managerv2?SFN=/virgo3/scratch/helloWorld2.txt file:/`pwd`/helloAgain.txt


Remove a replica:  

lcg-del -s ccsrm2.in2p3.fr lfn:/grid/virgo/helloDir/helloWorld.txt


Remove all replicas and unregister the file from LFC:

lcg-del -a lfn:/grid/virgo/helloDir/helloWorld.txt

Remove a replica without invoking LFC and BDII

lcg-del -b -l -T srmv2 srm:storm-fe-archive.cr.cnaf.infn.it:8444/srm/managerv2?SFN=/virgo3/helloWorld2.txt


More advanced examples

Remove the GUID, SURL and LFN of a file in case of problems (e.g. file removed without invoking LFC, in this case lfc-rm gives "file exists" error):

1. lcg-lr <LFN> (list all the replicas of file)

2. lcg-lg <srm://replica> (get the guid)

3. lcg-uf <guid> <surl> to delete entry in the LFC (but not the actual physical files !).

Once the last replica is deleted, the alias is deleted as well. This procedure ensures the catalog consistency. 


Copy a file to two different remote storages and register it in the LFC


Copy file to storage A: 

lcg-cp  -b -v  -U srmv1  file:/scratch/virgo/collaalb/fakeffl/raw-1256580844-10.gwfx srm://virgo-se01.roma1.infn.it:8443/srm/managerv2?SFN=/dpm/roma1.infn.it/home/virgo/prova

Copy file to storage B: 
lcg-cp -b -v  -T srmv2 --vo virgo file:/scratch/virgo/collaalb/fakeffl/raw-1256580844-10.gwfx srm://storm-fe-virgo.cr.cnaf.infn.it:8444/srm/managerv2?SFN=/virgo3/collaalb/prova

Register replica A to LFC: 

lcg-rf -v -b -T srmv1 -l /grid/virgo/collaalb/prova srm://virgo-se01.roma1.infn.it:8443/srm/managerv2?SFN=/dpm/roma1.infn.it/home/virgo/prova 
guid:1dc3ad25-9875-4703-8ad8-fb9d66facee5   (GUID is returned!)


Register from storage B to LFC (Note that you need to specify the GUID already provided, otherwise it fails!)
lcg-rf -b -v -T srmv2 -l /grid/virgo/collaalb/prova  -g 1dc3ad25-9875-4703-8ad8-fb9d66facee5  srm://storm-fe-virgo.cr.cnaf.infn.it:8444/srm/managerv2?SFN=/virgo3/collaalb/prova

WARNING: There is no consistency check on the copies: you can register with the same name two different files!!!