Personal tools
You are here: Home Virgo Data Analysis VDAS Grid (LCG) Data Management
Log in

Forgot your username or password?
Document Actions

Grid (LCG) Data Management

by Alberto Colla last modified 2013-04-11 18:15

Examples how to transfer and retrieve data using LCG tools

1. Legenda:

  • GUID: Grid Unique ID
  • LFC: Logical File Catalogue
  • LFN: Logical File Name
  • SFN: Site File Name

2. Obtaining a proxy, setting environment

  • First of all, initialize the voms proxy, e.g.:
    voms-proxy-init --voms virgo
  • Set catalogue endpoint into LFC_HOST:
    export LFC_HOST=`lcg-infosites --vo virgo lfc`
  • Set VO name into LCG_GFAL_VO. With this variable set, you won't have to type "--vo voname" each time you issue a command:
    export LCG_GFAL_VO=virgo

3. Summary of the catalogue, file and replica management commands:

  • Catalogue management

    • lfc-chmod Change access mode of a LFC file/directory
    • lfc-chown Change owner and group of a LFC file/directory
    • lfc-delcomment Delete the comment associated with a file/directory
    • lfc-getacl Get file/directory access control lists
    • lfc-ln Make a symbolic link to a file/directory
    • lfc-ls List file/directory entries in a directory
      Note: set variable LFC_HOME to your lfc home! (e.g. /grid/virgo)
    • lfc-mkdir Create directory
    • lfc-rename Rename a file/directory
    • lfc-rm Remove a file/directory
    • lfc-setacl Set file/directory access control lists
    • lfc-setcomment Add/replace a comment
    • lfc-entergrpmap Defines a new group entry in the Virtual ID table
    • lfc-enterusrmap Defines a new user entry in Virtual ID table
    • lfc-modifygrpmap Modifies a group entry corresponding to a given virtual gid
    • lfc-modifyusrmap Modifies a user entry corresponding to a given virtual uid
    • lfc-rmgrpmap Suppresses group entry corresponding to a given virtual gid or group name
    • lfc-rmusrmap Suppresses user entry corresponding to a given virtual uid or user name.
  • Replica Management
    • lcg-cp Copies a Grid file to a local destination (download)
    • lcg-cr Copies a file to a SE and registers the file in the catalogue (upload)
    • lcg-del Deletes one file (either one replica or all replicas)
    • lcg-rep Copies a file from one SE to another SE and registers it in the catalogue (replicate)
    • lcg-gt Gets the TURL for a given SURL and transfer protocol
    • lcg-sd Sets file status to \u201dDone\u201d for a given SURL in an SRM\u2019s request
  • File Catalogue Interaction
    • lcg-aa Adds an alias in the catalogue for a given GUID
    • lcg-ra Removes an alias in the catalogue for a given GUID
    • lcg-rf Registers in the catalogue a file residing on an SE
    • lcg-uf Unregisters in the the catalogue a file residing on an SE
    • lcg-la Lists the aliases for a given LFN, GUID or SURL
    • lcg-lg Gets the GUID for a given LFN or SURL
    • lcg-lr Lists the replicas for a given LFN, GUID or SURL
    • lcg-get-checksum check the checksum of file

4. Some examples of the most common commands:

Choose a SE where to register your first file (e.g. CNAF):

lcg-infosites --vo virgo se |grep cnaf

44546443359 165556640  SRM 

Copy a file on the grid and register it on the LFN:

1. Simple mode

lcg-cr -d -l lfn:/grid/virgo/helloDir/helloWorld.txt file:`pwd`/helloWorld.txt 

  • -d indicates the destination SE;
  • -l specifies the logical file name;

2. Without invoking BDII service, that is -b option (BDII retains the low-level information about the storage server), in two steps: 

lcg-cp -b -v -U srmv2 file://`pwd`/helloWorld.txt  srm://  

lcg-rf -v -b -T srmv2 -l /grid/virgo/helloDir/helloWorld2.txt srm://


  • -b: do not invoke BDII
  • -v: verbose
  • -U (-T): remte storage SRM version (typically srmv2)

List the LFC

lfc-ls /grid/virgo/helloDir/ 


Read the SFN of the file you've just uploaded:

lcg-lr lfn:/grid/virgo/helloDir/helloWorld2.txt  


Read file's GUID:

lcg-lg lfn:/grid/virgo/helloDir/helloWorld2.txt


Replicate file (e.g. from Cnaf to Lyon,

lcg-rep -d lfn:/grid/virgo/helloDir/helloWorld.txt  

Replica between two remote storages (third party replica) without invoking BDII and LFC

lcg-cp -v -b -T srmv2 -U srmv2 srm:// srm://


  • -T is the destination srm version

  • -U is the source srm version

Check file checksum:

lcg-get-checksum [--checksum-type MD5|ADLER32|...] srm://file

Download file to local disk: 

1. Simple mode

lcg-cp lfn:/grid/virgo/helloDir/helloWorld3.txt file:/`pwd`/helloAgain.txt   

Note: Local file path must be a full (not relative) path!

2. Without invoking the LFC:

lcg-cp srm:// file:/`pwd`/helloAgain.txt  

3. Withouth invoking LFC and BDII:

lcg-cp -b -U srmv2 file:/`pwd`/helloAgain.txt

Remove a replica: 

lcg-del -s lfn:/grid/virgo/helloDir/helloWorld.txt

Remove all replicas and unregister the file from LFC:

lcg-del -a lfn:/grid/virgo/helloDir/helloWorld.txt

Remove a replica without invoking LFC and BDII

lcg-del -b -l -T srmv2

More advanced examples

Remove the GUID, SURL and LFN of a file in case of problems (e.g. file removed without invoking LFC, in this case lfc-rm gives "file exists" error):

1. lcg-lr <LFN> (list all the replicas of file)

2. lcg-lg <srm://replica> (get the guid)

3. lcg-uf <guid> <surl> to delete entry in the LFC (but not the actual physical files !).

Once the last replica is deleted, the alias is deleted as well. This procedure ensures the catalog consistency. 

Copy a file to two different remote storages and register it in the LFC

Copy file to storage A:

lcg-cp  -b -v  -U srmv1  file:/scratch/virgo/collaalb/fakeffl/raw-1256580844-10.gwfx srm://

Copy file to storage B:
lcg-cp -b -v  -T srmv2 --vo virgo file:/scratch/virgo/collaalb/fakeffl/raw-1256580844-10.gwfx srm://

Register replica A to LFC:

lcg-rf -v -b -T srmv1 -l /grid/virgo/collaalb/prova srm:// 
guid:1dc3ad25-9875-4703-8ad8-fb9d66facee5 (GUID is returned!)

Register from storage B to LFC (Note that you need to specify the GUID already provided, otherwise it fails!)
lcg-rf -b -v -T srmv2 -l /grid/virgo/collaalb/prova  -g 1dc3ad25-9875-4703-8ad8-fb9d66facee5  srm://

: There is no consistency check on the copies: you can register with the same name two different files!!!