...
| Code Block |
|---|
guc is globus-url-copy - acronym invented by Levente
srm is a software layer over that - it makes sure transfers complete
There is a –r recursive copy option;
Or else you can do a loop in shell:
set daq = (1380001.daq 1380003.daq 1380004.daq);
foreach f ($daq)
globus-url-copy -p 25 file:/star/bla/bla/$f
gsiftp://pdsfgrid.nersc.gov/bla/bla/bla/$f
end
If you specify a source ending in "XXX/" , it will treat XXX as a
directory and transfer all files in XXX from the XXX directory
Use -p 25 option for having more streams, but isn't 25 overdoing it? You got 1 file
cut into 25 buffer chunks and all this has to be re-assembled on arrival.
I'd try +-2 around 8.
Use different end-point machine: dtn01.nersc.gov to avoid STAR,ATLAS conflicts
there is also carvergrid.nersc.gov gatekeeper I have access too.
pull from carver, you need to load the osg module ,and then you have the globus commands available
globus-job-run works like remote ssh command
Look at this page:
http://rcsg-gsir.imsb-dsgi.nrc-cnrc.gc.ca/globus_tutorial/#running
for globus-job-run, globus-job-submit and globus-job-get-output
(job is a command or script, not necessarily something that goes into
the batch system)
dtn01 is a data transfer node. It has a gsiftp server but no gatekeeper.
carvergrid has both but dtn01 has a 10Gb erthernet and carvergrid only 1Gb.
You can use dtn01 for globus-url-copy and carvergrid for globus-job... type commands.
You can test a gridftp from stargrid node (01,02,03,04) right now to either gsiftp://dtn01.nersc.gov/global/scratch/sd/balewski/... or to
gsiftp://pdsfdtn1.nersc.gov/project/projectdirs/star/...
|
...
| Code Block |
|---|
Ok. Let's do a transfer. 1st check for endpoints (assuming your globus online user name is same as on your laptop, otherwise add yourname@ in front of the cli...) ssh -t cli.globusonline.org endpoint-list should show nothing. But add the -p, ssh -t cli.globusonline.org endpoint-list -p This should produce a public list that includes nersc#dtn & nersc#hpss. (always just use the -t to hide any passwords you might need ...) Activate both those endpoints, ssh -t cli.globusonline.org endpoint-activate nersc#* This will prompt for a pass phrase - use your nim password. Now list again & this should show the endpoints with a time associated with it ( we can increase the time to 11 days as needed). ssh -t cli.globusonline.org endpoint-list Now copy 1 file from HPSS to disk. Here's my example. echo "nersc#hpss/home/p/porter/auk.tar nersc#dtn/project/projectdirs/star/porter/auk.tar" | ssh -t cli.globusonline.org transfer It will return a task id and you can query that specifically via ssh -t cli.globusonline.org details your-task-id If this all works, then try to sync a directory tree from a source to a destination - here is my example from disk to hpss echo "nersc#dtn/global/scratch/sd/porter/reco/2007ProductionMinBias/ nersc#hpss/home/p/porter/ -r -s 1 " | ssh -t cli.globusonline.org transfer You need the slashes at the end of the directory names. The 's 1' means use filesize to determine success. Let me know how that goes. The same syntax could be used to copy files/directories between rcf and nersc once the rcf endpoint is configured. Jeff |
Jeff2:
| Code Block |
|---|
Let's do a transfer. 1st check for endpoints (assuming your globus online user name is same as on your laptop, otherwise add yourname@ in front of the cli...) ssh -t cli.globusonline.org endpoint-list should show nothing. But add the -p, ssh -t cli.globusonline.org endpoint-list -p This should produce a public list that includes nersc#dtn & nersc#hpss. (always just use the -t to hide any passwords you might need ...) Activate both those endpoints, ssh -t cli.globusonline.org endpoint-activate nersc#* This will prompt for a pass phrase - use your nim password. Now list again & this should show the endpoints with a time associated with it ( we can increase the time to 11 days as needed). ssh -t cli.globusonline.org endpoint-list Now copy 1 file from HPSS to disk. Here's my example. echo "nersc#hpss/home/p/porter/auk.tar nersc#dtn/project/projectdirs/star/porter/auk.tar" | ssh -t cli.globusonline.org transfer It will return a task id and you can query that specifically via ssh -t cli.globusonline.org details your-task-id If this all works, then try to sync a directory tree from a source to a destination - here is my example from disk to hpss echo "nersc#dtn/global/scratch/sd/porter/reco/2007ProductionMinBias/ nersc#hpss/home/p/porter/ -r -s 1 " | ssh -t cli.globusonline.org transfer You need the slashes at the end of the directory names. The 's 1' means use filesize to determine success. Let me know how that goes. The same syntax could be used to copy files/directories between rcf and nersc once the rcf endpoint is configured. Jeff |