...
| Code Block |
|---|
The path is via any of 3 machines: stargrid02, stargrid03, stargrid04. I think stargrid04 is the one to use. Therefor you will need to define that endpoint. I just made my definition public, so you should be able to see porter#star4 as an endpoint. You will need to activate it - BUT here is the next step. NERSC recognizes your cert, stargrid04 doesn't. For star to recognize it, you need to make a *** web request with that CERT ***. To make a web request you need to load it into your browser. Ok, It's a pain but at least it's a one-time operation. I'll show you the steps. 1) on a stargrid node, do the 'myproxy-logon -s nerscca.nersc.gov' 2) copy the proxy file, /tmp/x509up_whateveritisforyou, to your laptop. 3) change it to pk12 format: openssl pkcs12 -in yourcopiedoverfile -export -out jans-nersc.p12 It will ask for a password to encrypt it. 4) import into your browser. I'll assume firefox: preferences->Advanced->Encryption->View Certificates Then click 'import'. It will ask for the password of the file (it may first ask for your password if yuo've locked the browser) 5) Once that's installed (it will be in the list in the preferences panel) go to: https://vo.racf.bnl.gov:8443/voms/star/ It should see your cert (displayed on the webpage). From the page, request to be a member of the STAR VO. You'll need to reply to an automatic email, but then Jerome will be notified. |
...
Optimization of globus transfer, various suggestions
| Code Block |
|---|
guc is globus-url-copy - acronym invented by Levente srm is a software layer over that - it makes sure transfers complete There is a –r recursive copy option; Or else you can do a loop in shell: set daq = (1380001.daq 1380003.daq 1380004.daq); foreach f ($daq) globus-url-copy -p 25 file:/star/bla/bla/$f gsiftp://pdsfgrid.nersc.gov/bla/bla/bla/$f end If you specify a source ending in "XXX/" , it will treat XXX as a directory and transfer all files in XXX from the XXX directory Use -p 25 option for having more streams, but isn't 25 overdoing it? You got 1 file cut into 25 buffer chunks and all this has to be re-assembled on arrival. I'd try +-2 around 8. Use different end-point machine: dtn01.nersc.gov to avoid STAR,ATLAS conflicts there is also carvergrid.nersc.gov gatekeeper I have access too. pull from carver, you need to load the osg module ,and then you have the globus commands available globus-job-run works like remote ssh command Look at this page: http://rcsg-gsir.imsb-dsgi.nrc-cnrc.gc.ca/globus_tutorial/#running for globus-job-run, globus-job-submit and globus-job-get-output (job is a command or script, not necessarily something that goes into the batch system) dtn01 is a data transfer node. It has a gsiftp server but no gatekeeper. carvergrid has both but dtn01 has a 10Gb erthernet and carvergrid only 1Gb. You can use dtn01 for globus-url-copy and carvergrid for globus-job... type commands. You can test a gridftp from stargrid node (01,02,03,04) right now to either gsiftp://dtn01.nersc.gov/global/scratch/sd/balewski/... or to gsiftp://pdsfdtn1.nersc.gov/project/projectdirs/star/... |
...
...
Testing globus, by Jeff:
| Code Block |
|---|
Ok. Let's do a transfer.
1st check for endpoints (assuming your globus online user name is same as on your laptop, otherwise add yourname@ in front of the cli...)
ssh -t cli.globusonline.org endpoint-list
should show nothing. But add the -p,
ssh -t cli.globusonline.org endpoint-list -p
This should produce a public list that includes nersc#dtn & nersc#hpss.
(always just use the -t to hide any passwords you might need ...)
Activate both those endpoints,
ssh -t cli.globusonline.org endpoint-activate nersc#*
This will prompt for a pass phrase - use your nim password.
Now list again & this should show the endpoints with a time associated with it
( we can increase the time to 11 days as needed).
ssh -t cli.globusonline.org endpoint-list
Now copy 1 file from HPSS to disk. Here's my example.
echo "nersc#hpss/home/p/porter/auk.tar nersc#dtn/project/projectdirs/star/porter/auk.tar"
| ssh -t cli.globusonline.org transfer
It will return a task id and you can query that specifically via
ssh -t cli.globusonline.org details your-task-id
If this all works, then try to sync a directory tree from a source to a destination
- here is my example from disk to hpss
echo "nersc#dtn/global/scratch/sd/porter/reco/2007ProductionMinBias/
nersc#hpss/home/p/porter/ -r -s 1 " | ssh -t cli.globusonline.org transfer
You need the slashes at the end of the directory names. The 's 1' means use filesize to
determine success.
Let me know how that goes. The same syntax could be used to copy files/directories between
rcf and nersc once the rcf endpoint is configured.
Jeff
|