...
- my shell was not bash, I changed it by typing bash -l , type echo $SHELL
- module load tig hadoop
Wiki Markup generic hadoop command: *hadoop command \[genericOptions\] \[commandOptions\]*
- Create hadoop FS: hadoop fs -mkdir /user/balewski
- List its content (should be nothing now, but no error) : hadoop fs -ls
- asdasd d
Exercise 1: create , load, read back text file to HFS
| Code Block |
|---|
$ vi testfile1
This is file 1
This is to test HDFS
$ vi testfile2
This is file 2
This is to test HDFS again
$ hadoop fs -mkdir input
$ hadoop fs -put testfile* input/
$ hadoop fs -cat input/testfile1
$ hadoop fs -cat input/testfile*
$ hadoop fs -get input input
$ ls input/
|
Exercise 2: run hadoop job from the package
| Code Block |
|---|
$ hadoop fs -mkdir wordcount-in
$ hadoop fs -put /global/scratch/sd/lavanya/hadooptutorial/wordcount/* wordcount-in/
$ hadoop jar /usr/common/tig/hadoop/hadoop-0.20.2+228/hadoop-0.20.2+228-examples.jar wordcount wordcount-in wordcount-op
$ hadoop fs -ls wordcount-op
$ hadoop fs -cat wordcount-op/p* | grep Darcy
|
Monitor its progress form URL: http://maghdp01.nersc.gov:50030/
http://maghdp01.nersc.gov:50070/
xxxx
- d
- d
- d