Sqoop 1.2.0安装和测试

安装Sqoop的前提是hadoop完全分布模式已正确安装、配置并运行

STEP 1
1) 下载Sqoop安装包(sqoop-1.2.0-CDH3B4.tar),安装在hadoop-master机器的/usr/sqoop-1.2.0-CDH3B4目录下,并给hadoop帐号该目录的权限。
2) 将hadoop-0.20.2-CDH3B4.tar中的hadoop-core-0.20.2-CDH3B4.jar(hadoop核心包)拷贝到/usr/sqoop-1.2.0-CDH3B4/lib目录下。
3) 拷贝Mysql的jdbc驱动mysql-connector-java-5.1.20-bin.jar到/usr/sqoop-1.2.0-CDH3B4/lib目录下。
STEP 2
修改/usr/sqoop-1.2.0/bin目录下的configure-sqoop文件,增加hbase和zookeeper行的注释

image

STEP 3
在Mysql中增加sqoop帐号的数据库访问权限。

在mysql服务器上给sqoop程序提供一个可以用来连接的账号sqoop,密码admin

mysql> grant all privileges on *.* to'sqoop'@'%' identified by 'admin';

use  test;

create table tb1(id int,name char(10));
insert into tb1 values (1,'aaa');
insert into tb1 values (2,'bbb');
select * from tb1 ;

 

 

提示找不到hadoop,为设置Haoop环境变量    在.bashrc~中添加

export HADOOP_HOME="/home/yangze/soft/hadoop-0.20.2"
export PATH=$PATH:$HADOOP_HOME/bin

 

测试sqoop连接mysql

bin/sqoop-list-databases --connect jdbc:mysql://192.168.70.135:3306 --username sqoop --password admin

image

 

Mysql数据导入hdfs    (test是库名, --table tb1 这个是表名)

bin/sqoop import --connect jdbc:mysql://192.168.70.135:3306/test --username sqoop --password admin --table tb1 -m 1
 

yangze@master:~/soft/sqoop-1.2.0-CDH3B4$ /bin/sqoop import --connect jdbc:mysql://192.168.70.135:3306/test --username sqoop --password admin --table tb1 -m 1

bash: /bin/sqoop: 没有该文件或目录

yangze@master:~/soft/sqoop-1.2.0-CDH3B4$ bin/sqoop import --connect jdbc:mysql://192.168.70.135:3306/test --username sqoop --password admin --table tb1 -m 1

13/05/07 15:05:19 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.

13/05/07 15:05:19 INFO tool.CodeGenTool: Beginning code generation

13/05/07 15:05:19 INFO manager.MySQLManager: Executing SQL statement: SELECT t.* FROM `tb1` AS t LIMIT 1

13/05/07 15:05:19 INFO manager.MySQLManager: Executing SQL statement: SELECT t.* FROM `tb1` AS t LIMIT 1

13/05/07 15:05:19 INFO orm.CompilationManager: HADOOP_HOME is /home/yangze/soft/hadoop-0.20.2/bin/..

13/05/07 15:05:19 INFO orm.CompilationManager: Found hadoop core jar at: /home/yangze/soft/hadoop-0.20.2/bin/../hadoop-0.20.2-core.jar

13/05/07 15:05:21 ERROR orm.CompilationManager: Could not rename /tmp/sqoop-yangze/compile/74f97a951456a21563b7992193b0b7bb/tb1.java to /home/yangze/soft/sqoop-1.2.0-CDH3B4/./tb1.java

13/05/07 15:05:21 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-yangze/compile/74f97a951456a21563b7992193b0b7bb/tb1.jar

13/05/07 15:05:21 WARN manager.MySQLManager: It looks like you are importing from mysql.

13/05/07 15:05:21 WARN manager.MySQLManager: This transfer can be faster! Use the --direct

13/05/07 15:05:21 WARN manager.MySQLManager: option to exercise a MySQL-specific fast path.

13/05/07 15:05:21 INFO manager.MySQLManager: Setting zero DATETIME behavior to convertToNull (mysql)

13/05/07 15:05:21 INFO mapreduce.ImportJobBase: Beginning import of tb1

13/05/07 15:05:21 INFO manager.MySQLManager: Executing SQL statement: SELECT t.* FROM `tb1` AS t LIMIT 1

13/05/07 15:05:23 INFO mapred.JobClient: Running job: job_201305071415_0001

13/05/07 15:05:24 INFO mapred.JobClient:  map 0% reduce 0%

13/05/07 15:05:36 INFO mapred.JobClient:  map 100% reduce 0%

13/05/07 15:05:38 INFO mapred.JobClient: Job complete: job_201305071415_0001

13/05/07 15:05:38 INFO mapred.JobClient: Counters: 5

13/05/07 15:05:38 INFO mapred.JobClient:   Job Counters

13/05/07 15:05:38 INFO mapred.JobClient:     Launched map tasks=1

13/05/07 15:05:38 INFO mapred.JobClient:   FileSystemCounters

13/05/07 15:05:38 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=12

13/05/07 15:05:38 INFO mapred.JobClient:   Map-Reduce Framework

13/05/07 15:05:38 INFO mapred.JobClient:     Map input records=2

13/05/07 15:05:38 INFO mapred.JobClient:     Spilled Records=0

13/05/07 15:05:38 INFO mapred.JobClient:     Map output records=2

13/05/07 15:05:38 INFO mapreduce.ImportJobBase: Transferred 12 bytes in 16.237 seconds (0.7391 bytes/sec)

13/05/07 15:05:38 INFO mapreduce.ImportJobBase: Retrieved 2 records.

yangze@master:~/soft/sqoop-

 
去hdfs中查看一下数据是否已经导入进来

yangze@master:~/soft/hadoop-0.20.2$ bin/hadoop fs -ls /user/yangze/tb1

Found 2 items

drwxr-xr-x   - yangze supergroup          0 2013-05-07 15:05 /user/yangze/tb1/_logs

-rw-r--r--   2 yangze supergroup         12 2013-05-07 15:05 /user/yangze/tb1/part-m-00000

 

image

您可以选择一种方式赞助本站