在 CentOS 上“连接 HDFS 客户端”通常有两种理解:
下面分别说明最常见、最实用的做法。
HDFS 依赖 JDK:
java -version
若未安装:
yum install -y java-1.8.0-openjdk
wget https://downloads.apache.org/hadoop/common/hadoop-3.3.6/hadoop-3.3.6.tar.gz
tar -zxvf hadoop-3.3.6.tar.gz -C /opt
ln -s /opt/hadoop-3.3.6 /opt/hadoop
vi ~/.bashrc
添加:
export HADOOP_HOME=/opt/hadoop
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
生效:
source ~/.bashrc
core-site.xml编辑:
vi $HADOOP_HOME/etc/hadoop/core-site.xml
示例(连接 NameNode):
<configuration>
<property>
<name>fs.defaultFS</name>
<name>hdfs://namenode-host:9000</name>
</property>
</configuration>
namenode-host替换为你的 NameNode 主机名或 IP
hdfs dfs -ls /
或:
hadoop fs -ls hdfs://namenode-host:9000/
hdfs dfs -put file.txt /user/test/
hdfs dfs -get /user/test/file.txt ./
hdfs dfs -mkdir /test
hdfs dfs -rm /test/file.txt
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>3.3.6</version>
</dependency>
Configuration conf = new Configuration();
conf.set("fs.defaultFS", "hdfs://namenode-host:9000");
FileSystem fs = FileSystem.get(conf);
Path path = new Path("/test");
System.out.println(fs.exists(path));
firewall-cmd --add-port=9000/tcp --permanent
firewall-cmd --reload
确保 /etc/hosts 中能解析 NameNode:
192.168.1.10 namenode-host
hdfs dfs -chmod 777 /user/test
那只需:
core-site.xml如果你能补充:
我可以给你更精确的步骤。