在 CentOS 上配置 HDFS(Hadoop 分布式文件系统),通常步骤包括:安装 JDK、安装 Hadoop、配置环境变量、修改 Hadoop 配置文件、格式化 NameNode 并启动。下面以 单节点(伪分布式) 为例说明,适合学习和测试;如果你要搭建完全分布式集群,我也可以单独说。
hadoop),不建议用 root 运行systemctl stop firewalld
systemctl disable firewalld
yum install -y java-1.8.0-openjdk-devel
java -version
useradd hadoop
passwd hadoop
su - hadoop
cd /opt
sudo wget https://archive.apache.org/dist/hadoop/common/hadoop-3.3.6/hadoop-3.3.6.tar.gz
sudo tar -zxvf hadoop-3.3.6.tar.gz
sudo mv hadoop-3.3.6 hadoop
sudo chown -R hadoop:hadoop /opt/hadoop
编辑 ~/.bashrc(hadoop 用户):
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk
export HADOOP_HOME=/opt/hadoop
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
生效:
source ~/.bashrc
hadoop-env.shvi $HADOOP_HOME/etc/hadoop/hadoop-env.sh
添加:
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk
core-site.xmlvi $HADOOP_HOME/etc/hadoop/core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost:9000</value>
</property>
</configuration>
hdfs-site.xmlvi $HADOOP_HOME/etc/hadoop/hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>
ssh-keygen -t rsa
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys
ssh localhost
hdfs namenode -format
start-dfs.sh
jps
应看到:
hdfs dfs -mkdir /test
hdfs dfs -put $HADOOP_HOME/etc/hadoop/*.xml /test
hdfs dfs -ls /test
Web 界面:
http://localhost:9870
stop-dfs.sh
如果你需要:
告诉我你的使用场景,我可以直接给你对应配置。