在 Debian 下启动 Hadoop,通常有两种常见模式:单机模式(Local) 和 伪分布式(Pseudo-Distributed)。下面以 Hadoop 3.x 为例,说明启动步骤。
java -version
若未安装:
sudo apt update
sudo apt install openjdk-11-jdk
wget https://downloads.apache.org/hadoop/common/hadoop-3.3.6/hadoop-3.3.6.tar.gz
tar -xzf hadoop-3.3.6.tar.gz
sudo mv hadoop-3.3.6 /opt/hadoop
nano ~/.bashrc
添加:
export HADOOP_HOME=/opt/hadoop
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64
生效:
source ~/.bashrc
直接运行示例:
hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.6.jar wordcount input output
etc/hadoop/core-site.xml<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost:9000</value>
</property>
</configuration>
etc/hadoop/hdfs-site.xml<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>
etc/hadoop/mapred-site.xml<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
etc/hadoop/yarn-site.xml<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>
sudo apt install ssh
ssh-keygen -t rsa
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
ssh localhost
hdfs namenode -format
start-dfs.sh
start-yarn.sh
jps
应看到:
NameNode
DataNode
ResourceManager
NodeManager
SecondaryNameNode
Web 界面:
stop-yarn.sh
stop-dfs.sh
JAVA_HOME 未设置
在 etc/hadoop/hadoop-env.sh 中设置:
export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64
Permission denied (publickey)
SSH 未配置好
如果你是用 Debian + systemd / 集群 / Docker,可以告诉我具体环境,我可以给你更精确的启动方式。