Install Hadoop on MacOS (Macbook M1)

The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models.

In this article I have shown how you can install hadoop on MacOS specific on Macbook M1 fixing the possible errors. So let’s get started.

Step 1 : Install HomeBrew

/bin/bash -c “$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

Use these commands to add homebrew on your bash_profile

echo ‘eval “$(/opt/homebrew/bin/brew shellenv)”’ >> /Users/arjungautam/.bash_profile

eval “$(/opt/homebrew/bin/brew shellenv)”

Step 2 : Install Hadoop using HomeBrew

brew install hadoop

Open the document containing the environment variable settings :

$ cd /opt/homebrew/Cellar/hadoop/3.3.1/libexec/etc/hadoop

then open hadoop-env.sh with any editor, I will use vscode

$code hadoop-env.sh

Make sure to have java jdk installed and JAVA_HOME setup on your system. If you haven’t follow this link ( Install Java 8 or 11 not newer versions)

Find JAVA_HOME on that file and add your local Java path

Use this command to find the path in terminal

$ /usr/libexec/java_home

and add the location on hadoop-env.sh

Step 3: Make changes to Core files

cd /opt/homebrew/Cellar/hadoop/3.3.1/libexec/etc/hadoop
code core-site.xml

and add the following configuration

<configuration> <property>  <name>fs.defaultFS</name>  <value>hdfs://localhost:9000</value> </property></configuration>

Step 4 : Make changes to hdfs files :

cd /opt/homebrew/Cellar/hadoop/3.3.1/libexec/etc/hadoop
code hdfs-site.xml

<configuration>  <property>    <name>dfs.replication</name>    <value>1</value>  </property></configuration>

Step 5 : Make changes to mapred files

cd /opt/homebrew/Cellar/hadoop/3.3.1/libexec/etc/hadoop
code mapred-site.xml

<configuration>    <property>       <name>mapreduce.framework.name</name>       <value>yarn</value>    </property>    <property>    <name>mapreduce.application.classpath</name>   
<value>
$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*
</value>
</property></configuration>

Step 6: Make changes to yarn files :

cd /opt/homebrew/Cellar/hadoop/3.3.1/libexec/etc/hadoop
code yarn-site.xml

<configuration>  <property>    <name>yarn.nodemanager.aux-services</name>    <value>mapreduce_shuffle</value>  </property>  <property>    <name>yarn.nodemanager.env-whitelist</name>     <value>
JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME
</value>
</property></configuration>

Now just go to the system preferences and

Now to run Hadoop

But before doing that just type following command on terminal.

hadoop namenode -format 

just open terminal and type start-all.sh

And type jps to have confirmation that all the parts of Hadoop have been installed and running. You should see something like this

now just type http://localhost:9870

Now to create the directory

$ hadoop fs -mkdir /user
$ hadoop fs -mkdir /user/arjun.gautam

Add files on the existing directory

$ hadoop fs -put Demo/ /user/arjun.gautam

If you followed this article and worked for you please help me by following on the medium.
Also visit my youtube channel ArjunCodes .

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store