Install Hadoop on MacOS (Macbook M1)

Install Hadoop in MacOS

The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models.

In this article I have shown how you can install hadoop on MacOS specific on Macbook M1 fixing the possible errors. So let’s get started.

Step 1 : Install HomeBrew

/bin/bash -c “$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

Use these commands to add homebrew on your bash_profile

echo ‘eval “$(/opt/homebrew/bin/brew shellenv)”’ >> /Users/arjungautam/.bash_profile

eval “$(/opt/homebrew/bin/brew shellenv)”

Step 2 : Install Hadoop using HomeBrew

brew install hadoop

Open the document containing the environment variable settings :

$ cd /opt/homebrew/Cellar/hadoop/3.3.1/libexec/etc/hadoop

then open hadoop-env.sh with any editor, I will use vscode

$code hadoop-env.sh

Make sure to have java jdk installed and JAVA_HOME setup on your system. If you haven’t follow this link ( Install Java 8 or 11 not newer versions)

Find JAVA_HOME on that file and add your local Java path

Use this command to find the path in terminal

$ /usr/libexec/java_home

and add the location on hadoop-env.sh

Adding JAVA_HOME

Step 3: Make changes to Core files

cd /opt/homebrew/Cellar/hadoop/3.3.1/libexec/etc/hadoop
code core-site.xml

and add the following configuration

<configuration> <property>  <name>fs.defaultFS</name>  <value>hdfs://localhost:9000</value> </property></configuration>

Step 4 : Make changes to hdfs files :

cd /opt/homebrew/Cellar/hadoop/3.3.1/libexec/etc/hadoop
code hdfs-site.xml

<configuration>  <property>    <name>dfs.replication</name>    <value>1</value>  </property></configuration>

Step 5 : Make changes to mapred files

cd /opt/homebrew/Cellar/hadoop/3.3.1/libexec/etc/hadoop
code mapred-site.xml

<configuration>    <property>       <name>mapreduce.framework.name</name>       <value>yarn</value>    </property>    <property>    <name>mapreduce.application.classpath</name>   
<value>
$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*
</value>
</property></configuration>

Step 6: Make changes to yarn files :

cd /opt/homebrew/Cellar/hadoop/3.3.1/libexec/etc/hadoop
code yarn-site.xml

<configuration>  <property>    <name>yarn.nodemanager.aux-services</name>    <value>mapreduce_shuffle</value>  </property>  <property>    <name>yarn.nodemanager.env-whitelist</name>     <value>
JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME
</value>
</property></configuration>

Now just go to the system preferences and

Localhost connection refused 22 resolved

Now to run Hadoop

But before doing that just type following command on terminal.

hadoop namenode -format 

just open terminal and type start-all.sh

Using start-all.sh

And type jps to have confirmation that all the parts of Hadoop have been installed and running. You should see something like this

Using jps command

now just type http://localhost:9870

HDFS running on localhost:9870

Now to create the directory

$ hadoop fs -mkdir /user
$ hadoop fs -mkdir /user/arjun.gautam

Add files on the existing directory

$ hadoop fs -put Demo/ /user/arjun.gautam

If you followed this article and worked for you please help me by following on the medium.
Also visit my youtube channel ArjunCodes .

--

--

--

Learn Code with Fun.

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Introduction To Docker Life Cycle

What is the order of execution for SQL statements

Introduction to Quantum Programming

Quantum computer

From a High School Coder to an International Tech Speaker — Interview with Arun Michael Dsouza

𝗪𝗵𝘆 𝗱𝗶𝗴𝗶𝘁𝗮𝗹 𝗺𝗮𝗻𝘂𝗳𝗮𝗰𝘁𝘂𝗿𝗶𝗻𝗴 𝗮𝗹𝗹𝗼𝘄𝘀 𝘆𝗼𝘂 𝘁𝗼 𝗰𝗿𝗲𝗮𝘁𝗲 𝗺𝗼𝗿𝗲…

Get to know Flexport Engineering: Bookings Team

How To Replay Recreated AWS DynamoDB Streams

Discovering my favorites topics in Hacker News with NLP

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Code With Arjun

Code With Arjun

Learn Code with Fun.

More from Medium

Build Your First Pypi Package

A simple bot to send emails in Google Cloud

Reduce your automation script development time using selenium, python and pytest

Wrapping Things up for 2021