You need to use these parameters as per your system. Note: In my system, MASTER IP is 192.168.5.101 and JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64. Now, to edit the spark-env.sh configuration file, run the following command:Īdd the following parameters (line of code) at the end of file, save and exit.Įxport SPARK_MASTER_HOST=''export JAVA_HOME= Traverse to the spark/ conf folder and make a copy of the spark-env.sh.template file as a spark-env.sh To setup Apache Spark Master configuration, edit spark-env.sh file. Run the following command to make effective changes in the. The above line sets the location (Path) where the spark software file is located to the PATH variable. bashrc file:Īdd the following line to the file and save. To set up the environment for Apache Spark, we need to edit the . $ sudo mv spark-3.1.1-bin-hadoop3.2 /usr/local/spark Run the following command to move the spark files to the spark directory ( /usr/local/bin): Now run the following command to untar the spark tar file: To download spark-3.1.1 with Hadoop-3.2, run the following command: I will install spark-3.1.1 with Hadoop-3.2. (On all the Virtual Machines – Master and workers)ĭownload the stable version of Apache Spark. You can exit from slave machine by type the command: Run the following command on master to connect to the slaves / workers: Now it’s time to check if everything installed properly. $ ssh-copy-id ssh-copy-id ssh-copy-id user name and IP will be different of your machines. ssh/authorized_keys (all the workers/slaves as well as master). Now copy the content of .ssh/id_rsa.pub form master to. $ cat ~/.ssh/id_rsa.pub > ~/.ssh/authorized_keys Run the following command to authorize the key:

For this purpose, run the following command: $ sudo apt-get install openssh-server openssh-client

To configure Open SSH server-client, run the following command: Now configure Open SSH server-client on master. To check the version of Scala, run the following command: Install Scala on the all the machines (master and the worker / slaves).

#to check version of java, run the following command. Run the following commands on all the Machines (master and workers / slaves).

Install spark locally windows 10