Write and execute what count program here we shall mention step by step that how to write and how to execute our what count program
So to write a map, produce program like our work program, we must be using some ID
So we'll be using say eclipse id in eclipse, create new Java project
create a package if it is necessary because it is optional and create the respective classes
We should attach Hado Jars to use our Java program as map task and all the jars available in this four locations
or paths are to be included in our respective newly created project
So after importing the jars, write mapper and reducer class and the main function and export the project as a jar file
So we'll be writing the code for the mapper and the reducer class
And obviously the main function will be there and there will be converting this project in the form of jar in the terminal
start the Hadoop using this. So this is a command with the help of which we shall be running our Hadoop on our system
run the jar file and the syntax is Hadoop Jar Jar path
Then the class name and then arguments. So if the class name is having some package
then you shall be writing package dot class name. So let us go for one practical demonstration for the easy understanding of this concept in
this video, I'm going to show you how to configure and run the whatcom program in our mapreduce environment
And we shall follow step by step to show you that how, what are the different steps with the help of which we can run
this? What program in our map we'll be writing, we'll be having our code in eclipse
we'll be running the Hadoop at first. So at first, we're going to initiate the Hadoop
we should run Hadoop at first because in the Hadoop environment only we'll be going for this program execution
So to init initialize Hadoop, we're going to type that is a dollar Hadoop underscore install all in capital letters
SB slash start hyphen all S. So this is the shell program we are supposed to execute to initiate the
Hadoop environment. The Hadoop software will be running on our system and then we shall go to the browser to check whether this hub is
working there or not. So here you can find the how to be running the
hub is initiating. Now we've got the dollar prompt. So the hard is running successfully
we are going for JP S to see that what are the different processes running
So data note, node manager name note, secondary name note. So all these processes are running so hard is running perfectly
Ok. We're going for the browser now. So going for the browser
we'll be typing local host colon. Let me initialize it. Yes, we'll be typing local host colon
570. So 50070 slash and it shall give slash and then it shall present it
You see this had gy is now effective by getting all the details about the name notes
Now you'll be clicking on the utilities and browse the file system to see what are the different file systems we're having under
this Hadoop route. So these are the file systems. We shall go for Hadoop my files
You can click on that and these are the respective programs. Are there files are there
we'll be working on the sample underscore file dot TXT, sample S capital
file F capital. So sample underscore file dot TXT. So this is a sample file which is containing some strings are there
We're supposed to count the number of words, the frequence of occurrence of these words in this particular a sample underscore file
dot TX. So let me copy, let me, let me go for the for the content
Let me show you the content. So DFS, dfs hyphen cat had my files so I can type sample underscore
file dot txt. Otherwise I can use the wild card character star here. So that only this particular file is there uh having got
the name sample underscore file dot txt. So instead of typing the full file name
I'm just giving a start as a wild card character. This is a corresponding text we are having
So this is a corresponding text we're having. And this particular text we're going to count the number of words there
So this is a text content. You can find that this is a text content. Now
we are going to make one directory, we're going to make one directory and the directory name
We're going to create this directory uh directly at the root of the Hadoop
So sample input, we're giving the directory name as sample input. So you can find that is DFSDFS minus MKDIR
sample slash sla sample unit in uh sample unit input. We're having the
the respective folder name and you know that slash means it will be created under this uh root of the Hado file system
So the f the folder has got created. Now, we shall copy this sample from this had my files
You'll be copying this one. You see the folder will be shown here if I go for the refresh
So first time it is not coming. So sample input, this folder is a blank folder. You can find that there is no file or
direct sub director are there. So under the Mle input. So from the uh from the Hadoop
my files will be doing the copy of the sample underscore file dot txt into this sample input folder
So we are going to issue the command that is S DFS, DFS minus CP
We'll be going for this minus CP here and then slash Hadoop my files
There's a source path slash means the hub dot So how do my file
Sample star? You can go for sample star here. So sample start and the target will be
under root sample input I capital sample input. So to this particular folder
this sample underscore file dot TXT file is getting copied. So this is a source path with the file name and this is a target
path. So pressing enter and the file has got copied. I can move to the browser
We shall go for the sample input. You can find that the sample underscore file dot TXT
The file has got copied there. So that is 159 byte file
159 byte. OK? Now we shall go for the eclipse. Now we shall go for the eclipse
So there is a Hadoop demo. There is a Hadoop demo. We're having the SRC folder there but there is some errors are there because
it requires some jar files. So let me include jar files here and you'll be importing these jars
You'll be including these jars from four different folders. So you'll be following one by one
So going for the right click on this project name, going for the right click
Then we shall go for this build path and configure build path
Then you shall click on the add external Js. You see we're having only this GRD system library JDK 1.8 0.0 underscore 181
So this library is there only we're going for a external Js
So at first we are going forward share, we are going for the Hado folder at first
So there is a home, so there is a big data home here
So you'll be going for had, then you shall go for the share
then she'll go for her Duke and then come on. So we're going to import these three jar files there
So from the four parts, we are going to import jars. So these are part number one that is slash Hadoop slash slash Hadoop slash
common. Now, next one going for the second one we'll be going for. Now there is a common lib
So we have gone for this Hadoop and then common, then lib. So multiple jars are there
We'll be going for the that is a con to a select all and then OK
So second part we have done the 3rd, 3rd part will be our there is a map Hadoop Mapreduce
So here we're having nine jar files are there I can find. So I'm going for adding the sectional
The last one will be Hadoop Madu Lib. Under this lead. We are having multiple jars
So I'm going for control. A that is a select all and clicking on
OK. So all the jar files have got imported here. So apply and close
You can find that the error symbol, the error market has gone up, gone out
So now we are having this uh package is the default package under SRC
We're having this my wordcount dot Java. So this is a program we are having just executing this one
You can find that this is our code. So our class name is my wordcount
And under this very class, we're having multiple inner classes. One is a WC mapper
So my class name is this one. So WC Mapper and then we're having main function and also we are having WC reducer
So WC mapper, WC reducer. So two inner classes we're having and the main function is there
So this WC mapper is extending this mapper which is having this wrong writeable
That is a value. Text is a key. So that will be taken as input and it will produce output as text as a text
as a key and long writeable as output. So it is showing the key value pair here
So long writeable is the value and text is a key. So that is our WC mapper which is extending mapper
And here we have defined some instance variable within this class. That is the WC mapper
WC stands for actually what count. So this is the map function we're going for this override this map function so long writeable key and
text value. So mapper wrong writeable comma text, comma text, comma long writeable
So this is a respective method we're going forward overriding. And this is about exceptions which will be thrown
We have mentioned that one. So the value which will be passed as input will be converted to string and put onto the string object
line. And from this line, we're using this tokenizer, string tokenizer. So token tokenize is the respective object
And from this token eyes, we'll be going for hash more tokens. And using the next token method
we'll be reading each and every token there. And that will be assigned onto the term token
That is a string object. And then we are replacing all these characters
That is our opening third becket dot We're having comma Kim Colon
semi colon closing third beer. So all these things will be replaced by the null string there
You can find that they will be replaced all and we were converting all of them
We are converting all the strings in the lower case. So, so let us suppose that is one
What is this? So this can be written in capital letters, small letters
a mixture of that, but they'll be converted to lower case letters. So whenever we are going to count this
this, so now this is about context dot Right token one. OK
So that is a WC reducer where you are overriding the method reduce
So text key and iterable long writeable values. So we're having this sum is zero so long writeable
Well, we're using the for each loop which will be working on values and some classical to well get and then result set num sum
here and context, right, result. So key and results that is a key value pair will be written in the
context. So these are our main functions. So this main function requires on configuration object
So configuration can physical new configuration. And we require the Kamal and input argument must be having if it is less than two
if the KKK and input count is less than two, then one error will be printed and system will get terminated because system exit minus
one. So I require two inputs. The first one is the source and next one will be the target
So that's why we, we shall be taking the K and input arguments when you'll be running this code
So here we have created one job instance we can give any name. So it is not like that
Only my job is allowed, you can give any name and then we're going for job dot set jar by class
So my what count do class? What is a my what count? So that is our class name
You can find that my what count is nothing but the class name. So that has been mentioned here
So my word count class OK. So to access the common and input argument for the first argument will be using a zero
For the second argument, we will be using aux one. So that is the add input path will be the AU zero and set output
path will be the aux one. So to access the input arguments K and input arguments will be using this A zero and a one
So there is a job dot set mapper class that is a WC mapper and job set reducer class WC reducer which you define earlier
as in a class. So job set output key class that is a text dot class
So output key will be text and output value will be long writeable dot class
So you see line number 66 and 67 we mentioned what to be the key type and what will be the value type
I mentioned that one line number 66 and line number 67. Now we are going to check whether the job is getting completed or not
So it is it will return, wait for completion. This method will return a bullion
So success. So using this success, if the success is true, then we'll be uh we'll be having the exit value zero
If the success is false, then you'll be having the exit value one. So in this way
these are our main function we have written. So we have imported, we have included all the jars
Now we are going to make, I shall go for this. So Hadoop
we shall go for the right click. Then we shall go for the export and the jar file is there
It's OK. So jar then I shall go for next. So this is a hard demo
So how about? So now here we'll be creating one output folder
here. So I'm creating one output folder
Let it be M jar files. So here the output will be dumped under the big data home path
So here is what count dot Jar, we've done it. We've just created the jar under Emad Jar files
So one warning is coming, I'm going for OK. So this jar has been exported to the respective folder there
So in my files, you see the WCO chart. If you double click here
you can find that we are having the class. That is a my what count dot class
We're having the inner classes there W CM class and WC reducer class
We're having the sample dot txt. So this is a jar we're having all the classes are there
So there is a sample input it containing sample file dot txt
So this part will be requiring. Now let us execute our program
run our program. So clearing this one using control L. So Hadoop Jar is
a home path, tailed slash Emard slash AMR Jar files
So there is a folder where the jar file was saved. So I shall go for these Amar jar files then slash what
count dot jar. Then you shall go for my word count. There is a class name
If you have the package, if you have the package, then obviously the word count is there
If you have the package, then obviously I should mention the package name also
But here this my word count class is under the default package
So that's why begging for my word count, there is a class name slash sample input
So there is a sample input from where the file will be taken
And then we shall go for the output folder. You see the output folder is not there. So you see there is no output folder
So it will be created, it will be created. You see there is no output folder there
it will be created when this command will be executed. You
see it is running perfectly. OK? So now I think the output folder has got created 168 bytes
have got written onto the file so I could fold. It is not that if let me go for refresh
you can find that the output folder has got created. If you click it
we're having two files. There one is the underscore success. Another one is a part hyphen R hyphen five zeros
OK? So the output is has been dump dumped on that part file
So we shall go for DFS, then DFS minus LS, then output which is under
the hado root. So let me see that what are the file names? We have also seen that one in the gy interface
So we're having two files. One is a part hyphen R hyphen 50 and underscore success
So part hyphen R hyphen 50. So this is the main file here
So let us go for S DFS DFS minus cat cat to see the file content
we shall go for slash output slash part. I
can go for the wild card characters. The rest part will be taken
So if you see the content, you can find that it is showing that how many words are there
what are the distinct words and their respective frequencies? How many times it has occurred
So as we converted all the words to the lower case, so uppercase and lowercase distinguish will not be done
They are not distinguished. So A has occurred for one time. You can find that say this file has occurred for four times
So respective, what count has been done. So this what count program actually we treat as a Hallow world program in our mapreduce
OK? So we know that if this output folder exists. So next time
next time the another program cannot get executed. So we should delete this output folder because next time if we execute any other map to
this program, so I shall go for this DFS and then minus R minus R
So that is a recursive all the folders and sub folders will get deleted and you shall go for this slash output
So in this way, we've shown you that how to run one map program in our had environment
And what are the different steps should be followed and how to import the external jar files and how to put
the copy of the sample underscore file dot txt onto a certain folder
There is a sample input how to create the folder. Everything we have done step by step to show you how to make the jar
files, how to execute the code, how to see the output we have, we have shown you everything into details
I think now it is cleared up your conceptions. So thanks for watching this video