WEBVTT

00:00.000 --> 00:05.240
In this video we are discussing median
and standard deviation map reduce

00:05.240 --> 00:10.500
Whenever we are having a set of data, at
first we should short it and the middle most

00:10.500 --> 00:15.660
data would be known as a median and standard
deviation measures the deviation of

00:15.660 --> 00:20.820
our data from the average value. So let
us discuss median and standard deviation

00:20.820 --> 00:25.920
map reduce. So at first we are starting
with the median so median from the data set

00:25.920 --> 00:29.980
The median is the numerical value
which is used to separate the data

00:29.980 --> 00:34.920
set into lower and upper halfs so lower
half means all the data which will be

00:34.920 --> 00:39.420
lesser than the median and upper half means
all the data values which will be higher

00:39.420 --> 00:44.700
than the median value he requires complete
data set and also in the shorted order

00:44.700 --> 00:50.040
in a simple map-produced task it is very
difficult to short data because here we are

00:50.040 --> 00:55.320
dealing with a huge amount of data so shorting
of those data will be a very difficult

00:55.320 --> 00:59.980
task we should use some better
logic to perform the operation very

00:59.980 --> 01:08.400
сред efficiently. Next we are going
to discuss standard division of data set

01:08.400 --> 01:14.880
So the standard division shows how much
variations are there in the data from the average

01:14.880 --> 01:21.320
value of the data. And we need to find
an average before reducing the task

01:21.320 --> 01:24.980
So in this case, in our example,
we will take the comments

01:24.980 --> 01:35.360
So this is one XML we are having already. So comments. XML and from
that find the median and standard division of the comment lengths

01:35.360 --> 01:41.660
So from the comments.xml file we shall calculate
the median and also the standard division

01:41.660 --> 01:50.400
of the comments length. So let us go for a
demonstration for the easy understanding of this concept

01:50.400 --> 01:57.680
So in this video we are discussing that is
our median and standard division MR task

01:57.680 --> 02:03.920
falling under the summarization design pattern so
we are having the comments dot xml is our input

02:03.920 --> 02:09.600
file under the folder input slash comments so
under this one we are having this comments

02:09.600 --> 02:17.040
So let me see the what is the current content
here so it is 37.9 8 mb so we have just taken

02:17.040 --> 02:23.600
a snap a portion of this comments dot XML and
we're just opening this one so this is about

02:23.600 --> 02:27.280
within the comments tag we're having
certain number of rows are there

02:27.680 --> 02:34.360
And each and every row has got multiple attributes.
So each and every row has got attributes

02:34.500 --> 02:42.860
We're having the attributes like your ID and then post ID, is code
and the comment that is a text, creation date and the user ID

02:43.580 --> 02:57.777
So these are the respective attributes are there under each and every row We having two classes are there That
is the made ST that is a standard division data data data and made is the MR taxed or Java so made is D data

02:57.777 --> 03:03.417
implements writable so this is writable
interface is getting implemented here we're

03:03.417 --> 03:07.257
having the set of this getter and
setter methods we're having the member

03:07.257 --> 03:11.737
variables like median and standard division
these two are the member variables so we're

03:11.737 --> 03:16.617
having some getter and setter methods are
there accordingly get median get sd set

03:16.617 --> 03:21.597
median set s d we're having well overriding
this read fields this particular

03:21.597 --> 03:25.917
method is getting overrated right is
also getting overridden and we're also

03:25.917 --> 03:30.477
overriding the two string method for the
proper output so we can find that we're

03:30.477 --> 03:34.617
having the read files so median and
standard division these two variables are

03:34.617 --> 03:40.257
getting calculated in data dot read in and
in data dot read double we're having the

03:40.257 --> 03:44.817
right method so right method is also
printing this median and standard

03:44.817 --> 03:49.617
division accordingly and we are having
the two string to string method is also

03:49.617 --> 03:55.017
there for the proper display so now
what we shall do we shall go for made

03:55.017 --> 04:01.857
ST MR tucks java so it is it is the
class and this class is a user's

04:01.857 --> 04:05.697
comment form at comments dot XML which
will be read and determine the median

04:05.697 --> 04:10.557
and standard division of the common lens
part hour of the day we're having that

04:10.557 --> 04:14.797
data format date format is also
defined here there is the hours so so

04:14.797 --> 04:18.657
we will be going for the date format date
format is also defined you can find the

04:18.657 --> 04:25.777
respective format there and we are having
two inner classes so there is a map

04:25.777 --> 04:32.817
made sd mapper which is inheriting
mapper class and also we are also

04:32.817 --> 04:37.697
are having the reducer class will be
also in rated in another inner class

04:38.897 --> 04:44.577
so we are having the uh that is output
hour and output comment length to

04:44.817 --> 04:50.337
variables are there within this inner
class and also we are overriding the

04:50.337 --> 04:56.697
map method XML parts will be updated by
the output of the method that is the XML

04:56.697 --> 05:01.857
to map so XML to map is a method which
we have written already which will take

05:01.857 --> 05:07.497
the XML as input and returns the hash
map object as output so it will read the

05:07.497 --> 05:11.637
XML and returns the hash map as output
and that will be dumped onto this XML

05:11.637 --> 05:20.517
parts to variable so we are having the create date and a comment so
from the xml parse dot get and xml pars dot get text xml parts.d. Get

05:20.517 --> 05:27.817
creation date and xm.pars. Dot. Wherein
the output hour set here we are having

05:27.817 --> 05:32.337
this get hours method is deprecated but it
will work no issues and then output comment

05:32.337 --> 05:36.897
length.net.net. There is a comment. length.
So you can find that we are writing this key

05:36.897 --> 05:52.035
value pairs onto the context We writing this key value pairs onto the context We having the made SD and
there is a median standard division reducer which will be extending the reducer class and obviously as other

05:52.035 --> 05:57.555
programs so here also will be overriding the
reduce method here this reduced method will

05:57.555 --> 06:03.155
be overwritten so what you shall do here in
this case will be calculating the median

06:03.155 --> 06:07.955
and also will be calculating this
here we'll define one array at a list

06:09.555 --> 06:13.315
you have defined one add a list so initially
the list has been made clear there is a

06:13.315 --> 06:19.875
comment length list and output value dot said sd
is equal to zero so at first the comment length

06:19.875 --> 06:26.115
list add a list will be cleared and also the op
values dot set is cleared now we are going for

06:26.115 --> 06:32.115
this uh comment length dot list add so for each
and every uh item is going to get added get the

06:32.115 --> 06:37.794
lengths and add them to the list and total length
also we are going to increase with this current

06:37.794 --> 06:43.735
value of the length and also the comment
count has to be increased here so after doing

06:43.735 --> 06:47.695
all this so we'll be going for the short
because we are going to calculate the

06:47.695 --> 06:52.535
median median is the middlemost value after
getting shorted so we have defined one

06:52.535 --> 06:59.014
variable that is a the median value and if
this comment count is even then we shall

06:59.014 --> 07:04.974
take the comment count value which is at
the n by 2 minus 1 and n by 2 plus and

07:04.974 --> 07:14.035
there some has to be has to be calculated and then it will be divided
by two so we are taking the average of the comment count values at the

07:14.035 --> 07:19.555
position n by 2 minus 1 and at the position n by
2 so those two comment count values are to be

07:19.555 --> 07:25.955
averished and that will be kept that will be
decided as a median median value but if the comment

07:25.955 --> 07:30.755
count is odd that means in the else part will be
taking the respective value which will be found

07:30.755 --> 07:37.775
at the middle and that is the median value and op
oop value dot set median will be writing this one as

07:37.775 --> 07:42.055
median valve so in this way the median
has to be has been calculated now let us

07:42.055 --> 07:46.135
calculate the standard division so at
first we're calculating the mean that is a

07:46.135 --> 07:50.215
total in by common count and then we'll
be going for the square sum so we're

07:50.215 --> 07:54.715
going to calculate the respective val
minus mean into val minus mean means we're

07:54.715 --> 07:59.575
going to do the square we're going to
calculate the square sums difference

07:59.575 --> 08:05.635
between val minus mean square and then
we'll be taking the square root after

08:05.635 --> 08:10.315
dividing it by common count minus one so
here we are calculating the squares and

08:10.315 --> 08:15.755
then opi val dot set s d will be going
for math dot square root that this

08:15.755 --> 08:20.995
squares come so squared sum by common
count minus one so in this way in this

08:20.995 --> 08:24.534
way the standard division has been calculated
and that has been written onto the

08:24.534 --> 08:30.115
context the key value pair has been
written onto the context so here we're

08:30.115 --> 08:43.412
having the main function we require two arguments from the common line if the length is not equal to 2 then we
be exiting with some error message exit to And then otherwise we be going for this job instance will be created

08:44.212 --> 08:49.252
And then we are going for add input
path, set output path with this job

08:49.252 --> 08:55.632
And then we know the respective mapper class
and reducer class we have inherited here

08:55.872 --> 09:05.472
So we're writing this one as job. dot set mapper job job dot set
reducer job dot set output key job dot set output value class

09:05.472 --> 09:10.592
output key class and output value class also
and we are going for the bullion and if the

09:10.592 --> 09:15.732
condition is true or false accordingly will be
returning the bullion value so in case of success

09:15.732 --> 09:20.392
will be returning zero in case of failure
will be returning one so similarly we have

09:20.392 --> 09:25.452
written this respective code so now let us
go for the jar file creation so we have gone

09:25.452 --> 09:30.612
for the export on the package name and then
we'll be giving the proper path and the file name

09:30.612 --> 09:35.932
and then you shall go for next and finish but
already we have created this jar file so we're

09:35.932 --> 09:41.372
skipping this particular step now so the jar
file has got created so we have come back to the

09:41.372 --> 09:49.452
command prompt so here the command is hadoo jar
and then we are going to give the the jar path

09:49.452 --> 09:54.972
and the respective jar file name
and then mED s d is the package name

09:54.972 --> 10:01.812
and the class name is METSD MR task that
is a class name and input file that is a

10:01.812 --> 10:06.192
comments dot XML is available under slash
input slash comments folder and output will be

10:06.192 --> 10:11.092
created under the folder output is a folder
name where the part file will be created

10:11.092 --> 10:16.332
as the output file so we have executed the
command so let me execute the comment here so

10:16.332 --> 10:22.092
we're getting this output I hope yes we
had a had a smooth journey so we got the

10:22.092 --> 10:27.532
output so it has got created under this
input no we shall go for the root then

10:27.532 --> 10:32.932
under the output we're having this
part hyphen r zeros so that is the

10:32.932 --> 10:37.452
respective part file name so let me open
that one so we shall go back to the

10:37.452 --> 10:47.652
terminal once again so we'll be issuing the command is DFS DFS minus
cat and then we shall give the respective path here and then part file

10:47.652 --> 10:52.072
name so go for part start so all files
starting with the part will be printed

10:52.072 --> 10:57.192
but we are having only one file there so that
will be printed here so here you can find

10:57.192 --> 11:02.032
that in the first column we are finding the ID
in the second column I'm finding the median

11:02.032 --> 11:07.652
and the third column I'm finding the respective
standard division so I've got this median

11:07.652 --> 11:14.612
and standard division calculated depending
upon the comment length so I've shown you that

11:14.612 --> 11:19.692
how to do that one let me delete this
output folder not mandatory but we are just

11:19.692 --> 11:23.892
deleting this one so that we can create
other Exxiv the other map produced programs

11:25.152 --> 11:28.292
So now it has got deleted. Thanks for watching