WEBVTT

00:00.000 --> 00:02.000
Welcome to the DataHolly2 class

00:02.000 --> 00:06.000
In today's video, we are
going to discuss how do we use

00:06.000 --> 00:12.000
the drop and the keep command. So why do
we use these two commands? We can use

00:12.000 --> 00:20.000
these commands to either drop variables
or keep the variables or we can use

00:20.000 --> 00:28.000
them to drop values within certain
variables. So they both work somewhat

00:28.000 --> 00:36.000
similarly except for because they are two
things of the same kind so I will say either

00:36.000 --> 00:40.000
you can drop certain variables or
you can keep them. So let's have the

00:40.000 --> 00:46.000
data set and we are using this same
auto data set that comes preloaded with

00:46.000 --> 00:50.000
Stata. So firstly we are going to
discuss how do we keep or drop

00:50.000 --> 00:54.000
variables. So if we want to drop
a variable like let's say there is

00:54.000 --> 01:00.000
a variable called make or model
of a car so what we can do is

01:00.000 --> 01:04.000
use the drop command and then
the name of the variable and that

01:04.000 --> 01:10.000
would drop or delete that
variable. We can use, we can delete

01:10.000 --> 01:16.000
multiple variables by just specifying
the drop command and then the list of

01:16.000 --> 01:20.000
the variables. So in this case, I want
to drop all these three variables

01:20.000 --> 01:26.000
mpg, the pair variable and the
date variable. So that would drop

01:26.000 --> 01:34.000
all these. We can also use some
kind of this write command where

01:34.000 --> 01:40.000
we have let's say certain variables
that come in a SQL. For example

01:40.000 --> 01:46.000
we have price, price1, price2 and so
on and so forth. These all variables

01:46.000 --> 01:56.000
have some kind of a subscript or rather
suffix, right and we can use that suffix

01:56.000 --> 02:02.000
along with the write command to delete all
those variables that start with a certain

02:02.000 --> 02:08.000
character. So let's just say let's create
a variable let's call it price2 and

02:08.000 --> 02:12.000
I'm just going to generate this variable
for demonstration purpose. We don't need

02:12.000 --> 02:18.000
to generate them ourselves. So now we have
two variables called price and price2

02:18.000 --> 02:22.000
and let's just say if I wanted
to drop them both, I do

02:22.000 --> 02:30.000
not need to specify drop and then
let's just say price and then price 2

02:30.000 --> 02:38.000
I can simply write it using a while card
because both these variables have a common

02:38.000 --> 02:44.000
character which is price and what I can
do is I can write that common character

02:44.000 --> 02:50.000
and then specify a while card. So that
will drop both the price variables

02:50.000 --> 02:54.000
Similarly let's just say we have
two variables that start with T

02:54.000 --> 02:58.000
and I want to drop both those
variables or all of those variables

02:58.000 --> 03:04.000
that start with T the alphabet T.
So I'm using the alphabet T and

03:04.000 --> 03:10.000
the while card so that would help that
to drop any variable that starts with

03:10.000 --> 03:16.000
a T. Similarly we can instead of using
drop we can use keep. So keep would

03:16.000 --> 03:20.000
obviously keep those variables and not
delete them and delete the rest of them

03:20.000 --> 03:26.000
So we have around 5 variables and
let's just say we just need all

03:26.000 --> 03:30.000
of them we just need for our
analysis we just need length

03:30.000 --> 03:34.000
and the following variables. So I'm going
to keep the length and the following

03:34.000 --> 03:40.000
variables. So that would just keep those
two variables. You can only keep a single

03:40.000 --> 03:44.000
variable and just say specify keep
and just the name of that variable

03:44.000 --> 03:48.000
and that would keep that specific
value. So should we use drop or

03:48.000 --> 03:54.000
should we use keep? I mean they both
work, they both perform the same job

03:54.000 --> 04:00.000
except for a different angle. So the
idea is if you want to drop a few

04:00.000 --> 04:04.000
variables then you should use drop
and if you want to keep a few

04:04.000 --> 04:08.000
variables then you should use
keep. So in this case I wanted

04:08.000 --> 04:14.000
to drop some of the variables just three
variables and we had around 15 variables. So

04:14.000 --> 04:18.000
instead of writing the name of all
those 15 variables or all those

04:18.000 --> 04:26.000
13 variables that I wanted to drop it
would have been better to use keep

04:26.000 --> 04:32.000
in that case. So the idea is to
use less number of or less name

04:32.000 --> 04:38.000
of the variables. So use drop if you want
to drop a few variables and use keep

04:38.000 --> 04:44.000
if you want to keep a few variables.
Okay we can also use drop and keep

04:44.000 --> 04:52.000
the same command to drop certain values that
fulfill certain condition. So let me just re-import

04:52.000 --> 04:58.000
the data because I have dropped all the
variables. So let us say I want to drop if

04:58.000 --> 05:04.000
the price of a car is greater than
12,000. So drop all those values. So in

05:04.000 --> 05:10.000
previous case I was dropping the
columns per se. So for example in our

05:10.000 --> 05:16.000
data set the columns represent the variable
name. So previously I was dropping

05:16.000 --> 05:22.000
the columns. But in this case
I want to drop a specific rows

05:22.000 --> 05:28.000
those rows that meet certain
condition. So I want to drop if

05:28.000 --> 05:32.000
price is greater than 12,000. So
that won't drop the price variable

05:32.000 --> 05:38.000
but rather it will drop all
those values or all those rows

05:38.000 --> 05:44.000
that meet certain condition. In this
case price is greater than 12,000

05:44.000 --> 05:52.000
and from the commands below we can see that
once we perform this command we can see that

05:52.000 --> 05:56.000
it shows that 5 observations have been
detected. So there were 5 observations

05:56.000 --> 06:02.000
that had, there were 5 rows that
had price greater than 12,000

06:02.000 --> 06:06.000
But if we wanted to keep all those
variables that have a price less than 10,000

06:06.000 --> 06:14.000
and again we would be keeping rows rather
than keeping the variables. So the idea

06:14.000 --> 06:20.000
is if you specify if and then certain
condition and that would work for the rows

06:20.000 --> 06:24.000
whether that would drop or keep
the rows and if you just specify

06:24.000 --> 06:30.000
the command drop and then variable name that
would drop the columns or the variables

06:30.000 --> 06:38.000
So let's keep all those columns that
have less than 10,000 so we know that

06:38.000 --> 06:46.000
there are 5 observations that were detected. We
can also use multiple conditions. So we can keep

06:46.000 --> 06:50.000
or drop based on multiple conditions.
So let's just say if we wanted to

06:50.000 --> 06:54.000
drop, if the price is greater
than, we want to drop all

06:54.000 --> 07:00.000
those rows, all those cases where
the price is greater than 5000 and

07:00.000 --> 07:06.000
the mileage is equal to 22. So we can use
other logical indications as well. There is

07:06.000 --> 07:12.000
just one car that had price greater than
5000 and its mileage was equal to 22

07:12.000 --> 07:18.000
We can use or condition, right. So
what I am trying to say is that drop

07:18.000 --> 07:24.000
if price is greater than 9000 or, so either
this condition should be met or this condition

07:24.000 --> 07:28.000
should be met. Or the mileage
is equal to 22. So all

07:28.000 --> 07:34.000
those cases, all those rows where
either price is greater than 9000 or

07:34.000 --> 07:40.000
mileage is equal to 22 all those observations
would be dropped. And we know that there are 6

07:40.000 --> 07:48.000
those observations. So we can also drop or keep
missing values. So let's summarize this data

07:48.000 --> 07:54.000
and from the observation column we know that
all those variables, all the variables that we

07:54.000 --> 08:00.000
have in our data set have 15 number of
observations. Except for this repair

08:00.000 --> 08:06.000
variable which have 40, 54 observations.
And if we look at the data, this repair

08:06.000 --> 08:10.000
variable, we know that there are
certain dots in them. And these dots

08:10.000 --> 08:18.000
represent missing values in the language of data. So
if we wanted to drop these missing values, what we

08:18.000 --> 08:24.000
would do is drop if rep 78
is equal to dot. That's how

08:24.000 --> 08:30.000
that would drop these missing values,
right? So we know that there were 3

08:30.000 --> 08:36.000
observations where rep 78 was missing.
And if we can quickly summarize, you would

08:36.000 --> 08:42.000
know that we have only 54
rows as opposed to 57 rows

08:42.000 --> 08:48.000
Or we could have just used drop if this
missing function which simply says that drop

08:48.000 --> 08:56.000
all the rows where the rep 78 contains
certain missing values. So rather than using

08:56.000 --> 09:02.000
the dot explicitly, we can use the missing
function and this one is better for certain

09:02.000 --> 09:06.000
reasons. We can use keep
instead of drop and we can keep

09:06.000 --> 09:12.000
all those values where the rep 78 is
missing or we can use this syntax. They

09:12.000 --> 09:16.000
both work the same way. So
either use this syntax or use

09:16.000 --> 09:22.000
this command. We can drop certain
range and when we say range that means

09:22.000 --> 09:28.000
certain number of tags. So what
we can do is we can write drop in

09:28.000 --> 09:34.000
1 to 10. So that means that
drop the rows that are first

09:34.000 --> 09:40.000
10 rows starting with the first row says
10 from. You would notice that all these

09:40.000 --> 09:48.000
10 rows would be dropped. Right?
We can keep, use the keep

09:48.000 --> 09:54.000
command to just keep the first or the last value
in a parallel data and for that I'm going to

09:54.000 --> 10:00.000
use a different data set. We have a stock
pricing data which contains different

10:00.000 --> 10:06.000
firms. So we have 3 firms and each
firm has different number of years

10:06.000 --> 10:14.000
and we have their prices. What if
I wanted to keep I just wanted to

10:14.000 --> 10:22.000
keep the rows for each firm
where that specific firm has a

10:22.000 --> 10:30.000
highest EFO. In this first firm, this
firm ID 1, the highest year is 2017 but

10:30.000 --> 10:36.000
in firm ID 2 the highest year
is 2015 and in firm ID 3 the

10:36.000 --> 10:40.000
highest year is 2019. We just
want to keep the highest year

10:40.000 --> 10:44.000
So what we do is we use by part
firms. So what we are saying is that

10:44.000 --> 10:50.000
we have a parallel data and we need
to repeat this command for each firm

10:50.000 --> 10:58.000
Now what this command is saying that is
keep if the number of observations is the

10:58.000 --> 11:02.000
highest and we have a video
on this where I will give

11:02.000 --> 11:06.000
a link in the description of the
difference between the small n and the

11:06.000 --> 11:14.000
bigger n. So this should keep 2017 for
the first firm, 2015 for the second

11:14.000 --> 11:22.000
and 2019 for the third firm. That's what
we get. Let me re-import this data and

11:22.000 --> 11:28.000
we will have just the opposite
way that is keep the smallest

11:28.000 --> 11:36.000
number of, smallest year for each
firm. So for firm 1 it will be 2010

11:36.000 --> 11:44.000
for second firm it will be 2005 and for
the first firm the smallest year is 2016

11:44.000 --> 11:48.000
So we do by part firm and
the other we should also

11:48.000 --> 11:54.000
use here in the bracket and what
that will do is it will perform

11:54.000 --> 12:02.000
this command this command for each firm but
before performing this command it should sort

12:02.000 --> 12:06.000
the data in here. So we already
have that sorted but in case

12:06.000 --> 12:10.000
if you don't have the data
sorted, you should do this way

12:10.000 --> 12:18.000
or rather you can explicitly use
sort firm year. So we want this

12:18.000 --> 12:22.000
command to be performed for each firm
but before performing this command

12:22.000 --> 12:30.000
we want stacker to sort the data
based on firm year. So that

12:30.000 --> 12:36.000
should keep the smallest number of
years, smallest year for each firm

12:36.000 --> 12:42.000
We can also drop all the variables,
so in this case we have 3 variables

12:42.000 --> 12:50.000
rather that we go back to
our auto data and we can drop

12:50.000 --> 12:56.000
use this drop underscore all command
and what that will do is it will drop

12:56.000 --> 13:02.000
all the variables. Now this drop all
is different from clear, clear removes

13:02.000 --> 13:08.000
macros or any value labels but drop all
only removes the variable and what I mean

13:08.000 --> 13:12.000
to say is that if we go to
the variable manager, we

13:12.000 --> 13:18.000
would see that, let me create a variable
let's say price is equal to dot

13:18.000 --> 13:26.000
I just created the variable so we
can use this variable manager. Now

13:26.000 --> 13:30.000
you can see that if we go to
a value label we can see that

13:30.000 --> 13:36.000
there was a value label that was
on to our form variable. That

13:36.000 --> 13:42.000
value label is still here. If we had used
clear this value label had been gone

13:42.000 --> 13:48.000
but because we had used drop all, so drop
all just drops the variable, it doesn't

13:48.000 --> 13:54.000
clear the environment. Right. So
let me just quickly use clear and

13:54.000 --> 14:00.000
then let me go back to let me generate this price
variable and go back to the variable manager

14:00.000 --> 14:04.000
and now you see that value
labels have been gone. So clear

14:04.000 --> 14:10.000
would delete everything from the environment,
not just the variables the data but all

14:10.000 --> 14:18.000
the elements that have been created
during the process. We can use

14:18.000 --> 14:24.000
the manuals obviously and that is from
the if you want to drop variables

14:24.000 --> 14:28.000
then you can use data and from
data you can use variable manager

14:28.000 --> 14:36.000
we can click over here for variable manager
if you want to drop observations then

14:36.000 --> 14:40.000
what we can do is we
can we can go to data

14:40.000 --> 14:50.000
create or change data and drop or
keep observations. Right. Lastly

14:50.000 --> 14:58.000
do remember that once you have dropped a
value on an observation we cannot go back

14:58.000 --> 15:04.000
there is no undoing that. So remember to
use either frames or preserve command

15:04.000 --> 15:12.000
before you drop certain variables if you
want to go back to the original data set

15:12.000 --> 15:16.000
Thanks for watching this video. Do
subscribe and do hit the bell icon