Wednesday, 25 February 2015
Computing Correlation Coefficients in R
Correlation Coefficients
The general formulae to compute correlation coefficient between 2 variables is -
where cov(A,B) is the covariance between A & B and SA and SB are the standard deviations.
Manual Way in R
#Define 2 verctors
>
> A <- c(1,2,4,5)
> B <- c(5,6,8,1)
>
> #Finding Covariance between A & B
> A_diff <- A - mean(A)
> B_diff <- B - mean(B)
>
> #Print both the variables created above
> A_diff
[1] -2 -1 1 2
> B_diff
[1] 0 1 3 -4
>
> #Do the summation and divide by N-1 to get the covariance between the two vectors
> #N = 3 in this case
> cov <- sum(A_diff*B_diff)/(3-1)
>
> #Finding the squared difference w.r.t to mean for the vectors
> A_sq <- A_diff^2
> B_sq <- B_diff^2
>
> #Using the standard deviation formulae
>
> A_sd <- sqrt(sum(A_sq)/(3-1))
>
> B_sd <- sqrt(sum(B_sq)/(3-1))
>
> #Print the standard deviation
> A_sd
[1] 2.236068
> B_sd
[1] 3.605551
>
> #Plugging in values to find the correlation coefficient
>
> corr <- cov/(A_sd*B_sd)
>
> #Printing the correlation obtained - Manual way
> corr
[1] -0.3721042
>
> #Using formulae for direct computation
>
> corr_test <- cor(A,B)
> corr_test
[1] -0.3721042
Using the in-built function and manual way, we get the same result.
Tuesday, 3 February 2015
All about Correlation - Part - 2
Is the correlation significant ?
Note:
This post is in continuation to the post All about correlation - Part - 1
We are using the mtdata set in R.
This post is in continuation to the post All about correlation - Part - 1
We are using the mtdata set in R.
Now that you have computed the correlation between 2 variables, you need to decide if the value is significant. In other words, the null hypothesis states -
Ho : The correlation is zero
H1: There is significant correlation
In R, we can test this using the function cor.test. This function does the test of association between paired samples, using one of Pearson's product moment correlation coefficient. Neither cor(), nor cov() produce tests of significance, but cor.test() function can be used to test a single correlation coefficient.
R Code
Thus, the sample should always be representative of the population. Any kind of sub-setting can change the correlation that is derived as such.
H1: There is significant correlation
In R, we can test this using the function cor.test. This function does the test of association between paired samples, using one of Pearson's product moment correlation coefficient. Neither cor(), nor cov() produce tests of significance, but cor.test() function can be used to test a single correlation coefficient.
R Code
Scatterplots
R Code
Interpreting Correlations
At times, non representative samples can cause wrong interpretation of the correlations. Thus, it is important to test out this effect by sub-setting the data.
In the mtcars datset, we have transmission column. It is a categorical variable, which is set to 0 for no transmission and 1 for transmission.
If we make subsets of our data and then calculate the correlations again, we will observe there will be a difference. Thus, our sample may not be a good representative of the population.
R Code
R Code
> #Needed for describeBy function > library("psych") > > #To read the first few lines of the datset > head(mtcars) mpg cyl disp hp drat wt qsec vs am gear carb Mazda RX4 21.0 6 160 110 3.90 2.620 16.46 0 1 4 4 Mazda RX4 Wag 21.0 6 160 110 3.90 2.875 17.02 0 1 4 4 Datsun 710 22.8 4 108 93 3.85 2.320 18.61 1 1 4 1 Hornet 4 Drive 21.4 6 258 110 3.08 3.215 19.44 1 0 3 1 Hornet Sportabout 18.7 8 360 175 3.15 3.440 17.02 0 0 3 2 Valiant 18.1 6 225 105 2.76 3.460 20.22 1 0 3 1 > > #Subseting the data > subset_trans <- subset(mtcars, am == 1) > subset_notrans <- subset(mtcars, am == 0) > > #print the subsetted data > head(subset_trans) mpg cyl disp hp drat wt qsec vs am gear carb Mazda RX4 21.0 6 160.0 110 3.90 2.620 16.46 0 1 4 4 Mazda RX4 Wag 21.0 6 160.0 110 3.90 2.875 17.02 0 1 4 4 Datsun 710 22.8 4 108.0 93 3.85 2.320 18.61 1 1 4 1 Fiat 128 32.4 4 78.7 66 4.08 2.200 19.47 1 1 4 1 Honda Civic 30.4 4 75.7 52 4.93 1.615 18.52 1 1 4 2 Toyota Corolla 33.9 4 71.1 65 4.22 1.835 19.90 1 1 4 1 > head(subset_notrans) mpg cyl disp hp drat wt qsec vs am gear carb Hornet 4 Drive 21.4 6 258.0 110 3.08 3.215 19.44 1 0 3 1 Hornet Sportabout 18.7 8 360.0 175 3.15 3.440 17.02 0 0 3 2 Valiant 18.1 6 225.0 105 2.76 3.460 20.22 1 0 3 1 Duster 360 14.3 8 360.0 245 3.21 3.570 15.84 0 0 3 4 Merc 240D 24.4 4 146.7 62 3.69 3.190 20.00 1 0 4 2 Merc 230 22.8 4 140.8 95 3.92 3.150 22.90 1 0 4 2 > > #Using the describeBy function, need to install the psych package for this > describeBy(mtcars,mtcars$am) group: 0 vars n mean sd median trimmed mad min max range skew kurtosis se mpg 1 19 17.15 3.83 17.30 17.12 3.11 10.40 24.40 14.00 0.01 -0.80 0.88 cyl 2 19 6.95 1.54 8.00 7.06 0.00 4.00 8.00 4.00 -0.95 -0.74 0.35 disp 3 19 290.38 110.17 275.80 289.71 124.83 120.10 472.00 351.90 0.05 -1.26 25.28 hp 4 19 160.26 53.91 175.00 161.06 77.10 62.00 245.00 183.00 -0.01 -1.21 12.37 drat 5 19 3.29 0.39 3.15 3.28 0.22 2.76 3.92 1.16 0.50 -1.30 0.09 wt 6 19 3.77 0.78 3.52 3.75 0.45 2.46 5.42 2.96 0.98 0.14 0.18 qsec 7 19 18.18 1.75 17.82 18.07 1.19 15.41 22.90 7.49 0.85 0.55 0.40 vs 8 19 0.37 0.50 0.00 0.35 0.00 0.00 1.00 1.00 0.50 -1.84 0.11 am 9 19 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 NaN NaN 0.00 gear 10 19 3.21 0.42 3.00 3.18 0.00 3.00 4.00 1.00 1.31 -0.29 0.10 carb 11 19 2.74 1.15 3.00 2.76 1.48 1.00 4.00 3.00 -0.14 -1.57 0.26 -------------------------------------------------------------------------------- group: 1 vars n mean sd median trimmed mad min max range skew kurtosis se mpg 1 13 24.39 6.17 22.80 24.38 6.67 15.00 33.90 18.90 0.05 -1.46 1.71 cyl 2 13 5.08 1.55 4.00 4.91 0.00 4.00 8.00 4.00 0.87 -0.90 0.43 disp 3 13 143.53 87.20 120.30 131.25 58.86 71.10 351.00 279.90 1.33 0.40 24.19 hp 4 13 126.85 84.06 109.00 114.73 63.75 52.00 335.00 283.00 1.36 0.56 23.31 drat 5 13 4.05 0.36 4.08 4.02 0.27 3.54 4.93 1.39 0.79 0.21 0.10 wt 6 13 2.41 0.62 2.32 2.39 0.68 1.51 3.57 2.06 0.21 -1.17 0.17 qsec 7 13 17.36 1.79 17.02 17.39 2.34 14.50 19.90 5.40 -0.23 -1.42 0.50 vs 8 13 0.54 0.52 1.00 0.55 0.00 0.00 1.00 1.00 -0.14 -2.13 0.14 am 9 13 1.00 0.00 1.00 1.00 0.00 1.00 1.00 0.00 NaN NaN 0.00 gear 10 13 4.38 0.51 4.00 4.36 0.00 4.00 5.00 1.00 0.42 -1.96 0.14 carb 11 13 2.92 2.18 2.00 2.64 1.48 1.00 8.00 7.00 0.98 -0.21 0.60 > > #Finding correlation in both the groups > corr <- cor(mtcars$mpg,mtcars$wt) > corr [1] -0.8676594 > corr_trans <- cor(subset_trans$mpg,subset_trans$wt) > corr_trans [1] -0.9089148 > corr_nontrans <- cor(subset_notrans$mpg,subset_notrans$wt) > corr_nontrans [1] -0.7676554 > #Combine all the results > Correlation <- cbind(corr,corr_trans,corr_nontrans) > # Notice the difference in correlation for the same set of variables but differently grouped/ subsetted > Correlation corr corr_trans corr_nontrans [1,] -0.8676594 -0.9089148 -0.7676554
Thus, the sample should always be representative of the population. Any kind of sub-setting can change the correlation that is derived as such.
Sunday, 1 February 2015
Data manipulation in R
DPLYR Package Basics
Dplyr is a very powerful package for data manipulation in R. The 6 main verbs for dplyr are -
1. Select
2. Filter
3. Arrange
4. Group
5. Mutate
6. Summarise
Together, these verbs can be used to perform powerful data analysis.
Select
All you need to know about select function.
Mutate
Mutate is used to create new columns. You can always reuse variables created within mutate function to create more columns. Always remember that the new columns are created to a copy of the dataset.
Let's write a code to create 2 new columns - TotalTaxiTime, GroundTime
Dplyr is a very powerful package for data manipulation in R. The 6 main verbs for dplyr are -
1. Select
2. Filter
3. Arrange
4. Group
5. Mutate
6. Summarise
Together, these verbs can be used to perform powerful data analysis.
Select
All you need to know about select function.
#Shown below are 2 ways to select the set of columns. Notice, how easy it becomes when we make use of the select function. The "%>%" is the piping operator and is of great use when longer codes need to be written.
> #Difficult way > hflights[c("Year","Month","DayOfWeek","DepTime","ArrTime")] Source: local data frame [227,496 x 5] Year Month DayOfWeek DepTime ArrTime 1 2011 1 6 1400 1500 2 2011 1 7 1401 1501 3 2011 1 1 1352 1502 4 2011 1 2 1403 1513 5 2011 1 3 1405 1507 6 2011 1 4 1359 1503 7 2011 1 5 1359 1509 8 2011 1 6 1355 1454 9 2011 1 7 1443 1554 10 2011 1 1 1443 1553 .. ... ... ... ... ... > #Easy Way > hflights %>% select(Year:ArrTime, - DayofMonth)
Mutate
Mutate is used to create new columns. You can always reuse variables created within mutate function to create more columns. Always remember that the new columns are created to a copy of the dataset.
Let's write a code to create 2 new columns - TotalTaxiTime, GroundTime
#Creating new columns hflights %>% mutate(TotalTaxiTime = TaxiIn+ TaxiOut, GroundTime = ActualElapsedTime - ArrTime)
Subscribe to:
Posts (Atom)