Agenda
- Questions
- Data wrangling
- Data types
- Descriptive statistics
- Data visualization
- Grammar of graphics
- Types of graphics
Image source: @allison_horst
One Minute Paper Results
What was the most important thing you learned during this class?
What important question remains unanswered for you?
Types of Data
- Numerical (quantitative)
- Continuous
- Discrete
- Categorical (qualitative)
- Regular categorical
- Ordinal
Data Types in R
Variance
Population Variance: S2=Σ(xi−ˉx)2N Consider a dataset with five values (black points in the figure). For the largest value, the deviance is represented by the blue line ( xi−ˉx ).
See also:
https://shiny.rit.albany.edu/stat/visualizess/
https://github.com/jbryer/VisualStats/
Variance (cont.)
Population Variance: S2=Σ(xi−ˉx)2N In the numerator, we square each of these deviances. We can conceptualize this as a square. Here, we add the deviance in the y direction.
Variance (cont.)
Population Variance: S2=Σ(xi−ˉx)2N
We end up with a square.
Variance (cont.)
Population Variance: S2=Σ(xi−ˉx)2N We can plot the squared deviance for all the data points. That is, each component in the numerator is the area of each of these squares.
Variance (cont.)
Population Variance: S2=Σ(xi−ˉx)2N The variance is therefore the average of the area of all these squares, here represented by the orange square.
Population versus Sample Variance
Typically we want the sample variance. The difference is we divide by n−1 to calculate the sample variance. This results in a slightly larger area (variance) then if we divide by n.
Population Variance (yellow): S2=Σ(xi−ˉx)2N
Sample Variance (green): s2=Σ(xi−ˉx)2n−1
Robust Statistics
Consider the following data randomly selected from the normal distribution:
set.seed(41)x <- rnorm(30, mean = 100, sd = 15)mean(x); sd(x)## [1] 103.1934## [1] 16.8945median(x); IQR(x)## [1] 103.9947## [1] 25.68004
Robust Statistics
Robust Statistics
Let's add an extreme value:
x <- c(x, 1000)Robust Statistics
Let's add an extreme value:
x <- c(x, 1000)
About legosets 
To install the brickset package:
remotes::install_github('jbryer/brickset')To load the load the legosets dataset.
data('legosets', package = 'brickset')The legosets data has 16355 observations of 34 variables.
names(legosets)## [1] "setID" "name" "year" ## [4] "theme" "themeGroup" "subtheme" ## [7] "category" "released" "pieces" ## [10] "minifigs" "bricksetURL" "rating" ## [13] "reviewCount" "packagingType" "availability" ## [16] "agerange_min" "US_retailPrice" "US_dateFirstAvailable"## [19] "US_dateLastAvailable" "UK_retailPrice" "UK_dateFirstAvailable"## [22] "UK_dateLastAvailable" "CA_retailPrice" "CA_dateFirstAvailable"## [25] "CA_dateLastAvailable" "DE_retailPrice" "DE_dateFirstAvailable"## [28] "DE_dateLastAvailable" "height" "width" ## [31] "depth" "weight" "thumbnailURL" ## [34] "imageURL"Structure (str)
str(legosets)## 'data.frame': 16355 obs. of 34 variables:## $ setID : int 7693 7695 7697 7698 25534 7418 7419 6020 22704 7421 ...## $ name : chr "Small house set" "Medium house set" "Medium house set" "Large house set" ...## $ year : int 1970 1970 1970 1970 1970 1970 1970 1970 1970 1970 ...## $ theme : chr "Minitalia" "Minitalia" "Minitalia" "Minitalia" ...## $ themeGroup : chr "Vintage" "Vintage" "Vintage" "Vintage" ...## $ subtheme : chr NA NA NA NA ...## $ category : chr "Normal" "Normal" "Normal" "Normal" ...## $ released : logi TRUE TRUE TRUE TRUE TRUE TRUE ...## $ pieces : int 67 109 158 233 NA 1 1 60 65 NA ...## $ minifigs : int NA NA NA NA NA NA NA NA NA NA ...## $ bricksetURL : chr "https://brickset.com/sets/1-8" "https://brickset.com/sets/2-8" "https://brickset.com/sets/3-6" "https://brickset.com/sets/4-4" ...## $ rating : num 0 0 0 0 0 0 0 0 0 0 ...## $ reviewCount : int 0 0 1 0 0 0 0 1 0 0 ...## $ packagingType : chr "{Not specified}" "{Not specified}" "{Not specified}" "{Not specified}" ...## $ availability : chr "{Not specified}" "{Not specified}" "{Not specified}" "{Not specified}" ...## $ agerange_min : int NA NA NA NA NA NA NA NA NA NA ...## $ US_retailPrice : num NA NA NA NA NA 1.99 NA NA 4.99 NA ...## $ US_dateFirstAvailable: Date, format: NA NA ...## $ US_dateLastAvailable : Date, format: NA NA ...## $ UK_retailPrice : num NA NA NA NA NA NA NA NA NA NA ...## $ UK_dateFirstAvailable: Date, format: NA NA ...## $ UK_dateLastAvailable : Date, format: NA NA ...## $ CA_retailPrice : num NA NA NA NA NA NA NA NA NA NA ...## $ CA_dateFirstAvailable: Date, format: NA NA ...## $ CA_dateLastAvailable : Date, format: NA NA ...## $ DE_retailPrice : num NA NA NA NA NA NA NA NA NA NA ...## $ DE_dateFirstAvailable: Date, format: NA NA ...## $ DE_dateLastAvailable : Date, format: NA NA ...## $ height : num NA NA NA NA NA ...## $ width : num NA NA NA NA NA ...## $ depth : num NA NA NA NA NA NA NA NA 5.08 NA ...## $ weight : num NA NA NA NA NA NA NA NA NA NA ...## $ thumbnailURL : chr "https://images.brickset.com/sets/small/1-8.jpg" "https://images.brickset.com/sets/small/2-8.jpg" "https://images.brickset.com/sets/small/3-6.jpg" "https://images.brickset.com/sets/small/4-4.jpg" ...## $ imageURL : chr "https://images.brickset.com/sets/images/1-8.jpg" "https://images.brickset.com/sets/images/2-8.jpg" "https://images.brickset.com/sets/images/3-6.jpg" "https://images.brickset.com/sets/images/4-4.jpg" ...RStudio Eenvironment tab can help 
Data Wrangling Cheat Sheet 
Tidyverse vs Base R 
Pipes %>% 
The pipe operator (%>%) introduced with the magrittr R package allows for the chaining of R operations. It takes the output from the left-hand side and passes it as the first parameter to the function on the right-hand side. In base R, to get the output of a proportional table, you need to first call table then prop.table.
You can do this in two steps:
tab_out <- table(legosets$category)prop.table(tab_out)Or as nested function calls.
prop.table(table(legosets$category))Using the pipe (%>%) operator we can chain these calls in a what is arguably a more readable format:
table(legosets$category) %>% prop.table()## ## Book Collection Extended Gear Normal Other ## 0.028798533 0.032100275 0.025191073 0.143564659 0.713420972 0.054050749 ## Random ## 0.002873739Filter
Filter
dplyr
mylego <- legosets %>% filter(themeGroup == 'Educational' & year > 2015)Base R
mylego <- legosets[legosets$themeGroups == 'Educaitonal' & legosets$year > 2015,]nrow(mylego)## [1] 61Select
dplyr
mylego <- mylego %>% select(setID, pieces, theme, availability, US_retailPrice, minifigs)Base R
mylego <- mylego[,c('setID', 'pieces', 'theme', 'availability', 'US_retailPrice', 'minifigs')]head(mylego, n = 4)## setID pieces theme availability US_retailPrice minifigs## 1 26803 103 Education {Not specified} NA 6## 2 26689 142 Education {Not specified} NA 4## 3 26804 98 Education {Not specified} NA 6## 4 26277 188 Education Educational 78.95 NARelocate
Relocate
dplyr
mylego %>% relocate(where(is.numeric), .after = where(is.character)) %>% head(n = 3)## theme availability setID pieces US_retailPrice minifigs## 1 Education {Not specified} 26803 103 NA 6## 2 Education {Not specified} 26689 142 NA 4## 3 Education {Not specified} 26804 98 NA 6Base R
mylego2 <- mylego[,c('theme', 'availability', 'setID', 'pieces', 'US_retailPrice', 'minifigs')]head(mylego2, n = 3)## theme availability setID pieces US_retailPrice minifigs## 1 Education {Not specified} 26803 103 NA 6## 2 Education {Not specified} 26689 142 NA 4## 3 Education {Not specified} 26804 98 NA 6Rename
Rename
dplyr
mylego %>% dplyr::rename(USD = US_retailPrice) %>% head(n = 3)## setID pieces theme availability USD minifigs## 1 26803 103 Education {Not specified} NA 6## 2 26689 142 Education {Not specified} NA 4## 3 26804 98 Education {Not specified} NA 6Base R
names(mylego2)[5] <- 'USD'head(mylego2, n = 3)## theme availability setID pieces USD minifigs## 1 Education {Not specified} 26803 103 NA 6## 2 Education {Not specified} 26689 142 NA 4## 3 Education {Not specified} 26804 98 NA 6Mutate
Mutate
dplyr
mylego %>% filter(!is.na(pieces) & !is.na(US_retailPrice)) %>% mutate(Price_per_piece = US_retailPrice / pieces) %>% head(n = 3)## setID pieces theme availability US_retailPrice minifigs Price_per_piece## 1 26277 188 Education Educational 78.95 NA 0.4199468## 2 25949 280 Education Educational 224.95 NA 0.8033929## 3 25954 1 Education Educational 14.95 NA 14.9500000Base R
mylego2 <- mylego[!is.na(mylego$US_retailPrice) & !is.na(mylego$Price_per_piece),]mylego2$Price_per_piece <- mylego2$Price_per_piece / mylego2$US_retailPricehead(mylego2, n = 3)## [1] setID pieces theme availability ## [5] US_retailPrice minifigs Price_per_piece## <0 rows> (or 0-length row.names)Group By and Summarize
legosets %>% group_by(themeGroup) %>% summarize(mean_price = mean(US_retailPrice, na.rm = TRUE), sd_price = sd(US_retailPrice, na.rm = TRUE), median_price = median(US_retailPrice, na.rm = TRUE), n = n(), missing = sum(is.na(US_retailPrice)))## # A tibble: 15 × 6## themeGroup mean_price sd_price median_price n missing## <chr> <dbl> <dbl> <dbl> <int> <int>## 1 Action/Adventure 31.3 29.9 20.0 1280 462## 2 Basic 13.1 12.8 7.99 843 473## 3 Constraction 15.1 14.0 9.99 501 125## 4 Educational 89.0 107. 59.7 452 294## 5 Girls 23.4 22.6 15.0 677 225## 6 Historical 25.5 27.7 15.0 473 125## 7 Junior 18.6 13.2 17.8 228 93## 8 Licensed 42.9 58.3 25.0 2060 467## 9 Miscellaneous 14.3 20.8 6.99 4925 2117## 10 Model making 52.8 65.1 30.0 582 166## 11 Modern day 31.2 33.7 20.0 1723 763## 12 Pre-school 23.8 19.4 20.0 1487 699## 13 Racing 24.8 30.2 10 270 59## 14 Technical 60.8 68.1 40.0 550 137## 15 Vintage 9.71 9.56 7.50 304 264Grammer of Graphics
Data Visualizations with ggplot2 
ggplot2is an R package that provides an alternative framework based upon Wilkinson’s (2005) Grammar of Graphics.ggplot2is, in general, more flexible for creating "prettier" and complex plots.Works by creating layers of different types of objects/geometries (i.e. bars, points, lines, polygons, etc.)
ggplot2has at least three ways of creating plots:qplotggplot(...) + geom_XXX(...) + ...ggplot(...) + layer(...)
We will focus only on the second.
Parts of a ggplot2 Statement
Data
ggplot(myDataFrame, aes(x=x, y=y))Layers
geom_point(),geom_histogram()Facets
facet_wrap(~ cut),facet_grid(~ cut)Scales
scale_y_log10()Other options
ggtitle('my title'),ylim(c(0, 10000)),xlab('x-axis label')
Lots of geoms
ls('package:ggplot2')[grep('^geom_', ls('package:ggplot2'))]## [1] "geom_abline" "geom_area" "geom_bar" ## [4] "geom_bin_2d" "geom_bin2d" "geom_blank" ## [7] "geom_boxplot" "geom_col" "geom_contour" ## [10] "geom_contour_filled" "geom_count" "geom_crossbar" ## [13] "geom_curve" "geom_density" "geom_density_2d" ## [16] "geom_density_2d_filled" "geom_density2d" "geom_density2d_filled" ## [19] "geom_dotplot" "geom_errorbar" "geom_errorbarh" ## [22] "geom_freqpoly" "geom_function" "geom_hex" ## [25] "geom_histogram" "geom_hline" "geom_jitter" ## [28] "geom_label" "geom_line" "geom_linerange" ## [31] "geom_map" "geom_path" "geom_point" ## [34] "geom_pointrange" "geom_polygon" "geom_qq" ## [37] "geom_qq_line" "geom_quantile" "geom_raster" ## [40] "geom_rect" "geom_ribbon" "geom_rug" ## [43] "geom_segment" "geom_sf" "geom_sf_label" ## [46] "geom_sf_text" "geom_smooth" "geom_spoke" ## [49] "geom_step" "geom_text" "geom_tile" ## [52] "geom_violin" "geom_vline"Data Visualization Cheat Sheet
Scatterplot
ggplot(legosets, aes(x=pieces, y=US_retailPrice)) + geom_point()
Scatterplot (cont.)
ggplot(legosets, aes(x=pieces, y=US_retailPrice, color=availability)) + geom_point()
Scatterplot (cont.)
ggplot(legosets, aes(x=pieces, y=US_retailPrice, size=minifigs, color=availability)) + geom_point()
Scatterplot (cont.)
ggplot(legosets, aes(x=pieces, y=US_retailPrice, size=minifigs)) + geom_point() + facet_wrap(~ availability)
Boxplots
ggplot(legosets, aes(x='Lego', y=US_retailPrice)) + geom_boxplot()
Boxplots (cont.)
ggplot(legosets, aes(x=availability, y=US_retailPrice)) + geom_boxplot()
Boxplot (cont.)
ggplot(legosets, aes(x=availability, y=US_retailPrice)) + geom_boxplot() + coord_flip()
Histograms
ggplot(legosets, aes(x = US_retailPrice)) + geom_histogram()
Histograms (cont.)
ggplot(legosets, aes(x = US_retailPrice)) + geom_histogram() + scale_x_log10()
Histograms (cont.)
ggplot(legosets, aes(x = US_retailPrice)) + geom_histogram() + facet_wrap(~ availability)
Density Plots
ggplot(legosets, aes(x = US_retailPrice, color = availability)) + geom_density()
ggplot2 aesthetics
Likert Scales 
Likert scales are a type of questionnaire where respondents are asked to rate items on scales usually ranging from four to seven levels (e.g. strongly disagree to strongly agree).
library(likert)library(reshape)data(pisaitems)items24 <- pisaitems[,substr(names(pisaitems), 1,5) == 'ST24Q']items24 <- rename(items24, c( ST24Q01="I read only if I have to.", ST24Q02="Reading is one of my favorite hobbies.", ST24Q03="I like talking about books with other people.", ST24Q04="I find it hard to finish books.", ST24Q05="I feel happy if I receive a book as a present.", ST24Q06="For me, reading is a waste of time.", ST24Q07="I enjoy going to a bookstore or a library.", ST24Q08="I read only to get information that I need.", ST24Q09="I cannot sit still and read for more than a few minutes.", ST24Q10="I like to express my opinions about books I have read.", ST24Q11="I like to exchange books with my friends."))likert R Package
l24 <- likert(items24)summary(l24)## Item low neutral## 10 I like to express my opinions about books I have read. 41.07516 0## 5 I feel happy if I receive a book as a present. 46.93475 0## 8 I read only to get information that I need. 50.39874 0## 7 I enjoy going to a bookstore or a library. 51.21231 0## 3 I like talking about books with other people. 54.99129 0## 11 I like to exchange books with my friends. 55.54115 0## 2 Reading is one of my favorite hobbies. 56.64470 0## 1 I read only if I have to. 58.72868 0## 4 I find it hard to finish books. 65.35125 0## 9 I cannot sit still and read for more than a few minutes. 76.24524 0## 6 For me, reading is a waste of time. 82.88729 0## high mean sd## 10 58.92484 2.604913 0.9009968## 5 53.06525 2.466751 0.9446590## 8 49.60126 2.484616 0.9089688## 7 48.78769 2.428508 0.9164136## 3 45.00871 2.328049 0.9090326## 11 44.45885 2.343193 0.9609234## 2 43.35530 2.344530 0.9277495## 1 41.27132 2.291811 0.9369023## 4 34.64875 2.178299 0.8991628## 9 23.75476 1.974736 0.8793028## 6 17.11271 1.810093 0.8611554likert Plots
plot(l24)
likert Plots
plot(l24, type='heat')
likert Plots
plot(l24, type='density')
Dual Scales 
Some problems1:
- The designer has to make choices about scales and this can have a big impact on the viewer
- "Cross-over points” where one series cross another are results of the design choices, not intrinsic to the data, and viewers (particularly unsophisticated viewers)
- They make it easier to lazily associate correlation with causation, not taking into account autocorrelation and other time-series issues
- Because of the issues above, in malicious hands they make it possible to deliberately mislead
This example looks at the relationship between NZ dollar exchange rate and trade weighted index.
DATA606::shiny_demo('DualScales', package='DATA606')My advise:
- Avoid using them. You can usually do better with other plot types.
- When necessary (or compelled) to use them, rescale (using z-scores, we'll discuss this in a few weeks)
Pie Charts
There is only one pie chart in OpenIntro Statistics (Diez, Barr, & Çetinkaya-Rundel, 2015, p. 48). Consider the following three pie charts that represent the preference of five different colors. Is there a difference between the three pie charts? This is probably a difficult to answer.
Pie Charts
There is only one pie chart in OpenIntro Statistics (Diez, Barr, & Çetinkaya-Rundel, 2015, p. 48). Consider the following three pie charts that represent the preference of five different colors. Is there a difference between the three pie charts? This is probably a difficult to answer.
