## 4.2 Introduction

First we need to load up a package to make loading data sets easier. We will be using the tidyverse set of packages for all of our data processing needs in R.1

We first will load up the tidyverse packages using the library function.

library(tidyverse)
── Attaching packages ─────────────────────────────────────── tidyverse 1.3.1 ──
✔ ggplot2 3.3.3     ✔ purrr   0.3.4
✔ tibble  3.1.1     ✔ dplyr   1.0.5
✔ tidyr   1.1.3     ✔ stringr 1.4.0
✔ readr   1.4.0     ✔ forcats 0.5.1
── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::lag()    masks stats::lag()

The first time you load up the tidyverse library, there will be some output that lists the packages tidyverse loads, along with any functions that share the same name as other functions (i.e., conflicts). As long as you are mindful about starting a new RStudio session before you work, you can ignore this output for now.

Now we can use all the functions within the Tidyverse to do our data processing. If you are following along and you run a piece of code and end up with an could not find function error, make sure you typed library(tidyverse) correctly and executed the line of code.

Let the below figures represent a ds4biomed folder on your Desktop on Windows and Mac, respectively.

C:\
|- Users\
|- Daniel\
|- Desktop\
|- ds4biomed\
|- data\
|    |- medicaldata_tumorgrowth.csv
|- output\
|- ds4biomed.Rproj
/
|- Users/
|- Daniel/
|- Desktop/
|- ds4biomed/
|- data/
|    |- medicaldata_tumorgrowth.csv
|- output/
|- ds4biomed.Rproj

Suppose we are “in” the ds4biomed folder, where we can see the data and output folders along with the 01-load_data.R, README.md, and ds4biomed.Rproj files.

We can reference our medicaldata_tumorgrowth.csv file inside the data folder as: data/medicaldata_tumorgrowth.csv. That is, we can use the backslash, / to move into folders. We can write data/medicaldata_tumorgrowth.csv because we are “starting from” the ds4biomed folder. This is called a relative path because the location of the csv file is relative to the ds4biomed starting point (aka working directory). If we want to refer to any arbitrary filer or folder on the computer, we can specify the full path of the file. The full path will start with a drive letter on windows, C:\Users\Daniel\Desktop\ds4biomed\data\medicaldata_tumorgrowth.csv, and a / on a Mac, /Users/Daniel/Desktop/ds4biomed/data/medicaldata_tumorgrowth.csv.

Suppose we create an analysis folder for our 01-load_data.R script so that our folder structure looks like this (only the mac version is shown in the example below):

/
|- Users/
|- Daniel/
|- Desktop/
|- ds4biomed/
|- data/
|    |- medicaldata_tumorgrowth.csv
|- output/
|- analysis
|- ds4biomed.Rproj

Now, if our working directory is now in the analysis folder, we need a way to reference one folder up to the ds4biomed folder and then back down to the data folder. The way we can relatively reference the previous folder is with 2 dots, .., ../data/medicaldata_tumorgrowth.csv

Given the following example folder structure where we have a chart_review and rct_m22-0305 folder in our projects folder:

/
|- Users/
|- Daniel/
|- projects/
|-chart_review/
|    |- data/
|    |    |- patients.csv
|    |- analysis/
|         |- demographics.R # you are working here
|- rct_m22-0305
|- data/
|- patients.csv

Let’s say we are currently in the chart_review/analysis folder, working on our demographics.R as denoted by the #.

1. Write the relative path to the patients.csv file in the rct_m22-0305 folder.
2. Write the absolute path to the patients.csv file in the chart_review folder.

### 4.3.1 Paths in Windows

When you are looking at file paths in the Windows Explorer, you will notice that all Windows paths will use the backslash, \, instead of the forward slash, / to refer to files. In a lot of programming languages, including R, the \ is a special character, so if you want to use \ for file paths in Windows, you will have to use 2 backslashes, e.g., ..\\data\\patients.csv\\. However, you can still use the regular / in Windows to refer to folders just like other operating systems.

## 4.4 Set your working directory

So far, we have been talking about a “starting point” or “working directory,” when we have been referring to files around our computer. In order to quickly and reliably set your working directory, we use RStudio Projects (Section 3.5).

## 4.5 Reading text files (CSV)

Now that we know how to find our files, let’s load up our first data set. When trying to type in a file path, you can hit the <TAB> key to autocomplete the files. This will help you with a lot of potential spelling mistakes.

read_csv("data/medicaldata_tumorgrowth.csv")

── Column specification ────────────────────────────────────────────────────────
cols(
Grp = col_character(),
Group = col_double(),
ID = col_double(),
Day = col_double(),
Size = col_double()
)
# A tibble: 574 x 5
Grp   Group    ID   Day   Size
<chr> <dbl> <dbl> <dbl>  <dbl>
1 1.CTR     1   101     0   41.8
2 1.CTR     1   101     3   85
3 1.CTR     1   101     4  114
4 1.CTR     1   101     5  162.
5 1.CTR     1   101     6  178.
6 1.CTR     1   101     7  325
7 1.CTR     1   101    10  624.
8 1.CTR     1   101    11  648.
9 1.CTR     1   101    12  836.
10 1.CTR     1   101    13 1030.
# … with 564 more rows

Debug help: - If the above code returns a could not find function "read_csv" make sure you have loaded up the proper library with library(tidyverse) - If the above code returns a does not exist in current working directory, make sure the working directory it lists is your expected “starting point” (i.e., working directory), and make sure the file path is spelled correctly.

read_csv will show us the columns that were read in, as well as the data type of that column (e.g., character, double – a number).

Loading a data set is great, but we need a convenient way to refer to the data set. We don’t want to re-load the data set every time we want to perform an action on it. We can take this loaded data set and assign it to a variable. We can do this with the assignment operator, <-. Note the way it is typed, a less than symbol (<) followed immediately by the dash (-) without any spaces in between. The right side of the assignment operator, <-, will be executed and then assigned to the variable on the left.

tumor <- read_csv("data/medicaldata_tumorgrowth.csv")

── Column specification ────────────────────────────────────────────────────────
cols(
Grp = col_character(),
Group = col_double(),
ID = col_double(),
Day = col_double(),
Size = col_double()
)

Notice this time we no longer see the dataset being printed. The “Environment” tab in the RStudio panel will now have an entry for the variable you used. Clicking on the right data set icon will open a view of your dataset, clicking on the arrow will show you the column-by-column text representation (technically it’s called the structure).

To look at our dataset we can execute just the variable we assigned the dataset to.

tumor
# A tibble: 574 x 5
Grp   Group    ID   Day   Size
<chr> <dbl> <dbl> <dbl>  <dbl>
1 1.CTR     1   101     0   41.8
2 1.CTR     1   101     3   85
3 1.CTR     1   101     4  114
4 1.CTR     1   101     5  162.
5 1.CTR     1   101     6  178.
6 1.CTR     1   101     7  325
7 1.CTR     1   101    10  624.
8 1.CTR     1   101    11  648.
9 1.CTR     1   101    12  836.
10 1.CTR     1   101    13 1030.
# … with 564 more rows

This tabular dataset that has now been loaded into R is called a data frame object (or simply dataframe), the tidyverse uses a tibble. For the most part, a data.frame object will behave like a tibble object.

To read an excel file we will need a separate library that can handle Excel files, readxl.

library(readxl)

The readxl library gives us access to the read_excel function which we can use to read in an Excel file.

tumor_xl <- read_excel("data/medicaldata_tumorgrowth.xlsx")

And we can view the loaded Excel file just like before.

tumor_xl
# A tibble: 574 x 5
Grp   Group    ID   Day   Size
<chr> <dbl> <dbl> <dbl>  <dbl>
1 1.CTR     1   101     0   41.8
2 1.CTR     1   101     3   85
3 1.CTR     1   101     4  114
4 1.CTR     1   101     5  162.
5 1.CTR     1   101     6  178.
6 1.CTR     1   101     7  325
7 1.CTR     1   101    10  624.
8 1.CTR     1   101    11  648.
9 1.CTR     1   101    12  836.
10 1.CTR     1   101    13 1030.
# … with 564 more rows

If you are ever lost as to what object you are working with, you can use the class function in R to tell you.

class(tumor)
[1] "spec_tbl_df" "tbl_df"      "tbl"         "data.frame" 
class(tumor_xl)
[1] "tbl_df"     "tbl"        "data.frame"

Once you have see a "tbl_df" or "data.frame" you can be sure that the dataframe processing functions we’ll learn will work.

Take away: Once you load a dataset and end up with either a data.frame or tibble object you can continue using the same data processing methods we cover in these materials.

1. Make sure you have the ds4biomed RStudio project open
2. Load up the exercise-010-050.csv file under data > spreadsheets (you can use the relative path or full path).
3. Save this dataframe object to the variable, spreadsheet_df
spreadsheet_df <- read_csv("data/spreadsheets/exercise-010-050.csv")

── Column specification ────────────────────────────────────────────────────────
cols(
first_name = col_character(),
last_name = col_character(),
time = col_time(format = ""),
diastolic = col_double(),
systolic = col_double(),
heart_rate = col_double(),
respiratory_rate = col_double(),
pulse_ox = col_double(),
temperature = col_double(),
temperature_unit = col_logical()
)

## 4.7 Selecting columns

Here is the tumor dataset we are working with.

tumor
# A tibble: 574 x 5
Grp   Group    ID   Day   Size
<chr> <dbl> <dbl> <dbl>  <dbl>
1 1.CTR     1   101     0   41.8
2 1.CTR     1   101     3   85
3 1.CTR     1   101     4  114
4 1.CTR     1   101     5  162.
5 1.CTR     1   101     6  178.
6 1.CTR     1   101     7  325
7 1.CTR     1   101    10  624.
8 1.CTR     1   101    11  648.
9 1.CTR     1   101    12  836.
10 1.CTR     1   101    13 1030.
# … with 564 more rows

In order to subset the columns of our dataset, we can use the select function. The first argument of the select function is the dataset we want to select columns from. Then, the other arguments are the columns we want to select.

select(tumor, Size)
# A tibble: 574 x 1
Size
<dbl>
1   41.8
2   85
3  114
4  162.
5  178.
6  325
7  624.
8  648.
9  836.
10 1030.
# … with 564 more rows

If we want multiple columns (or to re-order columns) we can specify each column we want selected.

select(tumor, Group, Day, Size)
# A tibble: 574 x 3
Group   Day   Size
<dbl> <dbl>  <dbl>
1     1     0   41.8
2     1     3   85
3     1     4  114
4     1     5  162.
5     1     6  178.
6     1     7  325
7     1    10  624.
8     1    11  648.
9     1    12  836.
10     1    13 1030.
# … with 564 more rows

### 4.7.1 The %>% pipe operator

A common way tidyverse code gets written is using the %>% operator. The %>% takes the object on the left, and passes that into the first argument to the function on the right. We can re-write the above select statements as follows. R interprets the code in the exact same way.

tumor %>%
select(Size)
# A tibble: 574 x 1
Size
<dbl>
1   41.8
2   85
3  114
4  162.
5  178.
6  325
7  624.
8  648.
9  836.
10 1030.
# … with 564 more rows

The way we pronounce the pipe, %>% is “then.” So the above statement is read “tumor, then, select size.”

tumor %>%
select(Group, Day, Size)
# A tibble: 574 x 3
Group   Day   Size
<dbl> <dbl>  <dbl>
1     1     0   41.8
2     1     3   85
3     1     4  114
4     1     5  162.
5     1     6  178.
6     1     7  325
7     1    10  624.
8     1    11  648.
9     1    12  836.
10     1    13 1030.
# … with 564 more rows

The benefits of the pipe may not be as apparent now, but when we start chaining multiple data sets together, it’ll make the code a little more readable.

## 4.8 Filtering rows

Now that we know how to select columns on our dataframe, we can now learn how to filter the rows.

To filter rows, we can use the filter function. The first argument of filter is the dataframe object (the same pattern as the select function), and the other arguments are statements that return some TRUE or FALSE value (this is known as a boolean condition).

For example if we want to filter the tumor dataset such that the Group value is 1 we can filter the rows using that condition.

filter(tumor, Group == 1)
# A tibble: 97 x 5
Grp   Group    ID   Day   Size
<chr> <dbl> <dbl> <dbl>  <dbl>
1 1.CTR     1   101     0   41.8
2 1.CTR     1   101     3   85
3 1.CTR     1   101     4  114
4 1.CTR     1   101     5  162.
5 1.CTR     1   101     6  178.
6 1.CTR     1   101     7  325
7 1.CTR     1   101    10  624.
8 1.CTR     1   101    11  648.
9 1.CTR     1   101    12  836.
10 1.CTR     1   101    13 1030.
# … with 87 more rows

We can write the “pipe” version as such:

tumor %>%
filter(Group == 1)
# A tibble: 97 x 5
Grp   Group    ID   Day   Size
<chr> <dbl> <dbl> <dbl>  <dbl>
1 1.CTR     1   101     0   41.8
2 1.CTR     1   101     3   85
3 1.CTR     1   101     4  114
4 1.CTR     1   101     5  162.
5 1.CTR     1   101     6  178.
6 1.CTR     1   101     7  325
7 1.CTR     1   101    10  624.
8 1.CTR     1   101    11  648.
9 1.CTR     1   101    12  836.
10 1.CTR     1   101    13 1030.
# … with 87 more rows

If we want to combine multiple conditions, where Group is 1 and Day is 0, we can put each statement as a separate parameter into the filter function.

tumor %>%
filter(Group == 1, Day == 0)
# A tibble: 8 x 5
Grp   Group    ID   Day  Size
<chr> <dbl> <dbl> <dbl> <dbl>
1 1.CTR     1   101     0  41.8
2 1.CTR     1   102     0  79.4
3 1.CTR     1   103     0  44.8
4 1.CTR     1   104     0  67.7
5 1.CTR     1   105     0  54.7
6 1.CTR     1   106     0  60
7 1.CTR     1   107     0  46.8
8 1.CTR     1   108     0  49.4

We can also use the & for the and operator.

tumor %>%
filter(Group == 1 & Day == 0)
# A tibble: 8 x 5
Grp   Group    ID   Day  Size
<chr> <dbl> <dbl> <dbl> <dbl>
1 1.CTR     1   101     0  41.8
2 1.CTR     1   102     0  79.4
3 1.CTR     1   103     0  44.8
4 1.CTR     1   104     0  67.7
5 1.CTR     1   105     0  54.7
6 1.CTR     1   106     0  60
7 1.CTR     1   107     0  46.8
8 1.CTR     1   108     0  49.4

Or use the | for the or operator.

tumor %>%
filter(Day == 0 | Day == 13)
# A tibble: 60 x 5
Grp   Group    ID   Day   Size
<chr> <dbl> <dbl> <dbl>  <dbl>
1 1.CTR     1   101     0   41.8
2 1.CTR     1   101    13 1030.
3 1.CTR     1   102     0   79.4
4 1.CTR     1   102    13  619.
5 1.CTR     1   103     0   44.8
6 1.CTR     1   104     0   67.7
7 1.CTR     1   105     0   54.7
8 1.CTR     1   105    13 1699.
9 1.CTR     1   106     0   60
10 1.CTR     1   107     0   46.8
# … with 50 more rows

## 4.9 Subsetting columns and rows

To subset on both columns and rows, we call the corresponding subseting function one after the other. Here we are taking the tumor column, then selecting the Group, Day and Size columns, then filtering the rows where the Size is greater than 2000.

tumor %>%
select(Group, Day, Size) %>%
filter(Size > 2000)
# A tibble: 14 x 3
Group   Day  Size
<dbl> <dbl> <dbl>
1     1    15 2406.
2     1    14 2163.
3     1    16 2125.
4     1    13 2343.
5     1    13 2296.
6     2    22 2047.
7     2    25 2269.
8     2    27 2115.
9     3    13 2059.
10     3    28 2362.
11     3    27 2157.
12     3    20 2160.
13     3    24 2189.
14     4    22 2343 

This is an example where the pipe, %>% makes the code a little easier to read. We can “un-pipe” our code and it looks like this:

filter(select(tumor, Group, Day, Size), Size > 2000)
# A tibble: 14 x 3
Group   Day  Size
<dbl> <dbl> <dbl>
1     1    15 2406.
2     1    14 2163.
3     1    16 2125.
4     1    13 2343.
5     1    13 2296.
6     2    22 2047.
7     2    25 2269.
8     2    27 2115.
9     3    13 2059.
10     3    28 2362.
11     3    27 2157.
12     3    20 2160.
13     3    24 2189.
14     4    22 2343 

## 4.10 Saving out data

Saving out your data frame objects is the first step in creating pipelines. Even if all you do is read in data, filter it, and write out the subsetted dataset for another use case, you have created your first pipeline. Now, you have a repeatable way of performing the same action when your input dataset is updated or changed.

Let’s say we want to work with a filtered version of our tumor dataset, we can first write the code to subset the data we want.

filtered <- tumor %>%
select(Group, ID, Day, Size) %>%
filter(Day == 0 | Day == 13)
filtered
# A tibble: 60 x 4
Group    ID   Day   Size
<dbl> <dbl> <dbl>  <dbl>
1     1   101     0   41.8
2     1   101    13 1030.
3     1   102     0   79.4
4     1   102    13  619.
5     1   103     0   44.8
6     1   104     0   67.7
7     1   105     0   54.7
8     1   105    13 1699.
9     1   106     0   60
10     1   107     0   46.8
# … with 50 more rows

We can then write out the datafrme to a file (in this case a CSV text file), using the same file pathing string format as we loaded a dataset earlier.

write_csv(filtered, "data/tumor_filtered.csv")

One thing to be mindful of is most programs and computer languages don’t like having spaces in file names. In general, you’ll end up typing more characters dealing with spaces in file names, instead of using an underscore, _, or dash, -. Also as a general rule, keeping files in lower-case will prevent a lot of potential spelling mistakes.

1. Read in all the patient data from the spreadsheets chapter (chapter 2) found in data > spreadsheets > all.csv
2. select the first_name, last_name, day, time, and temp_f columns.
3. filter the rows such that the person’s name is “Aniya Warner”
4. Save the filtered dataset into the output folder named aniya_warner.csv.