This chapter explains how to rearrange your data to meet your needs. Basic sorting, data extraction and re-combination are discussed.
To get the most of this chapter, you should have worked through Chapter 3 on entering data manually. You do not need to have completely read through Chapter 4 on importing data as the examples here do not rely on its content. Ultimately, you will want to use the material in Chapter 4 in conjunction with what follows here.
This chapter starts using the functionality that is built into R, but we also take a look at an add-on package called dplyr that is becomeing more and more commonly used.
The sort() command is illustrated in this section. It works for sorting a single vector of numeric values into ascending or descending order.
We use the order() command to work with data that are non-numeric vectors, as well as data in matrix or data.frame format. For example, we first sort the names of me and my three children initially given in descending order of age.
So we see that the order() command doesn’t actually do the re-ordering after all, but that we need to use the subscripts to extract the elements (the full set) from the pre-existing set.
We do need to take note of how the upper and lower case letters will be ordered.
The order() command can prove a useful method of re-ordering experimental data if the run order has been stored as a variable. To illustrate this point we create a data.frame with two elements.
Note that the sample() command used here has created a random permutation of the numbers one to ten.
To illustrate the extraction of a set of observations from a data.frame, we make use of one of the internal datasets provided with the base installation of R. We retreiv the airquality dat using:
An explanation of this command is given in Section 4.5. This data set has its own help page that can be viewed by typing ?airquality at the R prompt.
We can ask for the rows of this data that match specified details using subscripting. Various operators exist for numerical comparisons; the simplest of which are < and >. For the purposes of brevity in our examples that follow, we use the nrow() command to show how many observations in the data meet the specified conditions. The following commands extract hte number of days where the wind speed was less than ten miles per hour and then the temperature was less than 60 degF.
If we want to know how many observations have an exact value, we use == not a single equals sign because it has a different meaning in R. The double equals sign is the one used for comparisons. For example, the number of days when the temperature was recorded as 69 degF can be found using:
To print out the rows of the airquality data.frame where the temperature was less than 60 degrees and the wind was less than ten miles per hour, we would use the & operator between the two conditions that identify which rows are required.
The dplyr package is an alternative to the basic R functionality shown in this chapter. The package creator (Hadley Wickham) wants his package to make data manipulation easier for the end-user, and more fficient with respect to computation time. He has simplified data manipulations to a small set of commands: filter(), arrange(), sample_n(), sample_frac(), mutate(), and transmute() are discussed here, while the others will be discussed in Section 8.5 where they are more relevant.
You’ll need to install the package (see Section 16.1 for instructions) before running the function
to gain access to the commands just listed. Note that the head() command is used in many of the following examples to ensure that only a handful of rows are printed out instead of the full result. An alternative is to create a new data structure using the tbl_df() command. Let’s make a second copy of the air quality data that is the same as the original except for being made ready for use with the dplyr functions.
and then see the structure of the data using the glimpse() function that is a substitute for the str() command we’ve used previously.
The glimpse() command works on a data.frame or the new structure called a tbl_df equally well.
The filter() and arrange() commands make many of the examples seen earlier in this chapter much easier. The filter() command is for extraction, while arrange() is for rearrangement. For example,
extracts the rows of the airquality data that meet the combinations of conditions specified. Use of filter() means we don’t need the data.frame attached, or to gain access to the variables in it using the dollar notation; we also don’t need the square brackets for common subscripting tasks. We could have stored the outcome in a new data.frame if we wished.
Perhaps the greatest gift of the dplyr package is its popularising a different way of combining commands, called the pipe operator. The last command issued can be re-written as:
This way of presenting commands is considered to be easier to read by many users which helps with the popularity of the dplyr package. The pipe operator comes from another package called magrittr so its use is not limited to the commands found in the dplyr package. This way of presenting multiple commands is called “chaining" in many resources. I like the simplicity that this offers so I have tried to use the pipe operator as often as I can from here on.
The filter() command is useful if we know the values a variable takes, but at times we just want to know more about the observations that are best or worst according to one (or more) variables. For example, if we wanted to find the hottest days, we can use the arrange() command instead of the order() command seen earlier.
We can add more variables to sort the data by if we like. We used the desc to get the temperatures to be in descending order in this example.
The temperatures just listed are measured on the Fahrenheit scale. I live in a country that uses the centigrade scale so a conversion is needed to make these results more interpretable. Enter the mutate() and transmute() commands.
will create a new column in the returned data. N.B. the new data was not stored in a new object so this command’s effect was purely temporary.
Again, notice that we didn’t need to do all that much to add a new column and that this code is perhaps a little simpler than the alternative:
which we saw earlier. The main difference between the two commands is that this one will add the new colun to the original data.frame. In this instance the difference is probably fairly trivial, but when we use much larger data sets, the storage of results is important because of speed and memory usage concerns.
The command
creates the new variable as a stand alone object. It hasn’t been stored as an object so is only temporarily available.
Finally, the dplyr package also gives users two simple ways to extract a random sample from a data.frame. The commands sample_n() and sample_frac() extract a sample of a predetermined size and a specified fraction respectively. For example:
and
The dplyr package is under active development and so we should expect more functionality to become available, but there is also work being done to make the processing of data even faster so that it can handle huge data sets.