Getting to Know Your Data

Before analyzing a dataset you'll want to familiarize yourself with the data. Computational tools like a spreadsheet make it easy to quickly get a sense of what your data looks like.

Make a copy of the data

You will need to get your own copy of the data to clean. If your data is currently in a Google Sheet you'll need to "Make a copy" of the data, as shown below.

Do This: Make your own copy of the data collected by your class

Making a copy of the dataset

Filtering Your Data

Your data may be too large to look at all at once. One way to address this problem is to filter the data so that you are only looking at some of the rows. Here's how you can do it in Google Sheets.

Do This: Follow the example in the animation below. Filter your own data on one of its columns

Filtering the dataset so that only rows in which a person slept for 7 hours are shown.

More Complex Filtering

Filters may include more than one value, just check all the values you'd like to include. You can even use a conditional statement as a filter. For example, you can make a filter that will only show values if the value is less than some number you specify. It's also possible to filter on multiple columns at the same time. More complex filters help refine how you look at your data.

Do This: Add filters to two different columns. In one column choose at least two values. In the other column use a Conditional filter, as shown below.

Filtering to include people who feel "Good" or "Great" and who slept 7 hours or more.

Sorting Data

Sorting will reorder your rows of data by one of the columns. This makes it easy to see the smallest or largest value in each column. You may also notice patterns in your data once it is in sorted order.

Do This: Sort your data by at least one of the columns, both A -> Z and Z -> A

Sorting the dataset by number of hours worked, both from smallest to largest and from largest to smallest

Continue

Why is it Important to Clean Data?

Data and data analytics have become a key factor in problem solving. Thanks to the internet, people have been able to crowd source data collection and develop even more insights about many prominent issues around the world-- ranging from election statistics to restaurant reviews. Leveraging the internet is an effective way to collect and share data, but can introduce problems when users input their information in different ways. To counteract the issues that come along with non-standardized data , it is necessary to 'clean' it. Through this process all inaccuracies, inconsistencies, and irrelevant data is either removed or corrected.

Limitations of the Computer

Using computational tools to analyze data has made it much easier to find trends and patterns in large datasets. When preparing data for this kind of analysis, however, it's important to remember that the computer is much less "intelligent" than we might imagine. Small discrepancies in the data may prevent accurate interpretation of trends and patterns and can even make it impossible to use the data at all. Cleaning data is therefore an important step in analyzing it, and in many contexts, it may actually take the largest amount of time.

In the Curriculum

In lesson 13 students will follow a guide in Code Studio, which demonstrates the common techniques of filtering and sorting data, to familiarize themselves with its contents. Then they will correct errors they find in the data by either hand-correcting invalid values or deleting them. Finally they will categorize any free-text columns that were collected to prepare them for analysis. This lesson introduces many new skills with spreadsheets and reveals the sometimes subjective nature of data analysis. The main purpose here is have students independently apply some of the data manipulation skills (in spreadsheets) that they've learned over the past few lessons, to a new dataset that is relatively uncurated. This is the beginning of the process of "extracting knowledge from data": look at the data and clean it up so that you can process it using computational tools.

Categorizing Data

In order to use computers to analyze data we usually need it to be standardized in some way.

Data collected as "free form text" will be particularly susceptible to this problem. If you ask people "What did you do last night" you will likely get a different response from every single person. Making charts or tables with this information would be meaningless and confusing.

Free form text data like this may be useful for a human to read but cannot easily be used by a computer.

In order to fix this you might need to create new columns of data by hand which categorize free form text into data that is more useful for computation.

In the example below you'll see one possible way to categorize responses to the question, "What did you do to relax last night?" In this case the data is being categorized by what the person was doing. The resulting column of data will be much easier to use when analyzing this data with a computer.

Standardizing how people relaxed by how they do the activity. This is only one of many different ways this data could be categorized.

Note: This process is being done by hand using only one of many possible methods for creating categories. There are always many different ways to categorize data and there will rarely be one clearly "right" way. You'll have to make judgment calls about what makes the most sense.

Do This:

Create at least one new column in your dataset that standardizes a column of data collected as "free form text".

To do this you will need to invent a set of categories that applies to your data. This may take a while to do. Consult with your partners and make sure you have a generally agreed-upon set of rules as you start.

Good luck!

Continue