Homework 3 - Exploring Data

Homework 3 - Exploring Data#

Written by Todd Gureckis with edits from Maya Malaviya

So far we’ve learned about using Jupyter, Python basics, and Pandas dataframes. We have also covered some initial ways to plot data.

In this homework, we are going to practice those skills in more depth, so we can really “explore” a data set. At this point we are not really going to be doing much in terms of inferential statistics. Our goals are just to be able to formulate a question about the data and then take the steps necessary to compute descriptive statistics and plots that might give us a sense of the answer. You might call it “Answering Questions with Data.”

This is a long notebook, here’s a summary of the sections:

  1. Reminder of Pandas Functions with Salary Dataset

    • run the code cells and see the output visuals

    • you don’t need to write code, we wrote it for you

  2. Exploration of OkCupid Data

    • 2.1. Plot a histogram & describe what you observe

    • 2.2. Find mean median and modes

    • 2.3. Another histogram of different variable

    • 2.4. Propose something you want to examine & do it

  3. Exploration of Citibike Data

    • 3.1. Look at columns and guess what they represent

    • 3.2. Use pandas.groupby() to group and analyze data in a new format

Reminders of basic pandas functions w/ Salary Dataset#

imports#

First, we load the basic packages we will be using in this tutorial. Remeber how we import the modules using an abbreviated name (import XC as YY). This is to reduce the amount of text we type when we use the functions.

Side Note: %matplotlib inline is an example of something specific to Jupyter call ‘cell magic’ and enables plotting within the notebook and not opening a separate window.

%matplotlib inline 

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import io
from datetime import datetime
import random

As a reminder of some of the pandas basics let’s revisit the data set of professor salaries we have considered over the last few lectuers.

salary_data = pd.read_csv(
    "salary.csv",
    sep=",",
    header="infer",
)

Notice that the name of the dataframe is now called salary_data instead of df. It could have been called anything as it is just our variable to work with. However, you’ll want to be careful with your variable names if you copy over any code from previous notebooks.

peek at the data#

salary_data.head()

Get the size of the dataframe#

In rows, columns format

salary_data.shape

Access a single column#

salary_data[["salary"]]

Compute some descriptive statistics#

salary_data[["salary"]].describe()

salary_data[["salary"]].count()  # how many rows are there?

Creating a new column based on the values of others#

salary_data["pubperyear"] = 0
salary_data["pubperyear"] = salary_data["publications"] / salary_data["years"]

Selecting only certain rows#

sub_df = salary_data.query("salary > 90000 & salary < 100000")

sub_df

sub_df.describe()

Get the unique values of a column#

salary_data["departm"].unique()

How many unique department values are there?

salary_data["departm"].unique().size

or

len(salary_data["departm"].unique())

Breaking the data into subgroups#

male_df = salary_data.query("gender == 0").reset_index(drop=True)
female_df = salary_data.query("gender == 1").reset_index(drop=True)

Recombining subgroups#

pd.concat([female_df, male_df], axis=0).reset_index(drop=True)

Scatterplot two columns#

ax = sns.regplot(x=salary_data.age, y=salary_data.salary)
ax.set_title("Salary and age")

Histogram of a column#

sns.displot(salary_data["salary"])

You can also combine two different histograms on the same plot to compare them more easily.

fig, (ax1, ax2) = plt.subplots(ncols=2, figsize=(10, 3), sharey=True, sharex=True)
sns.histplot(
    male_df["salary"],
    ax=ax1,
    bins=range(male_df["salary"].min(), male_df["salary"].max(), 5000),
    kde=False,
    color="b",
)
ax1.set_title("Salary distribution for males")
sns.histplot(
    female_df["salary"],
    ax=ax2,
    bins=range(female_df["salary"].min(), female_df["salary"].max(), 5000),
    kde=False,
    color="r",
)
ax2.set_title("Salary distribution for females")
ax1.set_ylabel("Number of users in age group")
for ax in (ax1, ax2):
    sns.despine(ax=ax)
fig.tight_layout()

Group the salary column using the department name and compute the mean#

salary_data.groupby("departm")["salary"].mean()

Group the age column using the department name and compute the modal age of the faculty#

First let’s check the age of everyone.

salary_data.age.unique()

Ok, there are a few people who don’t have an age so we’ll need to drop them using .dropna() before computing the mode.

Since there doesn’t seem to be a mode function provided by default we can write our own custom function and use it as a descriptive statistic using the .apply() command. Here is an example of how that works.

def my_mode(my_array):
    counts = np.bincount(my_array)
    mode = np.argmax(counts)
    return mode, counts[mode]


# we need to drop the
salary_data = salary_data.dropna()
salary_data['age'] = salary_data['age'].astype(int)
salary_data.groupby("departm")["age"].apply(my_mode)

OkCupid Data#

This document is an analysis of a public dataset of almost 60000 online dating profiles. The dataset has been published in the Journal of Statistics Education, Volume 23, Number 2 (2015) by Albert Y. Kim et al., and its collection and distribution was explicitly allowed by OkCupid president and co-founder Christian Rudder. Using these data is therefore ethically and legally acceptable; this is in contrast to another recent release of a different OkCupid profile dataset, which was collected without permission and without anonymizing the data (more on the ethical issues in this Wired article).

profile_data = pd.read_csv(
    "https://github.com/rudeboybert/JSE_OkCupid/blob/master/profiles_revised.csv.zip?raw=true",
    compression="zip",
)

## your histogram code here

## your stats code here

## your histogram code here

## your choice of analysis and visualization here

Citibike Data#

As you know, Citibikes are the bike share system in NYC. What you might not realize is that a lot of the data about Citibikes is made public by the city. As a result it is a fun dataset to download and to explore. Although this dataset is not exactly “cognition and perception,” it is readily available and is a great way to train up your exploratory data analysis skills!

Loading Data

The data for Citibike are provided on a per-month basis as a .zip file. As a result we first have to download the file and unzip it. Luckily python can do that for you!

Remember, we use pd.read_csv() to load a .csv file into a Pandas dataframe. In this case our Citibike data frame will be called trip_data. Be sure to read the code and try to get a sense of what it is doing. Ask questions if you are unsure.

trip_data = pd.read_csv(
    "JC-202401-citibike-tripdata.csv"
)

This command loads the data from Jan 2024 (The data is called JC-202401-citibike-tripdata.csv). Can you guess how we know the date of the data?

## code for investigating trip_data here

## code using groupby to group data by starting station

Turning in homeworks#

To turn in your homework, follow these steps:

  1. Download your completed notebook in .ipynb format

  2. Email the notebook to Maya & Andy

    • Title the email: “Lab in C&P Homework 3 Submission”

    • Title the notebook: hw3_lastname_firstname.ipynb

    • (The standardized format will help us filter out these emails from other ones better!)

Please reach out to the instructor or TA if you have any problems or questions!