from IPython.display import HTML, Markdown, display
import numpy as np
import numpy.random as npr
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
import scipy.stats as stats
import statsmodels.formula.api as smf
import pingouin as pg
import math
import ipywidgets as widgets
import ipywidgets as widgets
from IPython.display import display, Markdown, clear_output
import numpy as np
import pandas as pd
from datetime import datetime
from ipycanvas import Canvas, hold_canvas
import os
from sdt_exp import *
# Enable plots inside the Jupyter Notebook
%matplotlib inline
2**4
16
Signal detection: Part 2#
Authored by Todd Gureckis.
In the previous lab you learned some of the basic analysis steps involved in signal-detection analysis. In particular, we learned how to compute a statistic we called d’ (d-prime) and c (bias/criterion). These variables are commonly used in studies of perception to understand the performance of a subject on a task independent of their response bias (here meaning tendency to give a particular response).
In the second part of the lab we will analyze data from a simple detection experiment you will have just conducted on yourselves. You will get to put your python knowledge into practice by reading in the data analyzing it.
The Experiment: Instructions#
In this experiment your job will be similar to that of a doctor. There are going to be small “clusters” of points which we might think of as a tumor or some other biological measurement. You can use the interaction element below to examine a few of these “clusters”.
Try using the widget below to slide between a lot of dots within a cluster and a small number. Sweep back and forth and get an idea of what a “cluster” could look like. The dots are spread out to a certain degree as you can see more clearly with the large number of dots.
@widgets.interact(dots=(1,75, 5))
def draw(dots):
return draw_stimulus(include_source=True, n_background_dots=0, n_source_dots=dots, source_sd=10, source_border=20, width=400, height=400)
The challenge is that these dots are going to be hidden in a background of “noise” which is other dots unrelated to the cluster. Here is an example of the background with no cluster present:
draw_stimulus(include_source=False, n_background_dots=30*30, n_source_dots=0, source_sd=10, source_border=20, width=400, height=400)
You can run the above cell many times to see different example patterns when the cluster is absent.
Next, here is an example of a very large cluster hidden in background noise:
draw_stimulus(include_source=True, n_background_dots=30*30, n_source_dots=60, source_sd=10, source_border=20, width=400, height=400)
From this example you can see a very clear “cluster” or tumor in the stimulus. Here is one where there is a cluster/tumor but it is a little less obvious:
draw_stimulus(include_source=True, n_background_dots=30*30, n_source_dots=50, source_sd=10, source_border=20, width=400, height=400)
Finally, here is one where it is pretty hard to tell that there is a cluster at all (oh but there is one involving only 20 dots!):
draw_stimulus(include_source=True, n_background_dots=30*30, n_source_dots=20, source_sd=10, source_border=20, width=400, height=400)
Your job in the experiment will be to view stimuli such as the ones above and judge if you think there is a cluster/tumor present in the image. It can often be hard to tell because even in the random dots there can be little clusters of dots that group together. Just try to do your best. Notice that there is a red “absent” button and a green “present” button. You should press “absent” if you think the cluster/tumor is not there, and press “present” if you think it is there.
There are a total of 200 trials. It won’t tell when you get to the end but after the last trial the dots will stop changing.
Before running the next cell, set the subject number to your NYU netid (I will anonymize this later but this will let me check that everyone has done the experiment).
Run the Experiment!#
subject_number = 'br35' # set your subject number here
exp = Detection_Experiment(subject_number)
exp.start_experiment()
Getting the data#
After you run the experiment, the data will be output to a .csv file tied to your subject number. Also, it is available as a pandas dataframe in the current kernel environment as exp.trials:
exp.trials.head()
| signal_present | signal_type | trial_num | button_position | subject_number | |
|---|---|---|---|---|---|
| 0 | 1 | 60 | 0 | left | br35 |
| 1 | 1 | 10 | 1 | left | br35 |
| 2 | 1 | 10 | 2 | left | br35 |
| 3 | 1 | 15 | 3 | left | br35 |
| 4 | 0 | 0 | 4 | left | br35 |
Experimental design#
In the experiment, you saw a number of trials where a field of randomly colored and positioned dots appeared. On some trials a small “cluster” of points was added to the stimulus. The number of dots in the cluster was manipulated across conditions so that it was more or less obvious that there was a cluster. In some cases the number of dots within the cluster was so small that it was hard to detect it, but in other cases it might have been more visually obvious. This is very similar to the type of visual diagnosis that a doctor or airport screener might perform (deciding if a tumor or a gun is present in an image). Your task was to indicate if it was a “stimulus/cluster present” trial or a “stimulus/cluster absent” trial. The computer recorded your response and your reaction time.
An overview of the situation is shown here:
The things we manipulated!#
There were actually four different types of clusters shown on any given “stimulus present” trial. One was a small stimulus made up of 10 dots, one was a medium one made up of 15 dots, one type was made up of 25, and finally one was made up of 60 dots:
As the number of dots increases the cluster becomes more clear and easier to detect.
In total there were 100 stimulus-absent trials and 25 of each of the four types of stimulus-present trials. The purpose of this was so that a perfect subject with robot-quality vision would say “yes” half the time in the experiment so there is no overall bias toward one response.
In this lab we are interested in analyzing the effect of the strength of the stimulus cluster (i.e., the number of dots) on the two signal-detection variables we learned about in the previous lecture and lab.
Analysis Steps#
Remember from the previous lab that our model for signal detection looks something like this:
and we showed how we can compute \(d'\) and \(c\) by computing only the number of false alarms and hits and subjecting those to a particular mathematical transformation. In this case we actually have four stimulus conditions (10, 15, 25, or 60). So our goal will be to compute the hits separately for the two different types of stimuli as well as the false-alarm rate. This will mean that each person in our experiment will be described by eight numbers:
A d’ (d-prime) under each of the cluster-visibility conditions (10, 15, 25, 60)
A c under each of the cluster-visibility conditions (10, 15, 25, 60)
Once we have this we want to perform a simple hypothesis test. Is performance, measured by \(d'\), higher in the high-visibility conditions? We have strong reason to suspect this is the case but it’ll be a nice check of our understanding if we can relate the abstract calculation we performed in the previous lab to answer this simple quesiton.
Answering this question will push our skills at using pandas to organize our data, and using pingouin to compute a \(t\)-test.
If we compute one d' for each participant for each stimulus conditon and want to compare performance in two of the conditions, what type of t-test should we most likely use?
Enter your plan for the analysis as a markdown cell.
To begin we want to read in the data from each participant into a pandas dataframe using `pd.read_csv()' and then use `pd.concat()' to combine them. Each file will have a name of the form 'sdt-SUBJNUM.csv' where SUBJNUM is the subject ID you typed when you ran the experiment. After you read in the data, inspect the columns and check the data for any inconsistencies that might affect our analysis (e.g., subjects who didn't perform any trials, etc.).
Enter your code here.
As a reminder the following cell shows a simple for loop (called a list comprehension) that lets you get the filenames for the lab data set. You can use this to find the file names that you need to read in with pd.read_csv().
# this is an example list comprehension that reads in all the files.
# the f.startswith() part just gets rid of any junk files in that folder
filenames=['sdt1-data/'+f for f in os.listdir('sdt1-data') if not f.startswith('.')]
filenames
['sdt1-data/data_5.csv',
'sdt1-data/data_12.csv',
'sdt1-data/data_21.csv',
'sdt1-data/data_7.csv',
'sdt1-data/data_0.csv',
'sdt1-data/data_6.csv',
'sdt1-data/data_2.csv',
'sdt1-data/data_4.csv',
'sdt1-data/data_20.csv',
'sdt1-data/data_13.csv',
'sdt1-data/data_16.csv',
'sdt1-data/data_15.csv',
'sdt1-data/data_8.csv',
'sdt1-data/data_3.csv',
'sdt1-data/data_11.csv',
'sdt1-data/data_14.csv',
'sdt1-data/data_10.csv',
'sdt1-data/data_18.csv',
'sdt1-data/data_1.csv',
'sdt1-data/data_22.csv',
'sdt1-data/data_17.csv',
'sdt1-data/data_19.csv',
'sdt1-data/data_9.csv']
Did you find any interesting or important points about the data that you want to share with the class? Let's get everyone on the same page so we get similar results so if you find something important (good or bad) please take a moment and share!
Now that we have the analysis dataframe let's perform our planned analysis on a single subject. Create a new dataframe, subselecting from the main frame the data from a single subject.
df = pd.DataFrame()
for f in filenames:
df = pd.concat([df, pd.read_csv(f)])
print (df.head ())
Unnamed: 0 signal_present signal_type trial_num button_position \
0 0 1 60 0 left
1 1 1 10 1 left
2 2 1 10 2 left
3 3 1 15 3 left
4 4 0 0 4 left
subject_number responses rt correct_resp
0 std_5 1 6991.284 1
1 std_5 0 2665.210 0
2 std_5 0 1682.308 0
3 std_5 0 1512.275 0
4 std_5 0 2217.154 1
Referring back to the first notebook from the lab, compute the hits and false alarms for this subject for each stimulus-present condition. One note about the false alarms: remember that false alarms are trials where the stimulus was absent but the subject said it was present. There is only one type of these trials in the experiment. So when you compute the statistics you really will create five numbers for this participant: the hits for each of the stimulus-present conditions and one overall false-alarm rate. What are these numbers for this subject? Remember also that if you take the `mean()' of a column of 0/1 values, it will tell you the proportion of those trials that are 1.
def process_s (s1):
def adjusted_rate (correct_resp):
# Apply Bayesian correction (to avoid 0s and 1s)
return (correct_resp.sum () + .5) / (correct_resp.size + 1)
sig_idx = s1.signal_present == 1
# Compute HRs
hrs = s1[sig_idx].groupby('signal_type').correct_resp.apply(adjusted_rate)
# Compute FAR
far = 1 - adjusted_rate (s1[~sig_idx].correct_resp)
# Concatenate hrs and far into 1D series.
rates = pd.concat([hrs, pd.Series (far)], ignore_index=True)
# Add appropriate labels
rates.index = ['hr_' + str (i) for i in hrs.index] + list (['far'])
# Return as single row data frame.
return (pd.DataFrame([rates]))
rates_s = df.groupby('subject_number').apply(process_s)
print (rates_s.mean ())
hr_10 0.106187
hr_15 0.199833
hr_25 0.574415
hr_60 0.918896
far 0.079423
dtype: float64
/tmp/ipykernel_437/1663103670.py:18: FutureWarning: DataFrameGroupBy.apply operated on the grouping columns. This behavior is deprecated, and in a future version of pandas the grouping columns will be excluded from the operation. Either pass `include_groups=False` to exclude the groupings or explicitly select the grouping columns after groupby to silence this warning.
rates_s = df.groupby('subject_number').apply(process_s)
Ok now compute the d' and c values for this subject. Refer back to the previous notebook for reference if you need the equations and python code. What can you tell from looking at your computed values? If for some reason you can't compute these values in your experiment (e.g., if the subject has no hits or no false alarms) then try question 4 again with a different subject.
def cd_fun (s1):
def d_fun (hr):
return (stats.norm.ppf(hr) - stats.norm.ppf(far))
def c_fun (hr):
return -(stats.norm.ppf(hr) + stats.norm.ppf(far)) / 2.0
rates = s1.iloc[0]
far = rates['far']
hrs = rates[['hr_10','hr_15','hr_25','hr_60']]
ds_cs = list (map (d_fun, hrs)) + list (map (c_fun, hrs))
df1 = pd.DataFrame ([ds_cs], columns=['d_10','d_15','d_25','d_60','c_10','c_15','c_25','c_60'])
return (df1)
cd_s = rates_s.groupby('subject_number').apply(cd_fun)
print (cd_s.mean ())
d_10 0.348301
d_15 0.761212
d_25 2.082693
d_60 3.369519
c_10 1.691729
c_15 1.485274
c_25 0.824533
c_60 0.181120
dtype: float64
Ok, so now you have performed the desired analysis for one subject. You have computed eight numbers for that subject as described above. Now, we want to repeat this for each subject. There are two ways to do this. One is using the pandas groupby() function. This is somewhat of an advanced version of how to code this analysis, but if you do this you can probably perform the analysis in only 5-6 lines of code. An alternative approach is to generalize the code you just wrote to use a for loop that iterates over each subject in the dataframe. Whichever way you choose you might find it helpful to refer to the Forty For Loops notebook I provided, which shows some example templates for using for loops to analyze individual subject data in pandas dataframes. The end result is that you want a dataframe containing the d' and c for each subject for each stimulus condition.
Enter your code here.
Now that you have your summary statistics computed, our next step is to ask if the conditions are different on either the d' or c measure. For now concentrate on the easiest and hardest conditions to see if there is an effect in the most dissimilar conditions. Then, compare the hardest and second-hardest conditions. Conduct the appropriate t-test to tell if these conditions are different. The chapter reading from this week has all the code you need. However, you might find it helpful to refer to the pingouin documentation for the t-test function.
pg.ttest(x=df_s_cd['d_10'], y=df_s_cd['d_60'], paired=True)
| T | dof | alternative | p-val | CI95% | cohen-d | BF10 | power | |
|---|---|---|---|---|---|---|---|---|
| T-test | -22.831252 | 22 | two-sided | 8.232322e-17 | [-3.64, -3.04] | 5.360455 | 5.52e+13 | 1.0 |
pg.ttest(x=df_s_cd['c_10'], y=df_s_cd['c_60'], paired=True)
| T | dof | alternative | p-val | CI95% | cohen-d | BF10 | power | |
|---|---|---|---|---|---|---|---|---|
| T-test | 22.831252 | 22 | two-sided | 8.232322e-17 | [1.52, 1.82] | 2.72726 | 5.52e+13 | 1.0 |
pg.ttest(x=df_s_cd['d_10'], y=df_s_cd['d_15'], paired=True)
| T | dof | alternative | p-val | CI95% | cohen-d | BF10 | power | |
|---|---|---|---|---|---|---|---|---|
| T-test | -3.43633 | 22 | two-sided | 0.002358 | [-0.75, -0.18] | 0.968255 | 16.807 | 0.993297 |
pg.ttest(x=df_s_cd['c_10'], y=df_s_cd['c_15'], paired=True)
| T | dof | alternative | p-val | CI95% | cohen-d | BF10 | power | |
|---|---|---|---|---|---|---|---|---|
| T-test | 3.43633 | 22 | two-sided | 0.002358 | [0.09, 0.37] | 0.305251 | 16.807 | 0.288256 |
This question is a bonus but asks you to verify the assumptions of the chosen t-test hold. Usually this involves looking to see that the distribution of scores you are analyzing are roughly "normally" distributed. The textbook showed a few example way to check including the histogram, the qqplot, and the Shapiro-Wilks test. Perform this test on your data. If the data seem very non-normal you might consider switching your t-test to the non-parametric alternative discussed in the chapter (the Wilcoxon or Mann-Whitney test).
Enter your code here.
Ok great job! You have your data, you have your t-test, you have everything you need to answer the following questions: 1. Is performance better in the low number-of-dots or high number-of-dots condition? How do you know? How would you report the statistical evidence in favor of your conclusion in a paper? Are the criteria different between the conditions? How do you know? Was overall was performance good or poor in this task?
Enter your Markdown text here.
The final question asks you to consider the limitations of this experimental design. What other analyses might be interesting to perform on these data? If you could change the experiment is there another type of question you would ask?
Enter your Markdown text here.