The video shows how I perform the statistical test Krippendorff alpha to check the reliability of a variable with nominal/dichotomous data. (Windows PC & SPSS.)
Step 1. Search for and download the Kalpha macro. I use the keywords download, macro, kalpha, spss to find it online.
Sometimes, you will find the file named "kalpha.sps". And sometimes, you will find a compressed, zipped file called "kalpha.zip." Either way, download this to your own computer.
Step 2. If it is the uncompressed file "kalpha.sps", leave it as it is. If it is the zip file, which is a compressed folder, you must extract it. Right-click on it, and extract. Remember where you save it because you need it later.
Step 3. Start SPSS and open a blank, empty page.
Step 4. File, Open, Syntax
Step 5. Search your computer, select and open the file "kalpha.sps". The macro.
As you can see, it is just a script. Do not change anything. Just leave it.
Step 6. Execute this macro by Run, All.
The window that pops up is normal. This is just a confirmation that all is OK. You can close it and the script if you want.
Step 7. Open your data. Make sure that your data is correct. Numeric type and nominal level of measure.
Step 8. It is time to run the statistical test. Normally, you select a test from a menu. But, since this test is not included inside SPSS, we need to execute it ourselves.
File, New, Syntax, and Type:
kalpha judges = obs1 obs2 obs3 obs4 obs5/level = 1/detail = 0/boot = 10000
I will explain what it means later.
Then: Run, All
This is how I would write my report:
Method
Krippendorff's alpha test was used (Hayes & Krippendorff, 2007) to estimate the inter-coder reliability, and these alpha (α) values are reported in the results below.
Results
The results show that the inter-coder reliability was relatively high (α = 0. 7831), i.e., the five coders agreed.
Discussion
(Here, you interpret and discuss plausible reasons why you think that the coders did (or did not) agree. It is beyond the scope of this example, where I only focus on the statistical test.)
References
Hayes, A. F., & Krippendorff, K. (2007). Answering the call for a standard reliability measure for coding data. Communication Methods and Measures 1(1), 77-89.