To see if any of my homogenization experiments are actually meaningful, it's important to check for statistical significance. I first used a standard paired t-test to compare the pair-wise distances between songs, hubness values, and r-precision values for each song obtained for both homogenization methods. And I made a table.
| | | | | | | homo by dist | | | | | | p-values | | | | | | t-test | | | | | | | 30-thresh | 25-thresh | 20-thresh | 15-thresh | | dist | 0 | 0 | 0 | 0 | | hubness | 1 | 1 | 1 | 1 | | r-precision | 0.3532 | 0.032 | ~0 | ~0 |
|
| | | | | | | homo by activation | | | | | | p-values | | | | | | t-test | | | | | | | -110-thresh | -100-thresh | -90-thresh | -80-thresh | | dist | ~0 | ~0 | ~0 | ~0 | | hubness | 1 | 1 | 1 | 1 | | r-precision | 0.1063 | 0.398 | 0.017 | ~0 |
|
We see that all of the song-to-song distances are significantly changed by homogenization.
The r-precision values are more mixed. For the distance-based method, only the highest threshold (the one with the least affect) was not significantly changed. For the activation-based method, only the lowest two thresholds are significant. This means our only hope at improvement (activation homogenization at a -100 threshold) does not have statistically large enough lead to mean anything. Oh well.
The reason we see no significance in the hubness with the t-test is that it's a constant sum measure, so all of the changes will cancel out and the mean remains the same (in fact, equals the number of occurances we observe, in this case 100). This way our null hypothesis of a zero mean difference distribution will always be true.
Looking at the distributions of hubness differences (
dist,
act) , it seems they aren't really normal: some seem to have a marked skew. A better significance test for hubness change is the Wilcoxon signed-rank test, where the null hypothesis is that the
median difference between pairs is zero. More tables!
| | | | | | | homo by dist | | | | | | p-values | | | | | | signed-rank-test | | | | | | | 30-thresh | 25-thresh | 20-thresh | 15-thresh | | dist | 0 | 0 | 0 | 0 | | hubness | ~0 | ~0 | 0.0049 | 0.8251 | | r-precision | 0.271 | 0.0112 | ~0 | ~0 |
|
| | | | | | | homo by activation | | | | | | p-values | | | | | | signed-rank-test | | | | | | | -110-thresh | -100-thresh | -90-thresh | -80-thresh | | dist | 0 | 0 | 0 | 0 | | hubness | 0.0045 | ~0 | 0.0475 | 0.3494 | | r-precision | 0.2258 | 0.1602 | 0.057 | ~0 |
|
Now, we see some significance. For the distance-based method, the top three thresholds have seemingly small difference medians (2, 2, and 1, meaning the homogenization decreased the median song's occurances by 2, 2, and 1 occurances, respectively) but large enough to be significant. The top three thresholds for the activation-based method were also significant (with 95% confidence). This is encouraging, but the changes are still small.
I'd love to hear any suggestions or complaints; my stats skills are admittedly a little rusty.
