Wednesday, January 23, 2008

R-precision as consistency

Last post, I mentioned r-precision as a way to measure the accuracy of a recommendation algorithm. I thought it may be pertinent to analyze in more detail the r-precision results of the certain bag-of-frames approach I'm working with.
For completeness, the results here are from using the uspop2002 dataset (105 artists, 10 tracks per artist, 20 MFCC's per 66.67 ms frame) modeled with 32 component GMM's and using the KL-divergence-based earth-mover's distance (KL-EMD) as the similarity metric. This is a standard introduced years ago, and one I'm inclined to stick with for comparison's sake.
Below are listed the top ranking artists by r-precision along with their average r-precision values. This means that their songs are more closely connected to each other in the similarity network than other artists'. Again, I'm only modeling timbre, so artists with a highly consistent "sound" will have high average r-precision.
  1. westlife - 0.711
  2. korn - 0.589
  3. mya - 0.456
  4. goo goo dolls - 0.422
  5. lionel richie - 0.411
  6. deftones - 0.378
  7. craig david - 0.378
  8. ricky martin - 0.367
  9. staind - 0.367
  10. savage garden - 0.356
We see Westlife, a Irish boy-band, at the top of the list. While I'd like to chalk this up to the homogeneous sound of teen pop music, I think some of these files are only fragments of songs, perhaps making the models particularly distant from others. But Korn and the Goo Goo Dolls don't have this excuse.
Looking at the bottom of the list:
  1. chemical brothers - 0.0
  2. depeche mode - 0.0111
  3. radiohead - 0.0222
  4. fatboy slim - 0.0222
  5. daft punk - 0.0222
  6. coldplay - 0.0222
  7. sting - 0.0333
  8. portishead - 0.0333
  9. pet shop boys - 0.0333
  10. oasis - 0.0333
So, it appears that artists we would naturally associate with being charmingly inconsistent are indeed at the bottom of the r-precision list. Furthermore, the collection actually uses song from multiple albums for several of these artists (3 + a single for Chemical Bros., 3 + a singles collection for Depeche Mode, and 5 for Radiohead), compared to the top artists (1 each except for Craig David's 2 + a single).
This shows that my content-based recommendation engine just may be doing what it's suppose to. A track from The Bends would not be the most appropriate result for a query seeded from a Kid A track, something I wouldn't expect a collaborative-filtering-based engine to necessarily deal with. This agnostic power is what appeals to me most about this approach. A machine trained to analyze, and dare I say "understand", music recommends based on the music as it is encoded as audio (which, after all, is how humans perceive it), not by any tags or hype that may be attached to it.

3 comments:

J Bass said...

I noticed that 7 out of 10 of the bands that were listed at the bottom of your R-scale thingy are UK-native bands. Is the software picking this up in the timbre of the music? Or is their something I missed? Maybe a possible geographical connection to similar timbral qualities?

Mark T. Godfrey said...

I'm not sure. I think it may just that they're more electronically oriented bands, as opposed to straight-up rock or pop, making them naturally more timbrally eclectic. And because the artists in this collection that fit this description come from the UK for the most part, that's why we might see this.

If you're interested, I posted the full list (sorted by descending r-precision) here.

It's funny Enya is so high on the list though. Consistently "new age"?

Anonymous said...

Where are the Eagles?