
Finding the most depressing Radiohead song with R - kevlar1818
http://rcharlie.com/2017-02-16-fitteR-happieR/
======
3chelon
So presumably the purpose of this experiment was to quantify just how bad
algorithms can be at inferring meaning?

Trying to measure the sadness of lyrics with an algo is tantamount to the
early scene in "Dead Poets' Society" where the students are supposed to
measure the greatness of poetry by plotting it on a chart.

It's demonstrated in the data: one of the most depressing songs of their
entire oeuvre is "Fitter, Happier" but because it's completely metaphorical
(or sarcastic), the algo rates it as one of the happiest.

This will always be a big problem in AI. Measuring the literal meaning of
words is easy. Measuring the semantics and context of words is a completely
different matter.

~~~
jmmcd
> This will always be a big problem in AI

I agreed with everything you wrote except for "always".

~~~
Grue3
Considering even actual humans often have disagreements about meanings of
songs, I'm not sure how an AI is supposed to deduce the "correct" one.

~~~
andrepd
Neither am I. But 10 years ago I would have said I had no idea how an AI was
supposed to recognise objects in a picture, and now they do it better than
humans. Tech moves fast.

~~~
jmmcd
Recognising objects in pictures is something that, in principle, a machine
could do better than humans, because there is ground truth.

I think it's not possible in principle for a machine to do better than humans
at recognising human emotions, because a human can't exactly be wrong at this.
(Other than edge cases such as not knowing the right name for an emotion,
perhaps.)

But suppose a sample of 1000 humans answered a survey on emotional content of
a song and a machine was able to predict the distribution within some
tolerance. In a sense, this would be superhuman performance, but even if we
don't like that interpretation, it would still be _very good_ performance. And
it is possible in principle. So the argument that humans sometimes disagree
doesn't prevent a machine doing well on the task.

~~~
anewhnaccount
Another possible output for this type of task is an opinion or interpretation
and a justification of it. A machine might feasibly be considered "good" if it
output an unpopular interpretation with a good justification.

~~~
jmmcd
Great point! The same type of argument is going in the AI subfield of
computational creativity. A computational system might produce a piece (of art
or music, say) which nobody initially likes, but could provide a
justification/interpretation which might be convincing. In some ways this
would be a big improvement over a pure Turing test-style judgement of the
output itself. I think Margaret Boden has written this idea into her
definition of computational creativity.

EDIT: "The ultimate vindication of AI-creativity would be a program that
generated novel ideas which initially perplexed or even repelled us, but which
was able to persuade us that they were indeed valuable.", from Boden,
Creativity and Artificial Intelligence, 1998. I believe, based on her other
writings, that she thinks of it as a sufficient, but not a necessary
condition, for AI creativity.

------
runeks
I think you will find wildly varying perspectives on whether a given song is
depressive or not. To my surprise, when I read the YouTube comments on
Radiohead's track _Nude_ , I found many of the commenters found the song
depressing, because of the lyrics:

"Don't get any big ideas, they're not gonna happen."

But, as far as I can see, this is only depressing if, first, you expect all
your big ideas to happen and, secondly, you associate a big idea happening
with happiness. I mean, why would a big idea not happening be depressing,
unless you believe this big idea will make you happy?

I quite like the reminder that big ideas usually don't happen, unless you put
in an earnest effort. And that the expectation of happiness from a big idea is
the only thing that can cause depression if it doesn't work out, which it
often doesn't.

In my experience, big ideas come out of happiness, not the other way around.

~~~
treerock
I think it's the "don't get any big ideas" bit that is depressing.

------
decasteve
Thom Yorke once said that after touring for OK Computer he was at a local pub
and some guy stopped him and said, "No Surprises" is the most miserable song
he'd ever heard and why had he written it? At the time Thom said he had to
concur with that assessment of the song. I can't remember what interview this
was from but I used to have the recording of it where he introduces the song
this way.

~~~
cjbprime
I once walked into my baby's daycare to pick him up after work and the babies
were listening to a happy glockenspiel version of No Surprises.

It felt very strange.

~~~
rcar
The makers of those albums are quite prolific:
[https://en.wikipedia.org/wiki/Rockabye_Baby](https://en.wikipedia.org/wiki/Rockabye_Baby)!

~~~
cjbprime
Oh thanks! I'd tried and failed to find it myself.

------
genericpseudo
Cute! But, well, this is obviously bullshit:

> the cheeriest, 15 Step

The title line: " _fifteen steps and then a sheer drop_ ". A song which makes
explicit references to, well, execution by hanging is...

(And "you used to be alright; what happened?" is a repeated motif.)

~~~
draugadrotten
> The title line: "fifteen steps and then a sheer drop"

This song is obviously about Thom Yorke's frustration with the Mario Party
minigame "Shy Guy Says". Think about it, 'first you reel me out then you cut
the string', 'won't take my eyes off the ball again' (ball, in this case,
meaning shy guy's flags). It makes perfect sense.

\-- lankeyjb202

------
wiremine
They're missing In Rainbow's Disk 2, which includes the song "4 minute
warning" which I'd consider the "most" depressing Radiohead song:
[https://www.youtube.com/watch?v=VtotpiSL700](https://www.youtube.com/watch?v=VtotpiSL700)

~~~
banku_brougham
thank you for pointing this out. i only recently discovered IR2, and i found
that "4 minute warning" fot my mood very well.

------
chjohasbrouck
I think this is an example of something machines can't quite do better than
humans yet, though it's an interesting problem to solve (trying to analyze how
people in general will feel about something).

Personally I'd give it to either How to Disappear Completely, or Exit Music
(For a Film).

~~~
marchenko
Exit Music might be their most bitter song, but I'd say Street Spirit or How
to Disappear are the saddest. I wonder how many people would have an immediate
answer at the ready, and how many would have to ponder and classify ( assuming
similar familiarity with the Radiohead catalogue)

~~~
yownie
[http://antiquiet.com/music/2012/06/flashback-thom-yorke-
expl...](http://antiquiet.com/music/2012/06/flashback-thom-yorke-explains-
street-spirit-breaks-our-hearts/)

“‘Street Spirit’ is our purest song, but I didn’t write it…. It wrote itself.
We were just its messengers… Its biological catylysts. It’s core is a complete
mystery to me… and (pause) you know, I wouldn’t ever try to write something
that hopeless… All of our saddest songs have somewhere in them at least a
glimmer of resolve… ‘Street Spirit’ has no resolve… It is the dark tunnel
without the light at the end. It represents all tragic emotion that is so
hurtful that the sound of that melody is its only definition. We all have a
way of dealing with that song… It’s called detachment… Especially me.. I
detach my emotional radar from that song, or I couldn’t play it… I’d crack.
I’d break down on stage.. that’s why its lyrics are just a bunch of mini-
stories or visual images as opposed to a cohesive explanation of its meaning…
I used images set to the music that I thought would convey the emotional
entirety of the lyric and music working together… That’s what’s meant by ‘all
these things are one to swallow whole’.. I meant the emotional entirety,
because I didn’t have it in me to articulate the emotion… (pause) I’d crack….
Our fans are braver than I to let that song penetrate them, or maybe they
don’t realize what they’re listening to.. They don’t realize that ‘Street
Spirit’ is about staring the fucking devil right in the eyes… and knowing, no
matter what the hell you do, he’ll get the last laugh…and it’s real…and true.
The devil really will get the last laugh in all cases without exception, and
if I let myself think about that to long, I’d crack. I can’t believe we have
fans that can deal emotionally with that song… That’s why I’m convinced that
they don’t know what it’s about. It’s why we play it towards the end of our
sets. It drains me, and it shakes me, and hurts like hell everytime I play it,
looking out at thousands of people cheering and smiling, oblivious to the
tragedy of it’s meaning, like when you’re going to have your dog put down and
it’s wagging it’s tail on the way there. That’s what they all look like, and
it breaks my heart.”

------
zackkatz
I vote for the happiest song being Anyone Can Play Guitar from Pablo Honey. It
is hopeful and encouraging. It speaks of life goals achieved.

This article demonstrates how far AI has to go.

[https://m.youtube.com/watch?v=GIWwfWaWuaE](https://m.youtube.com/watch?v=GIWwfWaWuaE)

~~~
JauntTrooper
Good choice! I've also always thought 'Lift' felt happy.

------
TheRealDunkirk
If the winning song wasn't written in D-minor, the algorithm will need further
tuning.

------
rodionos

      > and then used the tidytext package to break the lyrics into words, 
      > eliminate common "stop words" like 'the' and 'a', 
      > and count the number with negative sentiment
    

Is 'not' a "stop word"? How did it count negations after "stop words" removed,
i.e. "I am not forgotten". Positive, negative?

~~~
ellisv
"not" is a stop word. I don't see an attempt to negate the word sentiments.

------
mpettitt
Would be interesting to see how it handled REM or Dubstar, both of whom have
some really upbeat tunes, with really dark lyrics - "The One I Love", for
example, with "a simple prop to occupy mind ", or "Not So Manic Now", a happy
bouncy pop song about assault...

------
beeswax
'Depressing' has a variety of facets made up of sadness, textual
content/lyrics, and (at least for me) the placement of the title within its
respective album context.

Anecdote: When I previewed Kid A via AudioGalaxy back then (it would not be
released for another two weeks in my region) I thought the stylistic choices
of 'Everything in its right place' were some kind of encoding error (quite
frequent back then) so I trashed the whole album after spending hours on the
download.

Imagine my surprise when I listened to the album from the physical album I
bought a fortnight later :)

~~~
loudmax
I got my first cassette tape player in 1982 and Prince's "1999" album to go
with it. The lead track starts with distorted vocals, and I thought my player
was malfunctioning. The intro is only about ten or fifteen seconds, so unlike
you, I didn't trash the tape.

Closer to topic, listening to the song "1999" as a kid, I thought it was a
cheerful party song. After Prince's death, I listened to the song as an adult
and realized it's about nuclear annihilation.

------
mwexler
While great to see, this also highlights just how far we are from where data
science needs to be. The amount of code necessary to just pull down and
reformat data before any real analytic work is done is always staggering; I
see it in my work but it's even painful in a fun example like this. I know,
80% of time in prep, 20% in analysis, yadda yadda, but I hope we can turn some
of this magical AI into making data transforms easier in the future so we can
actually get to the analysis portion faster, along with more accessible "tidy"
data to get to the analysis portion faster.

And no, tools like Paxata or Tamr (or
[http://www.predictiveanalyticstoday.com/data-preparation-
too...](http://www.predictiveanalyticstoday.com/data-preparation-tools-and-
platforms/) etc.) are indeed a good start, but are not really the solution
yet.

We'll always have to code our way out of some wacko data situations, but I
look forward to (and dream of!) better and better libraries and approaches to
getting prep prepped faster.

~~~
minimaxir
R in general is _inefficient_ with nontabular data manipulation with text data
and web scraping, which is why I fall back to Python for those tasks.

But even in the context of the article, the code the author posted is
extremely confusing and is an odd mix of base and dplyr code.

------
Hydraulix989
"Honey Pablo"?

------
cdelsolar
Definitely Exit Music for a film.

------
peeters
If I can nitpick a bit about your presentation here:

[http://rcharlie.com/htmlwidgets/fitterhappier/album_chart.ht...](http://rcharlie.com/htmlwidgets/fitterhappier/album_chart.html)

If you're going put each album on its own point in the X-axis, could you not
just label those axis instead of using colour-coding and requiring us to look
up the corresponding colour in a legend (that's not ordered in the same way).

------
aethant
They need to do this with morrissey.

~~~
xellisx
As my dad said back in the day(80's), Morrissey is the music you listen to
while cleaning your guns.

------
cjbprime
Is there a way to have Spotify users create the mapping, e.g. by using song
selection to detect their mood and finding the cluster of Radiohead songs that
are only ever listened to in a depressive mood?

------
dmichulke
Note how listening to all suggestions mentioned in the HN comments but only to
few or none of those recommended in the blog post is a good indicator of what
you think about the power of ML/AI.

------
elchief
Guessing "How To Disappear Completely" before I read it

~~~
elchief
and...not on the list. Exit Music and Let Down are pretty glum though

Ingo Mierswa (of RapidMiner) wrote some papers on analyzing audio data, like

[http://link.springer.com/article/10.1007%2Fs10994-005-5824-7](http://link.springer.com/article/10.1007%2Fs10994-005-5824-7)

[http://dl.acm.org/citation.cfm?id=1150523](http://dl.acm.org/citation.cfm?id=1150523)

------
VeejayRampay
I'm waiting for someone to find out the most stolen songs from Radiohead's
repertoire using R now :)

------
sampl
_True Love Waits_

Edit: according to the article :)

~~~
zzalpha
How on earth is Street Spirit not even on the list?

Yorke himself had called it "our purest, saddest song.", and went on to say:

 _I can’t believe we have fans that can deal emotionally with that song.
That’s why I’m convinced that they don’t know what it’s about. It’s why we
play it towards the end of our sets. It drains me, and it shakes me, and hurts
like hell every time I play it, looking out at thousands of people cheering
and smiling, oblivious to the tragedy of its meaning, like when you’re going
to have your dog put down and it’s wagging its tail on the way there. That’s
what they all look like, and it breaks my heart. I wish that song hadn’t
picked us as its catalysts, and so I don’t claim it. It asks too much. I
didn’t write that song._

True Love Waits is a sweet, quiet, sad little song about yearning and the fear
of loss.

Street Spirit is pure dark tragedy.

~~~
genericpseudo
Apparently the second happiest song on OK Computer is "Fitter Happier". (A pig
/ in a cage / on antibiotics.)

So much of Radiohead's bleakness is in their use of irony.

~~~
marchenko
Irony and taunting really confounds sentiment analysis. I mean, street spirit
ends in "immerse your soul in love" repeated 2X but is still incredibly bleak.

------
snyp
I can't believe the author got the name Of their first album wrong!

------
dcdevito
How to Disappear Completely is their most depressing song, ever.

------
mhd
I wonder how Gloomy Sunday would score with that heuristic.

------
foxhedgehog
you could have just asked and I would have told you it's "let down"

------
jheriko
is the answer ALL OF THEM

(j/k)

------
SloopJon
This is a repost (syndication?) of a blog post at:

[http://blog.revolutionanalytics.com/2017/02/finding-
radiohea...](http://blog.revolutionanalytics.com/2017/02/finding-radioheads-
most-depressing-song-with-r.html)

which is a summary of a blog post at:

[http://rcharlie.com/2017-02-16-fitteR-
happieR/](http://rcharlie.com/2017-02-16-fitteR-happieR/)

I didn't know that Spotify has an API. I see mention of a rate limit in the
docs, but I can't find the actual limit. If I want to sort the tracks from
albums released in a given year (say, ten to fifty out of a thousand-album
collection) by Spotify popularity, will the limit get in the way?

~~~
carlob
Yeah can we please change this to the original blog post at rcharlie.com?

~~~
bbrady1992
Agreed. I'm not sure why the original link is necessary at all. It doesn't add
anything and seems to have been written for no reason other than to write a
blog post. The author refers to Radiohead's first album as 'Honey Pablo', so
it doesn't look like he actually paid attention to the original article.

------
sjclemmy
TLDR: It's High and Dry.

I could have guessed that. But where is Black Star in the list? That is my go
to song if I'm really lamenting the state of the world. ;)

~~~
executesorder66
No it's not. It's actually True Love Waits.

High and Dry is 8th.

TLDR; RTFA

~~~
sjclemmy
You're right. I skimmed it and only saw the first list of Saddest Lyrics.

I'm going to go away now, and listen to Radiohead, whilst I consider my lack
of focus, past failures and impending mortality. ;)

