Neural Networks for Learning Lyrics

I created a Twitter account which was inspired by a couple Twitter accounts that applied a particular type of machine learning technique to learn how two (at the time) presidential hopefuls spoke. I thought, why not see what a model like this could do with lyrics from my favorite rock n roll artist?

Long short term memory (LSTM) is a recurrent neural network (RNN) that can be used to produce sentences or phrases by learning from text. The two twitter accounts that inspired this were ‘@deeplearnthebern’ and ‘@deepdrumpf’ which use this technique to produce phrases and sentences.

I scraped a little more than 300 of his songs and have fed them to a LSTM model using R and the mxnet library. Primarily I used the mxnet.io/ to build and train the model…great site and tools.  The tutorials on their site are very helpful and particularly this one.

The repository is here that contains the code for the scraper and other information.

Follow deeplearnbruce for tweets that are hopefully entertaining for Springsteen fans or anyone else. 

State of the Union Speeches and Data

I've done a couple posts on the SOTU speeches.  In the past these dealt with word count, approval, and the vague notion that the applause the president receives has a relationship with his approval rating at that time (which had a lower correlation this year in fact).

Wired had a good article highlighting the sentiment in the current and previous State of the Union (SOTU) speeches.  They went through the speech for several of the past years, highlighted the events that occurred each year, and gave the corresponding frequency or usage of terms in the speech that communicated the impact of those events.  This blog post is not duplicating the article.  I did see the graph though and wanted to see if I got a similar sentiment score for the speeches.  I used the 'syuzhet' library in R to conduct the analysis (big thanks to Matthew Jockers for the package).

The graph is similar to the one in the Wired article, but not entirely.  Some smoothing was involved and perhaps a different sentiment analysis technique.  We do see a similar finding in the most recent SOTU speech:  it ended with the highest sentiment score out of all the speeches.  Several of the speeches in my analysis showed a curving up toward the end, which would in general go along with "ending on a positive note".  Additionally, one can see the "valleys" or lower sentiment values occurring between the 50 and 75 time intervals.  This isn't too surprising in that the same speech writer is being used and that the SOTU has perhaps a more standard sentiment form (another analysis perhaps?).  

This same library has a function which scores certain words to emotional categories.  These 10 categories include a positive/negative categorization.  Along with these, I added in the applause count for each speech and the approval rating for each year for the time period of the speech.  The matrix below depicts the correlation values of each category with corresponding color.  Additionally, I added in a p-value scoring for each relationship, those >.1 were given bubbles.

There's a lot here in terms of what could be said about the speeches but I'll only say a few things that I thought were interesting.  The applause/approval rating correlation showed a weaker value than last year (-.5), which isn't too surprising since this is probably spurious anyways.  Negative word categorization and applause had a higher correlation than positive word categorization and applause.  Meaning, when comparing applause and negative word use across speeches, these counts varied in a similar way (applause count higher - negative word count higher and vice versa).  Speeches with words categorized as "anger" or "fear" had a weak correlation to the applause count.  Conversely, speeches with words categorized in emotions like "joy", "surprise", and "trust" portray a stronger correlation with applause count in those same speeches.  So perhaps to get more applause in general, certain positive words are better than others?  Yoda's advice about fear would make sense here in that words associated with fear tend to vary similarly to words associated with anger.

We also see a decent amount of correlation among more positive emotions as well as within more negative emotions.  This refers back to the common "curve" that these speeches may have.  In that the sentiment used year over year tend to be similar, or at least the emotional categorization of words follow similar patterns.

Thanks to Matthew Jockers, Taiyun Wei, and Hadley Wickam for their work on the 'syuzhet', 'corrplot', and 'ggplot' packages respectively.  Code for the above analysis is on my github page.

State of the Union Speech Comparison

The State of the Union is historically a speech used to set the tone for the year...at least I think that's what it is.  The process of writing these speeches has been discussed before and first-hand detail is available in this article.  I was interested in seeing the difference in language between the 2013 and 2014 State of the Union address in terms of what was said.  A lot was said in each speech so I narrowed it down to words that were mentioned 20 times total in the two speeches.  Below is a wordcloud showing these words, with the larger words having a higher frequency.

You can see just from this wordcloud how these speeches seem to take a more emphatic/determined/action-oriented tone with the word "will" having the highest frequency (aside from another word we will see below).  Also, you can get an idea of some of the policy issues he discussed.

As noted before, I wanted to compare the two speeches to see which words were being used more/less when comparing the two.  The number of times a word is mentioned in the 2013 State of the Union address is on the X-axis and the 2014 address is on the Y-axis.

Probably one of the odder yet clearer takeaways from this graph is the applause given in 2014 vs 2013 (in the speech text this appears when there is pause for applause also for cheers).  The President went into this speech having a very low approval rating.  That being said, was the applause a function his party knowing his lack-luster rating?  Or was it just that the speech was better and elicited much more applause than last year?  It is interesting that this year's speech received more than 10 additional pauses for applause than 2013.  "Jobs" was used more last year in the speech than this year whereas "help" was used more this year.  This may reflect some of the policy decisions with the Affordable Healthcare Act having been past and other programs intended to provide "help" for Americans.   Last year after the election jobs and employment were a large part of what worried many Americans (and for many still does).  "Energy" was mentioned much less in this year's SOTU.  Perhaps the controversy over the Keystone XL pipeline influenced the topic of energy as a priority in the speech this year.  Many of the words were also very linear in their use (seen clustered around 10).

Context is certainly important in speeches and this technique of comparing words by no means completely depicts what is important or how things were communicated.  However, the words that are chosen for these speeches are strategic and outline (if not in function than in form) the year for the president and his priorities.