Retrieving Reading Levels with R

For those that don't work in education or aren't aware, there is a measurement for a child's reading level called a Lexile ® Level.  There are ways in which this level can be retrieved using different reading assessments.  The measurement can be used to match a child's reading level to books that are the same level.  So the company maintains a database of books that they have assigned levels.  There are other assessments that measure how well a child is reading, but I'm not aware of any systems that assign books as comprehensively.  The library at the school I work at lacked Lexile Levels for their books.  This is unfortunate because of the fact that teachers are not able provide students with books for their respective Lexile Level.  Fortunately though the library had a list of ISBNs for the books.

On the Lexile website there is a way to search for books using ISBN numbers to retrieve their Lexile ® Level if the book is available in their database.  Entering every ISBN number available is a task fit for something not human.

rvest to the rescue.

Below is the script to retrieve the Lexile Levels of books if a list of ISBNs is available.  This was an incredible time save provided by some R code and hopefully someone else out there could use it.
 

	library(rvest) library(httr)
	library(htmltools)
	library(dplyr)
	##Prep for things used later
	url<-"https://www.lexile.com/fab/results/?keyword="
	url2<-"https://lexile.com/book/details/"
	##CSV file with ISBN numbers
	dat1<-read.csv("~isbns.csv",header=FALSE)
	##dat1<-data.frame(dat1[203:634,])
	dat<-as.character(dat1[,1])%>%trimws()
	##dat<-dat[41:51]
	blank<-as.character("NA")
	blank1<-as.character("NA")
	##blank2<-as.character("NA")
	##blank3<-as.character("NA")
	all<-data.frame("A","B","C")
	colnames(all)<-c("name","lexiledat","num")
	all<-data.frame(all[-1,])
	for(i in dat) {
	sites<-paste(url,i,sep="")
	x <- GET(sites, add_headers('user-agent' = 'r'))
	webpath<-x$url%>%includeHTML%>%read_html()
	##Book Name
	name<-webpath%>%html_nodes(xpath="///div[2]/div/div[2]/h4/a")%>%html_text()%>%trimws()
	##Lexile Range
	lexile<-webpath%>%html_nodes(xpath="///div[2]/div/div[3]/div[1]")%>%html_text()%>%trimws()%>%as.character()
	##CSS change sometimes
	lexiledat<-ifelse(is.na(lexile[2])==TRUE,lexile,lexile[2])
	test1<-data.frame(lexiledat,NA)
	##Breaks every now and then when adding Author/Pages
	##Author Name
	##author<-webpath%>%html_nodes(xpath='///div[2]/div/div[2]/span')%>%html_text()%>%as.character()%>%trimws()
	##author<-sub("by: ","",author)
	##Pages
	##pages<-webpath%>%html_nodes(xpath='///div[2]/div/div[2]/div/div[1]')%>%html_text()%>%as.character()%>%trimws()
	##pages<-sub("Pages: ","",pages)
	##Some books not found, this excludes them and replaces with NA values
	df<-if(is.na(test1)) data.frame(blank,blank1) else data.frame(name,lexiledat,stringsAsFactors = FALSE)
	colnames(df)<-c("name","lexiledat")
	df$num <- i
	all<-bind_rows(all,df)
	}
	master<-rbind(all1,all)

   
Link to code

Neural Networks for Learning Lyrics

I created a Twitter account which was inspired by a couple Twitter accounts that applied a particular type of machine learning technique to learn how two (at the time) presidential hopefuls spoke. I thought, why not see what a model like this could do with lyrics from my favorite rock n roll artist?

Long short term memory (LSTM) is a recurrent neural network (RNN) that can be used to produce sentences or phrases by learning from text. The two twitter accounts that inspired this were ‘@deeplearnthebern’ and ‘@deepdrumpf’ which use this technique to produce phrases and sentences.

I scraped a little more than 300 of his songs and have fed them to a LSTM model using R and the mxnet library. Primarily I used the mxnet.io/ to build and train the model…great site and tools.  The tutorials on their site are very helpful and particularly this one.

The repository is here that contains the code for the scraper and other information.

Follow deeplearnbruce for tweets that are hopefully entertaining for Springsteen fans or anyone else. 

Supreme Court Politics

I had wanted to post this before the US Election, but time constraints didn't allow.  With the potential for new Supreme Court Justices in the next four years, many voters and namely single-issue voters rallied behind Donald Trump for his seeming support for a conservative justice.  Most of the people I spoke with were primarily concerned with the potential appointment of a justice who could help in concentrating efforts in overturning abortion.  For a more in-depth look on Trump, his stances and commentary on overturning abortion I found this article to be helpful.

I was curious in the past how the political leanings of the Justices has changed over time and if opportunities like a conservative bench with a republican president have occurred in the past.  I was curious because I wondered if an appointment during the Trump administration would change anything.

The graph below shows the amount of abortions over time during different presidents and each point shows the political split of the bench (Democrat-Republican).

Political leanings on a bench aren't indicative of a pro-life vote.  However, learning more conservative politically does provide potential for a favorable pro-life vote.  I found this information interesting since the split on the court has predominantly been conservative for the last 40 years and only recently has it become more liberal.  The graph speaks for itself and the code for it is available here.

This information isn't put out here to necessarily change minds on this issue.  That's best done around tables where both sides can listen, but I thought this was helpful for understanding some of the numbers and history on this particular circumstance.  

State of the Union Speeches and Data

I've done a couple posts on the SOTU speeches.  In the past these dealt with word count, approval, and the vague notion that the applause the president receives has a relationship with his approval rating at that time (which had a lower correlation this year in fact).

Wired had a good article highlighting the sentiment in the current and previous State of the Union (SOTU) speeches.  They went through the speech for several of the past years, highlighted the events that occurred each year, and gave the corresponding frequency or usage of terms in the speech that communicated the impact of those events.  This blog post is not duplicating the article.  I did see the graph though and wanted to see if I got a similar sentiment score for the speeches.  I used the 'syuzhet' library in R to conduct the analysis (big thanks to Matthew Jockers for the package).

The graph is similar to the one in the Wired article, but not entirely.  Some smoothing was involved and perhaps a different sentiment analysis technique.  We do see a similar finding in the most recent SOTU speech:  it ended with the highest sentiment score out of all the speeches.  Several of the speeches in my analysis showed a curving up toward the end, which would in general go along with "ending on a positive note".  Additionally, one can see the "valleys" or lower sentiment values occurring between the 50 and 75 time intervals.  This isn't too surprising in that the same speech writer is being used and that the SOTU has perhaps a more standard sentiment form (another analysis perhaps?).  

This same library has a function which scores certain words to emotional categories.  These 10 categories include a positive/negative categorization.  Along with these, I added in the applause count for each speech and the approval rating for each year for the time period of the speech.  The matrix below depicts the correlation values of each category with corresponding color.  Additionally, I added in a p-value scoring for each relationship, those >.1 were given bubbles.

There's a lot here in terms of what could be said about the speeches but I'll only say a few things that I thought were interesting.  The applause/approval rating correlation showed a weaker value than last year (-.5), which isn't too surprising since this is probably spurious anyways.  Negative word categorization and applause had a higher correlation than positive word categorization and applause.  Meaning, when comparing applause and negative word use across speeches, these counts varied in a similar way (applause count higher - negative word count higher and vice versa).  Speeches with words categorized as "anger" or "fear" had a weak correlation to the applause count.  Conversely, speeches with words categorized in emotions like "joy", "surprise", and "trust" portray a stronger correlation with applause count in those same speeches.  So perhaps to get more applause in general, certain positive words are better than others?  Yoda's advice about fear would make sense here in that words associated with fear tend to vary similarly to words associated with anger.

We also see a decent amount of correlation among more positive emotions as well as within more negative emotions.  This refers back to the common "curve" that these speeches may have.  In that the sentiment used year over year tend to be similar, or at least the emotional categorization of words follow similar patterns.

Thanks to Matthew Jockers, Taiyun Wei, and Hadley Wickam for their work on the 'syuzhet', 'corrplot', and 'ggplot' packages respectively.  Code for the above analysis is on my github page.

Philip Glass Composition and Exploding Boxplot

This post will highlight a couple of my favorite things.  R programming and composer Philip Glass.  For those of you not familiar with his works, he basically pioneered the "minimalist" style of piano playing.  He has been writing and performing music since the 60s, and his pieces are still heard in film and other genres.  Much information on the composer can be found at philipglass.com

I put together a file for all his compositions, their date and composition style.  Using a new R package 'explodingboxplot', the compositions and dates can be visually inspected.  Click the image below to navigate to the interactive plot.

Philip Glass Compositions by Date and Type

In general when looking at the compositions of an artist over time like this, we can make a few observations.  The style of music that he composed the most evenly distributed overtime was "Chamber" music.  The wikipedia page describing the time-periods of his work mentions today as the "Chamber music" time-frame.  Early in his career we can see his compositions and really his reputation was built on his ensemble and solo compositions, though his solo compositions continued into more recent years (boxplot in red).  His ballets were mostly confined to the 80s and 90s, where his music accompanied the modern choreographer Twyla Harp in her pioneering contribution to contemporary ballet.

In general, this visualization is quite useful when considering summary information about categorical time-series data.  As it pertains to the compositions of an artist, it is a nice way to quickly survey their life-time work.  The package 'explodingboxplot' is available on github.  Much thanks to Kent Russell for his work on this R package.

Marathon Races Shiny App

About a year ago I posted about men's and women's marathon (and longer) distance races from the Arrs.net dataset.  In the meantime, Shiny development and the open source announcement of plot.ly have brought data visualization to the next level.  As an avid (at least former) runner, exploring marathon data is interesting at both the personal and "data science" (or is that personal too?) levels.  Thus, I finished a Shiny app that explores this dataset from 2014.  Unfortunately, 2015 data is not being updated for one reason or another, but 2014 provides a lot of observations about marathon and longer distance races.

Click here to access the app.

The values can be toggled between months for 2014 and a searchable table of all the data is below the graph.  You will notice many of the points are small ultra-marathons around the world.  Plot.ly provides nice graph interactive abilities found when hovering in the upper right corner of the graph.

Thanks to RStudio for all their work on Shiny and to Plot.ly for their plotly package and charting library.

Continental Language Diversity

Since language data provides for the demonstration of many visualization techniques, I thought of using another set showing official languages spoken across continents using a new visualization in the "UpSetR" package.  The graph can be used for comparing sets of data numerically.  It provides an easier way to understand data for something like a Venn diagram more quantitatively.  Whereas the appeal of Venn diagrams is in their aesthetic, they do not provide understanding for the numeric value of sets when the sets reach a number that is visually challenging to interpret. 

Displaying the grouping of languages by continent was one way I thought to illustrate the use of the "UpSetR" package.  The data limits each country to one primary language.  The countries are then grouped into continents based on their location.  Below we can see the different continents and respective bar graphs associated with a continent or a set of continents.  The bar graph on the left side (y-axis) measures the number of languages in each continent.  The bar graph at the top (x-axis) measures the number of occurrences for each set of languages seen in the filled circles.  So each filled dot or set of dots represent a set or grouping of languages. 

For instance the red dot indicates 27 different languages in Europe that only occur in Europe, making it seemingly the most diverse in terms of official languages spoken.  The yellow dot set represents 2 languages that are spoken in countries located on 5 different continents (any guesses?*).  This graph does not capture the many different languages spoke in different countries but only the "lingua franca" associated with a continent.  For instance, Africa though linguistically diverse has as official status languages from the colonial-era, thus showing above a lack of diversity.

Alternatively we can see the transposition of this graph below.  Here we see the size of the intersects is small (because we are now considering continent intersects of which there are 1 for each continent).  Guessing the continents for this graph is perhaps a bit easier than guess the different languages for each continent above.

For those having data whose organization is in sets, visualizing sets in this way allows for various dimensions of data to be understood in a way not captured by other visualizations.  This tool for this particular language data set is an interesting view on official language use across continents.  The data used for these graphs is shown below.

The very few lines of code it took to make these graphs is available here.  Much thanks to the developers of this package!

*English and French

	rows	Africa	Antarctic	Asia	Europe	North.America	Oceania	South.America	Countries
1	Albanian	0	0	0	1	0	0	0	1
2	Arabic	1	0	1	0	0	0	0	17
3	Armenian	0	0	0	1	0	0	0	1
4	Azerbaijani	0	0	1	0	0	0	0	1
5	Belarusian	0	0	0	1	0	0	0	1
6	Bosnian	0	0	0	1	0	0	0	1
7	Bulgarian	0	0	0	1	0	0	0	1
8	Catalan	0	0	0	1	0	0	0	1
9	Croatian	0	0	0	1	0	0	0	1
10	Czech	0	0	0	1	0	0	0	1
11	Danish	0	0	0	1	0	0	0	1
12	Dutch	0	0	0	1	1	0	0	3
13	English	1	0	0	1	1	1	1	39
14	Estonian	0	0	0	1	0	0	0	1
15	Filipino	0	0	1	0	0	0	0	1
16	Finnish	0	0	0	1	0	0	0	1
17	French	1	0	0	1	1	1	1	22
18	Georgian	0	0	0	1	0	0	0	1
19	German	0	0	0	1	0	0	0	4
20	Greek	0	0	0	1	0	0	0	2
21	Heard	0	1	0	0	0	0	0	1
22	Hebrew	0	0	1	0	0	0	0	1
23	Hindi	0	0	1	0	0	0	0	1
24	Hungarian	0	0	0	1	0	0	0	1
25	Icelandic	0	0	0	1	0	0	0	1
26	Indonesian	0	0	1	0	0	0	0	1
27	Italian	0	0	0	1	0	0	0	2
28	Japanese	0	0	1	0	0	0	0	1
29	Khmer	1	0	0	0	0	0	0	1
30	Korean	0	0	1	0	0	0	0	2
31	Lao	0	0	1	0	0	0	0	1
32	Latvian	0	0	0	1	0	0	0	1
33	Lithuanian	0	0	0	1	0	0	0	1
34	Malay	0	0	1	0	0	0	0	1
35	Maltese	0	0	0	1	0	0	0	1
36	Mandarin	0	0	1	0	0	0	0	1
37	Norwegian	0	0	0	1	0	0	0	1
38	Persian	0	0	1	0	0	0	0	1
39	Polish	0	0	0	1	0	0	0	1
40	Portuguese	1	0	0	1	0	0	1	7
41	Romanian	0	0	0	1	0	0	0	1
42	Russian	0	0	1	0	0	0	0	2
43	Slovak	0	0	0	1	0	0	0	1
44	Slovenian	0	0	0	1	0	0	0	1
45	Spanish	1	0	0	1	1	0	1	22
46	Swahili	1	0	0	0	0	0	0	1
47	Swedish	0	0	0	1	0	0	0	1
48	Thai	0	0	1	0	0	0	0	1
49	Turkish	0	0	1	0	0	0	0	1
50	Ukranian	0	0	0	1	0	0	0	1
51	Vietnamese	0	0	1	0	0	0	0	1