Generating Country Music Lyrics with a Recurrent Neural Network

the world to see,
 is to someone who saved the world,
that take the wheel and the glory,
the ones it shows our lives,
’cause there ‘s nothing there at all …

~ My CountryNet Model

In a previous post, I sought to analyze a large dataset of country music lyrics.  Consider this a follow-up. However, instead of conducting a static analysis of lyrics again, the goal of this post is to train and utilize a model that can “generate” country music lyrics. As I continue to improve this model I’ll update this post with more lyrics.

Data

In the textual analysis, I scraped a site called Cowboy Lyrics for all of my data. This was less than ideal, as it wasn’t very clean and took some time to isolate the lyrics. When refactoring some code for this post I discovered the Genius API. This is a free API where you can query information and lyrics for artists across genres. It’s very convenient for a project of this nature. API keys are free, go get yours.

In terms of Artists, I found an arbitrary list of the Top 100 Country Artists and combined it with some of the artists on Wikipedia’s list of country music performers (2000-2009 era). We start by calling the API to get Genius’s ID for each artist.

import requests
from bs4 import BeautifulSoup
import sys
import time
#your-api-key below from Genius
base_url = "http://api.genius.com"
headers = {'Authorization': 'Bearer your-api-key'}
search_url = base_url + "/search"
artist_ids = []
##Get Genuis Ids for Artists
for artist in artists:
  params = {'q': artist}
  response = requests.get(search_url, params=params, headers=headers)
  json = response.json()
  for hit in json['response']['hits']:
  if hit["result"]["primary_artist"]["name"] == artist:
   artist_id = hit['result']['primary_artist']['id']
   break
  try:
   if song_info:
    pass
  except NameError:
   continue
 artist_ids.append(artist_id)
 time.sleep(1)
 print(artist, artist_id)

With all the artist codes in artist_ids, we can loop through and perform a search for songs belonging to each artist’s ID. When we get a song in the search results, we’ll store its URL to obtain the full lyrics later. I harvested results from 2 pages of artist results in an attempt to get around 100 songs (if that many were availible) from each artist.

urls=[]
base_url = "http://api.genius.com/artists/"+str(artist_id)+"/songs?per_page=50"
response = requests.get(base_url, headers=headers)
json = response.json()
response2 = requests.get(base_url+str('&page=2'), headers=headers).json()
comb = json['response']['songs'] + response2['response']['songs']
for item in comb:
 urls.append(item['url'])

With a large list of URL’s, we can query each and extract the lyrics, artist, and song name from each page.

page = requests.get(url)
html = BeautifulSoup(page.text, "html.parser")
#remove script tags that they put in the middle of the lyrics
[h.extract() for h in html('script')]
#at least Genius is nice and has a tag called 'lyrics'!
lyrics = html.find('div', class_='lyrics').get_text() #updated css where the lyrics are based in HTML
artist = html.find('a', class_='header_with_cover_art-primary_info-primary_artist').get_text()
song = html.find('h1', class_='header_with_cover_art-primary_info-title').get_text()

The data isn’t perfect yet, there are a lot of instances of \n\n and some text with brackets like [Verse 1] or [Chorus] that I wanted to scrub out. In addition, I wanted to add spaces before and after any \n tokens. Lastly, I deleted any songs with Instrumental in them. After cleaning, I have fourteen thousand songs to use as a dataset.

The Model

A recurrent neural network is a model that deals exceptionally well with sequence data. I find these images featured on WildML from the journal Nature very intuitive for understanding how the model works.

A recurrent neural network and the unfolding in time of the computation involved in its forward computation.

x_t is our input at any given time. For our purposes, x_t is a representation of a word (For now, think of it like cat = 1, dog = 2, table = 3….). This representation will be multiplied by some weights, U, and combined with some other data, s_{t-1}, to get s_ts_t is called the “hidden state” and is an intermediate representation of the word, obtained by x_t and some information about all previous words before it, s_{t-1}. We can transform this intermediate representation of s_t into some output, o_t, using weights V. In this exercise, x is a song. We will feed the model x and for each x_t obtain a o_t.  We will attempt to train the model so that o_t will tell us what word is most likely to be the next word in the song. To train, we feed in a song and examine our o_t‘s. We determine how wrong we are and use the information about how we were wrong to alter the weights, U, W, V, so that next time we try to predict this song, we are “less wrong”. This process of feeding a song, examining the outputs, and passing back the error to adjust the weights, is repeated many many times over the dataset. We hope that after this process, the model will have internalized some information about how words in country music lyrics interact with each other in its weights, U, W, V.

The above is a very high-level description, but I believe sufficient for understanding the basics of the model and the rest of this post. There are plenty of resources available if you wish to learn more about this type of model. Take a look at WildML, or this post or this one.

Using keras, you can quickly construct neural networks with ease. I used Theano, a now discontinued deep learning framework, to manually build my first version of this. It took a significant amount of time to build and was hard to debug. But, building it manually certainly helped me understand what was going on under the hood. With Keras, I can build a similar structure in minutes. It’s less customizable, but for someone like me who seeks to apply neural networks as tools, rather than research new architectures, it is fantastic. My model is quite simply constructed using the code below.

model = Sequential()
model.add(Embedding(VOCAB_SIZE, 64, input_length=400, mask_zero=True))
model.add(GRU(256, return_sequences=True))
model.add(GRU(128, return_sequences=True))
model.add(Dense(VOCAB_SIZE, activation='softmax'))
 Calling model.summary() will show us how the model looks and the number of parameters.
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
embedding_1 (Embedding)      (None, 400, 64)           409600    
_________________________________________________________________
gru_1 (GRU)                  (None, 400, 256)          246528    
_________________________________________________________________
gru_2 (GRU)                  (None, 400, 128)          147840    
_________________________________________________________________
dense_1 (Dense)              (None, 400, 6400)         825600    
=================================================================
Total params: 1,629,568
Trainable params: 1,629,568
Non-trainable params: 0
_________________________________________________________________

Training Datasets

We have the songs, and with some context on what the model will be doing, I can now explain a bit about how we construct the training data for this exercise.

I begin by taking each song, making everything lowercase, and appending some characters that represent the “start” and “end” of each song. This is done with the hope that the model might learn what types of things happen at the beginning and ends of songs, as well as note a stopping point when we are generating text later.

songs = ['%s %s %s' % (SONG_START_TOKEN, x.lower(), SONG_END_TOKEN) for x in lyrics]

From there, we tokenize each song. Tokenization is the process of splitting a string of text into an array of strings and characters, tokens. For instance "The quick brown fox" becomes ["The", "quick", "brown", "fox"].

Next, we pre-determine a number of words to have in our model’s “vocabulary”. This is because some words occur very rarely in lyrics and I’m comfortable replacing them with a generic token, UNKNOWN, to decrease the model’s size. I arbitrarily chose 6400 words. Out of a total of 29000 unique words in the dataset, this is a significant reduction. Despite omitting over 22000 words from our vocabulary, we still manage to keep all words that occur more than 13 times. Take a look at the word frequency chart below to get an understanding of how many words occur at each frequency level. Screenshot 2018-04-13 16.22.56.png

With a vocabulary set, we can turn each word into a number 1-6400. This number represents an index to an embedding matrix. Each word will obtain a series of values that “describe” the word and how it exists in the context of other words. These values are called “Word Vectors” and a general description of them can be found here. Having the significance of words represented by a series of numbers seems like an abstract thought, but word vectors are a powerful and impressive tool. These word vectors will be what the model ultimately ingests and learns from.

Text Generation

With the above settled, we can begin training the model. I stop training when the reduction in loss slows significantly.

gpumodel=multi_gpu_model(model, gpus=2)
gpumodel.compile(optimizer='rmsprop',loss='sparse_categorical_crossentropy',metrics=['accuracy'])
gpumodel.fit(working_x_train,np.reshape(working_y_train, (14394, 400, 1)), epochs=1000, batch_size=125)

Because the output at each step is a probability distribution of which word will show up next, we can use that distribution to sample from the words in our vocabulary and hopefully construct some comprehensible lyrics. We initialize by feeding the model a sequence of length one, a “start” token, obtain an output, add that to our sequence, feed the sequence back to the model, and repeat. We hope that eventually, we will get an “end” token that will tell us to stop sampling. There’s a chance we won’t so I add a cutoff to stop sampling after 300 words.

# Start with start
new_sentence = [word_to_index[SONG_START_TOKEN]]
# End with End
while not new_sentence[-1] == word_to_index[SONG_END_TOKEN]:
 pad_next = tf.keras.preprocessing.sequence.pad_sequences(
  [new_sentence],
  maxlen=400,
  dtype='int32',
  padding='pre',
  truncating='pre',
  value=0.0)
 next_word_probs = model.predict(pad_next)[0][-1]
 next_word_probs = [float(x)/sum(next_word_probs) for x in next_&word_probs]
 #print(sum(next_word_probs))
 samples = np.random.multinomial(1, next_word_probs)
 sampled_word = np.argmax(samples)
 if len(new_sentence) = 300:
  return new_sentence
if len(new_sentence) < min_length:
 return None
return new_sentence

Below is one of the better outputs

SONG_START rain falls at night
two another lover
spinning round and tears
one eye always holds an empty ground
oh, it feels like me
when we get to slow
try to drive its wings
it ‘s the shore
it ‘s bleeding
it ‘s falling tonight
oh, yeah, yeah, yeah
something ‘s wrong, baby
but baby, what are you gonna do ?
on my side
oh, yeah
let’s love up the night

it ‘s just me
the no reason is for
I don’t know where to go
you just can’t ignore the truth
baby, we ain’t got
it ‘s just a day
but it ‘s all inside
and she won’t change one day this has happened and goes
ashes to ashes, off of letting that go
we are this long morning
out of this life that fall breathes
yeah, it ‘s a hell of a bus girl SONG_END

“Songs”

Here are some outputs after 75 epochs of training

SONG_START oh mister what is a good way i know there ‘s one boy in what ‘s missin ‘ on the line one lone leaves plain just like tomorrow ‘s complete and everybody ‘s forgotten it ‘s wrong cause you meet the wife a dime and we turn them boys hoping , america , we mean that i do n’t try you cause i remember i came to remember baby i found my gal down low and leavin ‘ time but comin ‘ to the saddle ‘s door , i was movin this trailer i.d a jar gets calling closed tv and the love that he spoke are on the back roads by the land of a bed he wakes me up with mary suitcase yes my prayers do n’t pity the best , my friends a little getaway on old pill to paradise i ‘m living for the lord to death when mother calls us along at night . and on help me one day you ‘ll say i ‘m goin ‘ back for laughter man , and women always upon the way there ‘ll be long days in steps by the river in a manger no crib forty by a jail memphis calls how warnin ‘ a minute there for the love was only show for for his wife i did n’t need the rent but “ do n’t take your spoil what kneel on to santa claus is comin ‘ to south alabama father to that christmas morn dance in the morning when the jug that flows tall to heaven i ‘m just a child dear frosty what a life you love for all the children who ‘ve hollered awake i hold back and comb your tender lips thought i could recall all the hurt in texas

There is a lot going on in this one. I see santa claus is comin ‘ to south alabama. Seeing this makes me question how many Christmas songs are in my dataset. Note that this one didn’t produce an end token. It stopped on our 300 length restriction.

SONG_START it is it sure for do no everyday but i ca n’t remember it if you never drinker let yourself you ‘ll cry for you like that yea knowing you ‘ve made everything it hurts i was everywhere that you ‘re feeling there if you were there today she opens those vows to the end you left me like you ‘re in this world god made your time he came home darling , to weep you ‘re too married to you are n’t ashamed ? so you smiled just close your eyes and may be ok , if you wake you and if you feel someone in your southern stops today i knew you was chance to hide but you did n’t dream but since you left up i saw the ring i know , you did n’t care some call too long to know that ( you have never after you cried before you leave your pain here but you ‘ve got to lose my head you ‘ll see it with someone else so you come to a fifteen friend and i love to live once i wonder why i ‘m all with you hey well i would already be worn out where you have nothing you would have says you want that there just a need to love you and i thought you could be some frame all the way that what had so ca n’t love me ’cause all your of grace is commin ‘ back to god you love oh SONG_END

This one seems to have country gospel roots.

SONG_START at every lady an old a-eee SONG_END

Short but sweet.

Here are some outputs after 150 epochs of training

SONG_START i do n’t claim to have a home i ‘ve never seen a mess did n’t you know i would n’t share one choice with bath , livin ‘ in tennessee bed , from a old movie show long but man i ‘ve seen a few i thought they were the driver opened up a lot of things i could let me believe some things at all and if i was here let me tell you i wait i was brought a seven leaves work , fun and fame but i ll never know a damn thing when i try to walk your treat all the same i ‘ve seen all these souvenirs , water like me full of money , makin ‘ plans for a while for ol ‘ damn or a sister a part i know when it comes bring tears to love i ‘m coming home to believe help me what i ‘m gon na do , is it sounds changin ‘ come on and help me wrong , that just wo n’t let me pay , i must feel that ohio ? i ‘m rich at the front of runway lyin ‘ cards , i could run my blue jeans and i ai n’t sticking to done but i ‘ve been gone from off to highway so i think i ‘m bad , i ‘m proud of this ole smile there ‘ll be no more way out high , when i am sure SONG_END 

SONG_START i hope you give them some things that you hold for seeing me so if you want your friends in mind some old lovin ‘ hold me down the line promise you but you ‘ll see your fingers get i play awhile and this wo n’t beat me cry up on it ‘s rain it ‘s going trouble now one of you you only love me then i ca n’t seem to feel my life of sin fallen through the tears turn from the stars with a dark wheel another a bar in the beauty of my dreams , but i do n’t know without you each other sleep all the night and hope that you wo n’t curse the sunshine out of a dream i ca n’t live without you you wo n’t ever want to do you always take one of that dreams to love you all alone if you had so many if your sweetheart is all gone anyhow as i recall , no , you ‘re always lonely as for i ca n’t give myself remember those misery love times oh it will be alright and then i ‘ll try a lover is found friend you got a song free fuel on your heart and we can get your heart off of a do n’t lie , i ‘m letting go around in your eyes well i guess i ‘ll just be there i ‘m a little too deep SONG_END
A bigger model and trained for longer that also captures new line characters:

SONG_START if you ‘ll ever make up being true
 you know i ‘d miss and tear up
 there ‘s just so good for you , a good way to fall apart
 what a fool , it wo n’t
 i ca n’t be without you

 if only you care at the heart i made
 ’cause i know i ‘m the one thing through in
 while i ‘m losing laying alone
 oh , but forget about you
 cause god just once were in vain
 so darling , when you ‘ve met me SONG_END

 

SONG_START daylight dark in springtime , lil there ‘s room from my woman
when you sleep in a tree beautiful her over the sun
that ’ s beautiful angel , i ‘m leavin ‘ on a frame ?
i ‘ll dream of you tonight , when shadows look on me ;
a moon on the face sand , even shine hold her eyes
until i find the way to look back woman my life
now i want you by my side
in the morning when i carry on SONG_END

 

SONG_START how i ‘d love holy moonlight tonight
the night is goin ‘ down
and the grass could make us stronger
when the sun lights in your window and my head is cold
can you find a place to say
the day rush and the air is fallin ‘
and the heartache ‘s and taken
and your eyes are drivin ‘ me insane
but baby if you ca n’t see me making your blues
tell her i ‘m sorry
tell me that you can be slow
baby do n’t you leave me now SONG_END

 

SONG_START i ‘m going to drop in these streets
for every one i be
i long to spend it all lay down the streets
and show you the truth
that this life still remains
we can be hard to be alive it keeps , the rhythm and the rain
will you take me back will you make it
i ‘ll breathe in our little beat
i ‘ll be coming back to you
because we all waiting together
we will see the part of our world
is always ending on a leap of feet
as a long journey earth would have to we have it all right with you
i will stand constantly
look about it all i want to be , let it get is more than i love
oh , i wo n’t fight the truth for you
because then i lead the making worst
and if i give your faith for you SONG_END

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s