How Word2Vec works?

Before we go to understand Word2Vec, we should understand the meaning of the term ‘Embedding’

Embeddings are the numerical vector representation of an object. In the context of word embeddings, it is a vector representation of a word.

How are these embeddings formed?

Let us say we have a corpus — “I am going to playground for watching cricket”.

Now, here are the steps involved in forming the word embedding for each word in this corpus.

Step 1:

Creation of a mapping where each word in the corpus is represented by input column and basis the window size, ’n’ number of words before that word are represented in output column. This is called skipgram method

Step 2:

Now next step, is to transform this dataset into below form and add some negative samples to the data as well. This is called negative sampling

Step 3:

Now, we initialize two randomly filled matrices — Embedding matrix and Context Matrix

The embedding_size is 300, while vocab_size is the number of unique words in the corpus on which the word2vec model will be trained.

For our case, vocab_size would be 8.

Step 4:

Now, we take dot product of the vector from the embedding matrix for the input word with the vector from context matrix for the output word.

Notice, we have also calculated sigmoid and error with respect to target variable.

Now, the model parameters for embedding and context matrix are updated basis this error.

This is a training step in the word2vec model training.

After all the training iterations are done, we discard the context matrix and the embedding matrix is kept for the further use.

There are two hyperparameters in this model that can be tuned -

1. window_size
2. negative_samples_count

This way a Word2Vector model is trained. We can feed any corpus in the training process to get our embeddings.

These embeddings look like below -

We can now use cosine similarity method to calculate which words are similar to each other by using their vectors.

Some of the applications of Word2Vec:

1. Text Similarity
2. Sentiment Analysis
3. Next Word Prediction
4. Recommendation Engines