Lecture 2 | Word Vector Representations: word2vec
Why Stack Overflow recommends it
Stack Overflow users link it when explaining word2vec details, such as why each word has separate center and context vectors in skip-gram, and when asking about the lecture's skip-gram matrices.
Recommended by 2 people across 2 different months, 2018 to 2019.
Why Data Science Stack Exchange recommends it
Data Science Stack Exchange users cite it when they need to understand word2vec's loss function, one-hot "localist" representations and how skip-gram training samples are built.
Recommended by 1 people across 1 different months, 2017 to 2019.
What it is
Lecture 2 of Stanford's CS224N Natural Language Processing with Deep Learning course (2017), taught by Chris Manning and Richard Socher. It covers representing words as numeric vectors, including SVD-based methods, skip-gram, CBOW, negative sampling, hierarchical softmax and word2vec.
What people said
Recommends www123 · 2018-04 · Data Science Stack Exchange · on “Skipgram - multiple formulations?”
Andrew Ng word2vec [LINK] @1:29. In the video, his choice of window size is more than one. In his example, if "orange" is the center word, the target word could be "juice", "glass", "my", which are all within the window size of 5. In your example, "cat sat on the mat", if you choose a window size of one, you can get 7 training samples: (cat, sat), (sat, on),(sat, cat), (on, sat), (on, the), (the, on), (the, mat) If this is the entire corpus, p(cat|sat), probability of sat occurs as the center word, cat being the context, is 0.5, p(sat|cat), cat being the center word, sat being the context is 1. The distributed word representation of "cat" and "sat" being the center word should be different.
Recommends Sssssuppp · 2019-02 · Stack Overflow · on “Which loss function should I use in my LSTM and why?”
This [LINK] link should give you an idea as to what cross-entropy does and when would be a good time to use it. Activation functions are used on an experimental basis. There are quite a few activation functions in keras which you could try out for your scenario. Please do refer to this Stanford video [this video] on youtube and this blog [LINK] , these both will provide you with the basic understanding of how the loss function is chosen. Good Luck!
Recommends corio · 2019-08 · Stack Overflow · on “Word shows up more than once in TSNE plot”
Each word has two vectors: as a center word and as a context word. Stanford University word2vec lecture [this video] starting at 41:37.
Cites mausamsion · 2017-10 · Data Science Stack Exchange · on “What are graph embedding?”
What are graph Embeddings ? "Graph Embeddings" is a hot area today in machine learning. It basically means finding "latent vector representation" of graphs which captures the topology (in very basic sense) of the graph. We can make this "vector representation" rich by also considering the vertex-vertex relationships, edge-information etc. There are roughly two levels of embeddings in the graph (of-course we can anytime define more levels by logically dividing the whole graph into subgraphs of various sizes): Vertex Embeddings - Here you find latent vector representation of every vertex in the given graph. You can then compare the different vertices by plotting these vectors in the space and
Cites Bratt Swan · 2018-03 · Stack Overflow · on “What does representation matrix of context word mean in SkipGram?”
I am learning Stanford NLP Course and I have issue understanding a concept in Skipgram from picture below. [LINK] From left to right, the first column vector is one-hot encoder, the second is the word embedding matrix from 1-layer neural network, the third is word representation vector. However, when it comes to the fourth one, which is a matrix with 'v by d' dimension. Not sure if I listen it correctly, but the speaker said this is a representation of context word and these three matrix are identical? My questions are: 1. Why these three matrix are identical but the three multiplication results are different? 2. How do we get this matrix (v by d dimension)? The link to the lecture is: [VIDE
Cites jcp · 2018-04 · Data Science Stack Exchange · on “Skipgram - multiple formulations?”
I've been reading about the Skipgram model and I have found what I interpreted as multiple definitions. 1 - Taking a look at this blog post [LINK] and Andrew Ng's Deep Learning Specialization [LINK] , I understood that, for each word, we generate one training sample for each context word . So, if we have the sentence "cat sat on the mat" we will have samples: (cat, sat) (sat, cat) (sat, on) And so on. Then you train your network which will have the dimensions. Input: $(V,1)$ Weights 1: $(d,V)$ Hidden layer: $(d,1)$ Weights 2: $(V,d)$ Output layer: $(V,1)$ Ok, so these dimensions match and we are good. For one given input, given that the weights are the same, we always have the same estimatio
Cites shaft · 2018-09 · Data Science Stack Exchange · on “Interpretation of the loss function for word2vec”
I am trying to understand the loss function which is used for the word2vec model, but I don't really follow the argumentation behind this video [this video] , at 29:30. The formula which is unclear is the following: $J(\theta) = \displaystyle-\dfrac{1}{T} {\sum_{t=1}^{T} \sum_{-m <= j <=m, \\j\ne0} log(p(w_{t+j}|w_t))}.$ $T$ is the number of words in the vocabulary $w_t$ is a given word and we try to calculate the probablity that another word $w_{t+j}$ occurs within a window of +/- $m$ words ahead. $\theta$ is the solution we're after. It is essentially a $2*d*Tx1$ dimensional vector which contains all the columns of the matrices V and U. At a first sight it looks all pretty clear: we're iteratin
Cites JJJohn · 2019-04 · Data Science Stack Exchange · on “What does localist one-hot vector mean in cs224n NLP course?”
Chris said [this video] one-hot is a "localist" representation. what does "localist" mean here? I've searched on recommended text, didn't find explanation. any clue?