BERT Document Classification Tutorial with Code
Why Stack Overflow recommends it
Stack Overflow users link it as a hands-on reference for fine-tuning BERT on whole documents, for document embeddings, and for semantic similarity search.
Recommended by 0 people across 0 different months, 2020 to 2021.
What it is
A 43-minute tutorial by Chris McCormick showing how to fine-tune BERT for document classification, using the Wikipedia Personal Attacks dataset as the worked example. A bonus section applies BERT embeddings to search for semantically similar comments. It builds on his earlier BERT fine-tuning tutorial with PyTorch.
What people said
Cites chrismcc · 2020-01 · Stack Overflow · on “How can I use BERT for long text classification?”
This paper compared a few different strategies: How to Fine-Tune BERT for Text Classification? [LINK] . On the IMDb movie review dataset, they actually found that cutting out the middle of the text (rather than truncating the beginning or the end) worked best! It even outperformed more complex "hierarchical" approaches involving breaking the article into chunks and then recombining the results. As another anecdote, I applied BERT to the Wikipedia Personal Attacks dataset here [this video] , and found that simple truncation worked well enough that I wasn't motivated to try other approaches :)
Cites user2543622 · 2020-04 · Stack Overflow · on “BERT embedding for semantic similarity”
I earlier posted this question [LINK] . I wanted to get embedding similar to this [this video] video, time 33 minutes onward. 1) I dont think that the embedding that i am getting from CLS token are similar to what is shown in the youtube video. I tried to perform semantic similarity and got horrible results. Could someone confirm whether embedding that i am getting are similar to embedding mentioned at 35.27 mark of the video? 2) If the answer of the above question is 'not similar' then how could i get the embedding that I am looking for using the code that i have written? 3) If the answer of the 1st question is 'they are similar' then why am i getting horrible results? do i need to finetune usin
Cites MRM · 2020-08 · Stack Overflow · on “BERT document embedding”
I am trying to do document embedding using BERT. The code I use is a combination of two sources. I use BERT Document Classification Tutorial with Code [this video] , and BERT Word Embeddings Tutorial [LINK] . Below is the code, I feed the first 510 tokens of each document to the BERT model. Finally, I apply K-means clustering to these embeddings, but the members of each cluster are TOTALLY irrelevant. I am wondering how this is possible. Maybe something is wrong with my code. I would appreciate if you take a look at my code and tell if there is something wrong with it. I use Google colab to run this code. # text_to_embedding function import torch from keras.preprocessing.sequence import pad_seque
Cites mon · 2021-07 · Stack Overflow · on “Training TFBertForSequenceClassification with custom X and Y data”
Fine Tuning Approach There are multiple approaches to fine-tune BERT for the target tasks. Further Pre-training the base BERT model Custom classification layer(s) on top of the base BERT model being trainable Custom classification layer(s) on top of the base BERT model being non-trainable (frozen) Note that the BERT base model has been pre-trained only for two tasks as in the original paper. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding [LINK] 3.1 Pre-training BERT ...we pre-train BERT using two unsupervised tasks Task #1: Masked LM Task #2: Next Sentence Prediction (NSP) Hence, the base BERT model is like half-baked which can be fully baked for the target
Cites mon · 2021-09 · Stack Overflow · on “How to Fine-tune HuggingFace BERT model for Text Classification”
Fine Tuning Approach There are multiple approaches to fine-tune BERT for the target tasks. Further Pre-training the base BERT model Custom classification layer(s) on top of the base BERT model being trainable Custom classification layer(s) on top of the base BERT model being non-trainable (frozen) Note that the BERT base model has been pre-trained only for two tasks as in the original paper. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding [LINK] 3.1 Pre-training BERT ...we pre-train BERT using two unsupervised tasks Task #1: Masked LM Task #2: Next Sentence Prediction (NSP) Hence, the base BERT model is like half-baked which can be fully baked for the target