Pre-trained Language Models

Pre-trained language models are models trained extensively on huge text corpora before being fine-tuned for downstream tasks. This gives them general linguistic knowledge that can be transferred and adapted.