UniLM

UniLM is a pre-trained language model that handles both understanding and generating text using a single Transformer architecture. Rather than requiring separate models for reading comprehension and text generation, UniLM uses specially designed attention masks to switch between different modes: reading left-to-right, reading both directions, or translating sequences. This makes it useful for tasks like summarising documents, answering questions, and translating between languages. The model is trained on three prediction objectives simultaneously, which helps it develop flexible language skills. It performs comparably to BERT on standard benchmarks like GLUE and SQuAD, whilst also handling generation tasks that BERT cannot do. Researchers and developers can access the pre-trained weights and source code on GitHub, making it practical to integrate into existing NLP pipelines.