All terms
Architectures
Encoder-Only
An architecture that reads a whole input with bidirectional attention to build rich representations.
Definition
An encoder-only model, such as BERT, processes an entire input at once with bidirectional attention, so every token can attend to every other token. This produces rich contextual representations well suited to classification, named entity recognition, retrieval, and embeddings. These models are not built to generate long text the way decoder-only models are, but they are often smaller and faster for understanding tasks.