GPT · Llama · Gemini · poloclub.github.io
Most models consist of multiple
Compiled by KHAO Editorial — aggregated from 1 source. See llms.txt for citation guidance.
◌ Single Source
The GPT-2 (small) model Hacker News is examining consists of 12 such blocks.
Key facts
- These embedding vectors are stored in a matrix of shape (50,257, 768), containing approximately 39 million parameters
- Transformer Explainer is powered by the GPT-2 (small) model which has 124 million parameters
- To convert a prompt into embedding, they need to 1) tokenize the input, 2) obtain token embeddings, 3) add positional information, and finally 4) add up token and position encodings to get the final
- The full vocabulary of tokens is decided before training the model: GPT-2's vocabulary has 50,257 unique tokens
Summary
Transformer is a neural network architecture that has fundamentally changed the approach to Artificial Intelligence. Fundamentally, text-generative Transformer models operate on the principle of next-token prediction: given a text prompt from the user, what is the most probable next token (a word or part of a word) that will follow this input? GPT-2 family of models are prominent examples of text-generative Transformers. Let's say you want to generate text using a Transformer model. To convert a prompt into embedding, they need to 1) tokenize the input, 2) obtain token embeddings, 3) add positional information, and finally 4) add up token and position encodings to get the final embedding. Tokenization is the process of breaking down the input text into smaller, more manageable pieces called tokens. The full vocabulary of tokens is decided before training the model: GPT-2's vocabulary has 50,257 unique tokens.