GPT-J

GPT-J

Logo
Developer(s)	EleutherAI
Initial release	June 9, 2021; 3 years ago (2021-06-09)
Type	Large language model Generative pre-trained transformer Foundation model
License	Open-source
Website	6b.eleuther.ai

GPT-J or GPT-J-6B is an open-source large language model (LLM) developed by EleutherAI in 2021.^[1] As the name suggests, it is a generative pre-trained transformer model designed to produce human-like text that continues from a prompt. The optional "6B" in the name refers to the fact that it has 6 billion parameters.^[2]

Architecture

GPT-J is a GPT-3-like model with 6 billion parameters.^[3] Like GPT-3, it is an autoregressive, decoder-only transformer model designed to solve natural language processing (NLP) tasks by predicting how a piece of text will continue.^[1]

Its architecture differs from GPT-3 in three main ways.^[1]

• The attention and feedforward neural network were computed in parallel during training, allowing for greater efficiency.
• The GPT-J model uses rotary position embeddings, which has been found to be a superior method of injecting positional information into transformers.^[4][5]
• GPT-J uses dense attention instead of efficient sparse attention, as used in GPT-3.

Beyond that, the model has 28 transformer layers and 16 attention heads. Its vocabulary size is 50257 tokens, the same size as GPT-2's.^[2] It has a context window^{[broken anchor]} size of 2048 tokens.^[6]

It was trained on the Pile dataset,^[2][3] using the Mesh Transformer JAX library in JAX to handle the parallelization scheme.^[2][7]

Performance

GPT-J was designed to generate English text from a prompt. It was not designed for translating or generating text in other languages or for performance without first fine-tuning the model for a specific task.^[2] Nonetheless, GPT-J performs reasonably well even without fine-tuning, even in translation (at least from English to French).^[8]

When neither is fine-tuned, GPT-J-6B performs almost as well as the 6.7 billion parameter GPT-3 (Curie) on a variety of tasks.^[3] It even outperforms the 175 billion parameter GPT-3 (Davinci) on code generation tasks.^[9] With fine-tuning, it outperforms an untuned GPT-3 (Davinci) on a number of tasks.^[1]

Like all LLMs, it is not programmed to give factually accurate information, only to generate text based on probability.^[2]

Applications

The untuned GPT-J is available on EleutherAI's website,^[10] NVIDIA's Triton Inference Server,^[11] and NLP Cloud's website.^[12] Cerebras^[1] and Amazon Web Services^[13][14] offer services to fine-tune the GPT-J model for company-specific tasks. Graphcore offers both fine-tuning and hosting services for the untuned GPT-J, as well as offering to host the fine-tuned models after they are produced.^[15] CoreWeave offers hosting services for both the untuned GPT-J and fine-tuned variants.^[16][17]

In March 2023, Databricks released Dolly, an Apache-licensed, instruction-following model created by fine-tuning GPT-J on the Stanford Alpaca dataset.^[18] NovelAI's Sigurd^[19] and Genji-JP 6B^[20] models are both fine-tuned versions of GPT-J. They also offer further fine-tuning services to produce and host custom models.^[21]

EleutherAI has received praise from Cerebras,^[1] GPT-3 Demo,^[3] NLP Cloud,^[12] and Databricks^[18] for making the model open-source, and its open-source status is often cited as a major advantage when choosing which model to use.^[9][15][22]

References

This page was last edited on 11 June 2024, at 02:56