Meta Releases Llama 3 With 8B and 70B Open Models

Llama 3 arrives

Meta introduced Llama 3 on April 18, 2024, calling it the next generation of its open large language model. The first release featured pretrained and instruction-fine-tuned models with 8B and 70B parameters, which Meta described as the best open source models of their class at those scales, and said Llama 3 would soon be available across AWS, Databricks, Google Cloud, Hugging Face, Microsoft Azure, NVIDIA NIM and more.

Architecture and training

Llama 3 used a relatively standard decoder-only transformer with a new tokenizer whose 128K-token vocabulary encodes language more efficiently, yielding up to 15% fewer tokens than Llama 2, and adopted grouped query attention across both the 8B and 70B sizes for better inference efficiency. It was pretrained on over 15 trillion tokens from publicly available sources, a dataset seven times larger than Llama 2's with four times more code, and trained on sequences of 8,192 tokens.

Post-training and evaluation

Meta combined supervised fine-tuning, rejection sampling, proximal policy optimization and direct preference optimization in post-training, which it said substantially reduced false refusal rates and improved reasoning and code generation. The company built a new human evaluation set of 1,800 prompts across 12 use cases, withheld even from its own modeling teams, and reported strong preference rankings for the 70B instruction model against comparable models.

Safety and what's next

The release added new trust and safety tools including Llama Guard 2, Code Shield, an inference-time guardrail for filtering insecure code, and CyberSec Eval 2. Meta co-developed Llama 3 with torchtune, a new PyTorch-native fine-tuning library, and previewed that models over 400B parameters were still training, with multimodality, longer context and multiple languages planned in the coming months.

Related on Skillo

See also: Llama 3.1 and the 405B frontier model, PyTorch 2.0 and torch.compile.

Sources

• Introducing Meta Llama 3 (ai.meta.com)

Published date reflects the original event date (2024-04-18). This article is original Skillo editorial written from the sources above; facts were verified in September 2026.