Falcon is a family of open-source large language models developed by the Technology Innovation Institute (TII) in Abu Dhabi. First released in March 2023, the Falcon models are built on a decoder-only architecture and are trained on high-quality web data, emphasizing efficiency and performance. The Falcon series includes models of varying sizes, from 1B to 180B parameters, with the largest models achieving competitive results on standard benchmarks while requiring fewer computational resources during inference compared to some contemporaries.
The Falcon models are notable for their use of multi-query attention (MQA) and flash attention, which reduce memory bandwidth and accelerate decoding. They are trained on the Falcon RefinedWeb dataset, a large-scale filtered corpus of public web text. Falcon models have been released under permissive licenses, with the smaller models under the Apache 2.0 license and the larger models under a custom license that restricts commercial use. The models have been widely adopted in the open-source community and have served as the foundation for various fine-tuned variants.
Architecture and Training
Falcon models employ a standard transformer decoder architecture, but with several optimizations. They use rotary positional embeddings and a combination of multi-query attention for the larger models, which shares key and value heads across all query heads, reducing memory usage and improving inference speed. The training data is sourced from the Falcon RefinedWeb dataset, which comprises billions of tokens filtered from Common Crawl, with additional curated sources for some models. The training process uses the AdamW optimizer and a cosine learning rate schedule, with a maximum sequence length of 2048 tokens for most models.
The largest Falcon model, Falcon-180B, was trained on 3.5 trillion tokens using 384 GPUs, and it demonstrates strong performance on reasoning, coding, and general knowledge tasks. Despite its size, Falcon-180B is designed to be efficient, with a focus on reducing the number of parameters used during inference through multi-query attention.
Model Variants and Releases
Falcon models have been released in several sizes and configurations. The initial release in March 2023 included Falcon-1B and Falcon-7B, both under the Apache 2.0 license. In May 2023, Falcon-40B was released, which topped the Open LLM Leaderboard at the time. Later, in September 2023, Falcon-180B was made available, with a custom license that allows free use for research and non-commercial purposes but restricts commercial deployment. Smaller variants, such as Falcon-3B and Falcon-11B, have also been released, with the 11B model using a different attention mechanism (multi-query attention) and achieving notable efficiency.
All Falcon models are available on the Hugging Face Hub, and they have been integrated into various inference frameworks, including vLLM and TensorRT-LLM. The models have also been fine-tuned for instruction following and chat applications, with official variants like Falcon-7B-Instruct and Falcon-40B-Instruct.
Performance and Benchmarks
Falcon models have demonstrated competitive performance on a range of benchmarks. Falcon-40B, for instance, outperformed many existing open-source models on the MMLU (massive multitask language understanding) benchmark, achieving a score of 60.4%. Falcon-180B further improved on these results, scoring 70.4% on MMLU and 68.2% on HellaSwag, and it performed comparably to proprietary models like PaLM-2 Large on several tasks. On coding benchmarks, Falcon-180B achieved a pass@1 score of 19.3% on HumanEval, which is competitive with other large models.
The efficiency of Falcon models is a key selling point. The use of multi-query attention reduces the memory footprint and accelerates inference, making them suitable for deployment on single GPUs. For example, Falcon-40B can be run on a single A100 GPU with quantization, and Falcon-180B requires multiple GPUs but still offers a favorable performance-to-resource ratio.
Impact and Ecosystem
Falcon models have had a significant impact on the open-source AI landscape. They have been used as base models for numerous fine-tuned variants, including instruction-tuned and chat models, and have been integrated into platforms like Hugging Face, Replicate, and AWS SageMaker. The release of Falcon-40B and Falcon-180B helped spur further innovation in efficient large model design, influencing subsequent models such as Llama 2 and Mistral. The Falcon models are also part of TII's broader research efforts in AI, which include work on natural language processing and computer vision.
The permissive licensing of the smaller Falcon models has encouraged widespread adoption in research and commercial applications, while the larger models have been used in academic studies and industry pilots. The Falcon family remains an important reference point in the development of open-source large language models.