: The source code for inference and model definitions is available on and the model weights can be found on Hugging Face 2. Architectural Highlights Causal Decoder-Only
The original source code for Falcon 4.0 (released in 1998) was unofficially leaked in April 2000 following the closure of the internal development team by Hasbro Interactive. falcon 40 source code exclusive
Most LLMs freeze their vocabulary post-training. Falcon 40’s source code shows a runtime flag ( --merge_on_the_fly ) that allows the model to infer new subwords by analyzing the input prompt’s entropy. This explains why Falcon 40 has historically scored higher on code generation benchmarks without a fine-tune; it adapts its token boundaries to syntax. : The source code for inference and model
This means you can run Falcon 40B for unlimited conversations on a single A100 80GB without OOM errors. Falcon 40’s source code shows a runtime flag
Perhaps the most valuable find in the is the distributed training scheduler. TII trained Falcon on a massive cluster of AWS Inferentia2 chips (not just NVIDIA). The source code includes a fault-tolerance protocol called CriticalCheckpoint .
in Abu Dhabi around May and June 2023. While initially released under a more restrictive license, the team quickly pivoted to the Apache 2.0 license