Skip to content

Meta

Llama 3.1 405B Instruct

405.9 B params · 128000 context · Open weights

Released 2024-07-16

Details

Provider
Meta
Family
Llama 3.1
Context length
128000 tokenssource
Open weights
Yessource
Licence
Llama 3.1 Community Licensesource
Parameters (total)
405.85 Bsource

API pricing

No hosted API price verified. Open-weight models can be self-hosted.

Running it locally

Memory needed for 405.853 B parameters at 8,192 tokens of context. Architecture not recorded, so a flat 1.0 GB KV allowance is used. Estimates, not measurements.

FormatWeightsKV cacheOverheadTotal
Q4 K_M248.1 GB1.0 GB12.9 GB262.0 GB
Q6 K332.9 GB1.0 GB17.1 GB351.1 GB
Q8 0431.2 GB1.0 GB22.1 GB454.3 GB
FP16811.7 GB1.0 GB41.1 GB853.8 GB

Catalogued hardware that fits at Q4

No catalogued hardware has enough memory yet, or none is listed.

Mini PCs reserve 8.0 GB for the operating system. “Comfortable” keeps at least 10% memory spare.

Benchmarks

No benchmarks for this model yet

We only publish results we can evidence, so there is nothing here yet and we have not filled the gap with estimates. Here is how we will test and label results.