Should I run Llama 70B on an NVIDIA H100 or A100?

Vishnu Subramanian

Founder @JarvisLabs.ai

H100 offers roughly 2-3x faster performance for Llama 70B inference compared to A100, but at a higher cost. Choose H100 for maximum performance and future-proofing; choose A100 for better cost-effectiveness if raw speed isn't critical.

Performance Comparison

When running Llama 70B with sufficient batch sizes, the H100 approximately halves the latency compared to the A100. This performance advantage comes from several architectural improvements:

• Tensor Cores: The H100 features fourth-generation Tensor Cores that deliver up to 4x the performance compared to the A100's third-generation cores.
• Memory Bandwidth: The H100 SXM has HBM3 memory that provides nearly 2x bandwidth increase over the A100, which is crucial for handling large models.
• FP8 Precision: H100 is optimized for transformer-based models like Llama 70B due to improved tensor core operations and FP8 support - the A100 lacks native FP8 acceleration.

These improvements translate to real-world gains. Inference performance comparison for Llama2 70B shows that H100 is approximately 4 times faster than A100 in some benchmarks.

Memory Considerations

Llama 70B is huge: its 140 GB of parameters won't fit on a single A100 or H100 GPU unless you use quantization.

Your options are:

• Use multiple GPUs with tensor parallelism
• Apply quantization techniques

With quantization techniques (such as 8-bit quantization using libraries like Hugging Face's bitsandbytes), you can significantly reduce memory usage, allowing deployment of Llama 70B on a single A100 or H100 80 GB GPU.

Cost Analysis

The performance boost of the H100 comes at a premium:

• H100 Cost: NVIDIA H100 pricing starts at USD $29,000 but can go up to USD $120,000 depending on your required server configurations
• Cloud Pricing: You can rent a virtual machine with one NVIDIA H100 on Jarvislabs for $2.99/hr or one NVIDIA A100 for $1.29/hr (prices vary by provider)

According to benchmarks by NVIDIA and independent parties, the H100 offers double the computation speed of the A100. This means engineering teams can iterate faster if workloads take half the time to complete. Interestingly, even though the H100 costs about twice as much as the A100, the overall expenditure via a cloud model could be similar if the H100 completes tasks in half the time.

When to Choose H100

I'd recommend the H100 if:

• You need maximum performance: The H100 significantly outperforms the A100 for LLM inference, especially for larger models > 70B models and real-time applications
• Your applications are latency-sensitive: For user-facing applications where response time matters
• You want future-proofing: You want future-proof hardware with FP8 precision support and improved efficiency
• Budget isn't your primary concern: When performance trumps cost considerations

When to Choose A100

The A100 remains an excellent choice if:

• Cost-effectiveness is crucial: You need a well-balanced, cost-effective solution with strong industry adoption
• You can leverage optimizations: If you're comfortable with quantization and other memory-saving techniques
• Performance isn't absolutely critical: For non-real-time applications where some latency is acceptable
• You have existing A100 infrastructure: If you're already invested in A100s

My Recommendation

Having bootstrapped Javis Labs, I know the importance of balancing performance with cost. Here's my take:

If you're running production services where user experience depends on fast responses, invest in H100s - the 2-3x speedup justifies the cost.

However, if you're in research, development, or running batch workloads, A100s still offer tremendous value. We still use A100s, A6000s for our internal development and non-critical workloads.

Remember that optimization techniques can significantly boost performance on either hardware. We've managed to get Llama 70B running with INT8 quantization on a single A100, but the quality trade-offs were noticeable compared to our H100 setup.

What's your specific use case? I can help you think through the particular constraints and requirements that might tip the scales one way or the other.