On this page
- Key facts
- Is the H100 worth the price for inference?
- Why does memory speed decide generation speed?
- What does FP8 change?
- How much memory does each box have, and which models fit?
- What does each box cost per hour in Canada vs US East?
- When is the A100 enough?
- Can you rent an H100 or A100 in Canada?
- What this comparison does not cover
- FAQ
- Sources
On 11 November 2026, AWS listed eight H100s in Montreal at $63.296 an hour and eight A100s at $25.252: 2.5 times the price. In a test by the AI search firm Perplexity, at matched settings the H100 did 1.8 times the work, so the A100 is cheaper per answer. The H100 pays off only with 8-bit maths and twice the memory.
If your model fits in 320 GB and runs well in 16-bit, the A100 box is the cheaper way to serve it.
If your model needs more than 320 GB, or you can serve it in 8-bit, the H100 box does more work per dollar.
If you need inference in Canada, both are listed in Montreal, and so is the H200 at $72.79 an hour, but plan on a one- to three-year commitment to actually get H100 or H200 capacity.
Key facts
The two cards are NVIDIA data-centre GPUs, one generation apart: Ampere (A100) and Hopper (H100). LLM inference means running a trained large language model to answer requests. GPU memory is the memory on the graphics card itself, and the model's weights and every user's working space have to fit in it. Gigabytes (GB) and terabytes measure it.
| A100 box (AWS p4d.24xlarge) | H100 box (AWS p5.48xlarge) | |
|---|---|---|
| GPUs | 8 x A100 | 8 x H100 |
| GPU memory per GPU | 40 GB HBM2 | 80 GB HBM3 |
| GPU memory per box | 320 GB | 640 GB |
| Memory speed per GPU (NVIDIA) | 1,555 GB/s | 3.35 TB/s |
| 8-bit (FP8) maths | No | Yes |
| On-demand, Montreal, per hour | $25.252 | $63.296 |
| On-demand, US East (N. Virginia), per hour | $21.958 | $55.04 |
Prices are from AWS's price list, read on 11 November 2026. Memory sizes are from AWS's P4 and P5 pages and its instance specifications. Memory speeds are from NVIDIA's A100 datasheet for the 40 GB server card. The H100 figure is from NVIDIA's H100 page, for the 80 GB HBM3 server card that AWS lists.
Is the H100 worth the price for inference?
Only if you use its 8-bit maths or its extra memory. Run the same settings on both and the H100 does less than 2.5 times the work, so it costs more per answer.
The cleanest public test ran on the same two AWS instance families. Perplexity served Llama 2 70B on two AWS boxes in November 2023. The H100 box was a p5.48xlarge. The A100 box was a p4de.24xlarge, which carries the 80 GB card.
Perplexity used NVIDIA's TensorRT-LLM v0.5.0. Each request had 1,024 input and 512 output tokens. A token is about three quarters of a word.
| Perplexity's setup | H100 throughput, relative to A100 |
|---|---|
| Both in 16-bit, 128 requests at once, model split over all 8 GPUs | 184% |
| H100 in 8-bit, same split | 202% |
| H100 in 8-bit, one copy per 2 GPUs, against A100 16-bit over 8 GPUs | 373%, with under 10% more waiting time |
At matched settings the H100 box did 1.84 times the work for 2.5 times the Montreal price. That makes the A100 the cheaper box per answer.
The picture flips in the last row. In 8-bit, Llama 2 70B fits on two H100s, so one box serves four copies at once. Perplexity measured that at 373% of the A100's throughput per GPU. Both boxes hold eight GPUs, so that ratio holds per box too, and 3.7 times the work for 2.5 times the price favours the H100.
A second, newer number comes from Hyperstack, a GPU cloud that rents both cards. Its benchmark page, was updated on 15 December 2025. On the H100 it reports 3,311 tokens a second. The A100 managed 1,148. Its A100s were the 80 GB version. The test served Llama 3.1 70B in vLLM's benchmark suite, with 64 requests at once.
The page does not say what number format or how many GPUs each side used. Treat it as a vendor's direction of travel, not a measurement to plan on.
Why does memory speed decide generation speed?
To write each new token, the GPU reads the model's weights out of its memory. With one user, speed is capped by how fast that memory moves bytes, not by how fast the chip does maths.
NVIDIA lists 1,555 GB/s for the 40 GB A100. The server H100 moves 3.35 TB/s, a bit over twice as fast. That is the ceiling for a single user typing into a chat window.
Serving many users at once changes the sum. One read of the weights can produce a token for every request in the batch, so throughput climbs with batch size until memory runs out. Each request needs its own working space for the conversation so far. More memory left over after the weights means a bigger batch, and the H100's 80 GB per GPU buys exactly that.
What does FP8 change?
The 8-bit Floating Point (FP8) format stores each number in 8 bits instead of 16. The weights take half the memory and half the time to read, and the H100 can do its maths directly in that format, which the A100 cannot.
NVIDIA's A100 datasheet lists no FP8 format. Its H100 page lists FP8 tensor maths at 3,958 teraflops for the server card, a figure NVIDIA quotes with sparsity. Real models rarely reach it.
The A100 can still load an 8-bit model. vLLM's FP8 documentation says FP8 maths needs compute capability 8.9 or higher, which the H100 meets and the A100 does not. Older cards, including the A100, run FP8 models as "weight-only", storing weights in 8 bits and doing the maths in 16. You save the memory but not the compute.
The missing FP8 maths shows up with recent models. Our MiniMax M3.1 Flash vs DeepSeek-V4.1-Flash comparison notes that both models ship part of their weights in 8-bit.
How much memory does each box have, and which models fit?
The A100 box has 320 GB of GPU memory and the H100 box has 640 GB. That number, more than speed, decides which open models you can serve from one machine.
Here are four recent downloadable models, by the size of their files on Hugging Face the same day. File size is the floor: you need that much GPU memory to load the weights, plus room for each user's working space.
| Model | Files on Hugging Face | A100 box (320 GB) | H100 box (640 GB) | H200 box (1,128 GB) |
|---|---|---|---|---|
| MiniMax M3.1 Flash | about 252 GB | Loads, little room left | Yes | Yes |
| DeepSeek-V4.1-Flash | about 510 GB | No | Yes, about 130 GB spare | Yes |
| DeepSeek-V4-Pro | about 865 GB | No | No | Yes |
| Kimi K3.1 Pro | about 1.56 TB | No | No | No |
A model that does not fit on one box can still be served across two, linked over the network. That doubles the hourly bill and adds a slower link inside every token. Our sibling guide on GPU memory to run open models works through those sums, and our Kimi comparison covers that model in full.
The next step up is the H200. AWS's p5en.48xlarge carries eight of them at 141 GB each, 1,128 GB in all, on AWS's instance specifications. AWS's price list put it at $72.79 an hour in Montreal that day, against $63.296 for the H100 box. That is 15% more money for 76% more memory, enough to load the largest DeepSeek model in the table on one machine. For any model past 640 GB, the H200 box is the one to price first.
What does each box cost per hour in Canada vs US East?
The A100 and H100 boxes each cost about 15% more in Montreal than in Virginia. The H100 box runs $63.296 an hour there against $55.04 in Virginia. The A100 box runs $25.252 against $21.958.
| On-demand, AWS price list, 11 Nov 2026 | Montreal (ca-central-1) | US East (us-east-1) | Montreal, per GPU-hour | Montreal, 730 hours |
|---|---|---|---|---|
| p4d.24xlarge (8 x A100 40 GB) | $25.252 | $21.958 | $3.16 | $18,434 |
| p5.48xlarge (8 x H100 80 GB) | $63.296 | $55.04 | $7.91 | $46,206 |
| p5en.48xlarge (8 x H200 141 GB) | $72.79 | $63.296 | $9.10 | $53,137 |
A dedicated box bills every hour it runs, busy or idle. Our post on token-based vs scale-to-zero vs scale-to-one covers when to keep one running and when to switch it off between jobs. Committed-use savings plans cut all three prices; we quote on-demand here because it is the number you pay while deciding.
When is the A100 enough?
The A100 is enough when your model fits in 320 GB with room for the batch you need, and 16-bit maths gives you the quality you want.
- The model is under about 250 GB. A 70-billion-parameter model takes about 140 GB at the usual precision. That leaves the A100 box room for a healthy batch.
- The traffic is steady but modest. A handful of staff, or overnight batch jobs, may never use an H100's extra capacity. At matched settings you pay 2.5 times as much for 1.84 times the work.
- You are still proving the workflow. For evaluation runs before production, the cheaper box answers the same questions.
Pick the H100 when the model is over 320 GB, when you can serve in 8-bit, or when you need a large batch at a fixed response time. For cards between the two, our RTX PRO 6000 vs H100 comparison covers the 96 GB workstation-class option.
Can you rent an H100 or A100 in Canada?
Both are listed in Montreal on AWS and in Canada Central on Azure, and Google lists the H100 in Toronto but no A100 anywhere in Canada. Listed is not the same as available. In our experience deploying for Canadian clients, H100 and H200 capacity in Canadian regions is hard to get on demand, and the providers usually ask for a one- to three-year commitment before they allocate it. A commitment also lowers the hourly price, so the on-demand ratio above is the worst case.
- AWS ca-central-1 (Montreal): the A100 box (p4d.24xlarge) costs $25.252 an hour. The H100 box (p5.48xlarge) costs $63.296. Both prices are from AWS's price list, read on 11 November 2026.
- AWS H200: the p5en (8 x H200) costs $72.79 an hour. The single-GPU p5.4xlarge, one H100, is not offered in Montreal.
- Azure Canada Central: our read of Azure's catalogue the same day listed three series. NC_A100_v4 carries up to four 80 GB A100s. NCads_H100_v5 carries one or two H100 cards with 94 GB each. ND_H100_v5 carries eight 80 GB H100s.
- Azure prices: Azure's retail price list, read the same day, put one A100 (NC24ads_A100_v4) at $4.41 an hour in Canada Central. One H100 (NC40ads_H100_v5) cost $8.59. Eight H100s (ND96isr_H100_v5) cost $117.98, close to twice AWS's Montreal price for the same count. Each runs about 20% above the same machine in Azure's Virginia region.
- Azure's direction: Microsoft's A100 page says new capacity in that family is going only to the NCads_H100_v5 series.
- Google, Toronto (northamerica-northeast2): the H100 in one Toronto zone, plus the smaller L4. Google's GPU regions page lists no A100 in Montreal or Toronto.
Either way, you request GPU quota before your first launch. On Azure it starts at zero. Our pillar on GPUs you can rent in Canada covers the smaller cards too.
XY Space deploys open-weight models on dedicated GPUs in the client's own Canadian cloud account or on-premises, sizes the hardware to the model, and builds the workflow on top. Your people run the work and the model takes part. See how we deploy, the Canadian residency guide and our comparison of on-premise, client cloud and managed cloud. If you need a model running with inference kept in Canada, bring one job to a free 30-minute call.
What this comparison does not cover
We have not run our own benchmark on either box. Perplexity's numbers are from November 2023, on TensorRT-LLM v0.5.0, with the 80 GB A100 that sits in p4de. Montreal's p4d carries the 40 GB card, which leaves less room for the batch. Serving engines have had three years of updates on both cards since. Hyperstack's page leaves out its number format and GPU count. Training is out of scope; this post covers serving a finished model.
FAQ
Is the H100 faster than the A100 for LLM inference?
Yes. NVIDIA lists 3.35 TB/s of memory speed for the server H100. The 40 GB A100 manages 1,555 GB/s. Generating text is mostly limited by memory speed. In Perplexity's 2023 test on AWS, the H100 did 1.84 times the A100's work at matched 16-bit settings. In 8-bit, with smaller model copies, it reached 3.73 times.
Is the H100 worth 2.5 times the price of the A100?
Only in some setups. In Montreal on 11 November 2026, the H100 box cost 2.5 times the A100 box per hour. At matched settings it did less than 2.5 times the work, so the A100 was cheaper per answer. With 8-bit serving, or a model too large for 320 GB, the H100 does more work per dollar.
Does the A100 support FP8?
Not for maths. NVIDIA's A100 datasheet lists no FP8 format, and vLLM's documentation puts FP8 maths at compute capability 8.9 and up, which starts with Ada Lovelace and Hopper. vLLM can still load an FP8 model on an A100 as weight-only: the weights sit in 8 bits, which saves memory, but the maths runs in 16 bits.
How much GPU memory does an 8 x A100 or 8 x H100 server have?
On AWS, the p4d.24xlarge has eight 40 GB A100s, 320 GB in all. The p5.48xlarge has eight 80 GB H100s, 640 GB in all. The p4de uses 80 GB A100s for 640 GB. Azure's NC_A100_v4 tops out at four 80 GB A100s, and its ND_H100_v5 matches AWS's 640 GB.
Can I rent an H200 in Canada?
Yes. AWS rents the p5en.48xlarge, eight H200s with 141 GB each, in Montreal. AWS's price list put it at $72.79 an hour on 11 November. That is about 15% above the H100 box. The extra memory, 1,128 GB against 640 GB, lets one machine load models the H100 box cannot. You still request GPU quota before the first launch, and in our experience H200 capacity in Canada usually needs a one- to three-year commitment.
Sources
- AWS price list (EC2 on-demand, ca-central-1 and us-east-1), read on 11 November 2026: Amazon EC2 On-Demand Pricing
- AWS: Amazon EC2 P4 instances, Amazon EC2 P5 instances and accelerated computing instance specifications, read 11 November 2026
- NVIDIA: A100 datasheet and A100 product page; H100 product page, read 11 November 2026
- Microsoft Learn: NC_A100_v4, NCads_H100_v5, ND_H100_v5; Azure Canada Central SKU list and the Azure retail prices API, read 11 November 2026
- Google Cloud: GPU regions and zones, read 11 November 2026
- Perplexity: Turbocharging Llama 2 70B with NVIDIA H100, 17 November 2023
- Hyperstack: LLM inference benchmark, A100 NVLink vs H100 SXM, updated 15 December 2025
- vLLM: FP8 quantization documentation
- Hugging Face model repositories (file sizes read 11 November 2026): MiniMax M3.1 Flash, DeepSeek-V4.1-Flash, DeepSeek-V4-Pro, Kimi K3.1 Pro
Written by
Cho Yin Yong
Principal AI Solutions Engineer, XY Space
Principal AI Solutions Engineer at XY Space. University of Toronto lecturer for five years, co-author of two patents, winner of two competitive AI awards, and nine years of regulated engineering leadership.
More from Cho Yin YongShare this article
Work with us
We build the systems these posts describe, and we'll tell you in the first call whether yours is worth building.
Start a project