🔍 Read the full analysis: Why Your Fine-tuning Job Takes Longer Than The FLOPs Say on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Hugging Face says a CatBoost model predicted the duration of 272 production fine-tuning jobs with a 1.71-hour mean absolute error in five-fold cross-validation. The best tested FLOP-based estimate had a 12.41-hour error, but the result comes from one production environment and has not yet been validated in a live scheduling trial.
Hugging Face reports that a CatBoost model predicted the wall-clock duration of 272 supervised fine-tuning jobs on shared NVIDIA H200 GPUs with a mean absolute error of 1.71 hours in five-fold cross-validation. The original analysis of why fine-tuning jobs take longer than FLOPs suggest compares this with the best tested FLOP-based estimate, which had a 12.41-hour error—a difference that could help teams plan GPU schedules, though the results come from one production pool and do not show savings from a deployed scheduler.
The jobs accounted for about 7,382 GPU-hours, with runtimes ranging from 1.6 to 172.4 hours. The dataset included model families up to 15 billion parameters, LoRA and full fine-tuning, different dataset sizes, and jobs assigned one, two or four GPUs. The H200s came from a shared pool of 8 to 32 GPUs across one to four nodes, where other training and inference workloads ran alongside the fine-tuning jobs.
Hugging Face defines runtime as the time from when a job receives its GPUs until it finishes; queue time is excluded. The predictor used information available before dispatch, such as model metadata, fine-tuning method, token volumes per GPU, evaluation cadence and checkpoint settings. It also included a FLOP-based duration estimate as one of its inputs. After feature selection and tuning, CatBoost used 24 features.
In the reported five-fold cross-validation, CatBoost had an R² of 0.98 and a root mean squared error of 3.52 hours; its median absolute error was 0.51 hours. The FLOP baseline’s best tested result was at 30% Model FLOPs Utilization, or MFU, with an R² of 0.31 and a 12.41-hour mean absolute error. At 45% MFU the error was 13.01 hours, and at 60% it was 14.62 hours. The preview says a wider 25% to 65% MFU sweep did not resolve the baseline’s shortcomings.
Why Fine-Tuning Takes Longer Than FLOPs Say
Hugging Face tested runtime predictions on 272 production fine-tuning jobs. Configuration-aware estimates were more accurate in this dataset, but results from one production pool do not establish performance elsewhere.
What the estimates capture
The target is time from GPU allocation to job completion; queue time is excluded.
The training loop
FLOP estimates approximate compute using an assumed Model FLOPs Utilization, or MFU.
The whole job
CatBoost uses job settings and model metadata, including evaluations, checkpointing and token volumes.
Other runtime costs
Setup, data handling, network and storage can add time alongside training.
Prediction error in the tested data
Lower error means predicted duration was closer to observed duration on average.
What this could mean for GPU scheduling
More useful duration estimates could help teams plan scarce GPU capacity.
- GPUs may sit idle until another job starts.
- Operators can misjudge when capacity will be free.
- Other work can be delayed.
- Delivery dates may shift.
How far the result reaches
The preview reports cross-validation on one corporate production environment.
Runs used shared NVIDIA H200 GPUs and varied configurations.
Accuracy on other clusters, software stacks and workload mixes is not established.
The preview reports prediction error, not savings in GPU time or scheduling delays.
More Reliable GPU Job Scheduling
Runtime estimates influence when cluster operators dispatch queued work and how they allocate scarce accelerators. If a job finishes earlier than expected, its assigned GPUs can sit idle until another task starts. If it runs longer, it can delay other jobs and affect delivery schedules. A more accurate estimate could help operators plan around those mismatches and make workload timing less uncertain.
The size of the reported error gap is notable within this dataset, but it is a measure of prediction accuracy, not operational impact. The preview does not report that the model reduced idle GPU time, costs or delays after being put into a scheduler. Whether those benefits occur would depend on how predictions are used and how well they hold up as cluster conditions change.
Products Worth Considering

Samsung Galaxy Tab A11+ Kids Edition, 128GB, Royal Blue

Samsung 32 Odyssey G5 QHD Gaming Monitor

Logitech MX Creative Console with 9 Customizable Keys, Graphite
As an affiliate, we earn on qualifying purchases.
Why FLOP Counts Miss Runtime
A common training estimate approximates dense Transformer compute as six times the parameter count and training-token count, then divides by the number of GPUs, their peak throughput and an assumed MFU. MFU represents the share of theoretical peak throughput achieved in practice. The preview places typical MFU in a broad 30% to 60% range, making the choice of assumption one source of uncertainty in a compute-based estimate.
That formula focuses on training computation, while the study’s target covers a job’s GPU-assigned wall-clock duration. Setup, model loading, tokenization, scheduled evaluations, checkpoint uploads and final storage can all add time. Hugging Face says these costs vary with configuration and can compete for network and storage capacity on a multi-tenant cluster. The researchers included scheduled evaluation tokens in the FLOP comparison and tested multiple MFU assumptions, rather than relying on a single setting.
“Everything is known before dispatch.”
— Hugging Face, in the study preview
How Far the Results Travel
The reported scores come from cross-validation within one corporate production dataset, not tests on independently collected clusters or a later, unseen period. The preview does not establish accuracy on other GPU types, software stacks, workload mixes or organizations. It also does not report a live scheduling trial or quantify any resulting change in GPU utilization, cost or job delays.
The full paper has not yet been posted, and the preview gives no release date. Further details about the dataset, evaluation design and feature selection may appear there. It is also not clear how frequently the predictor would need retraining as job configurations and shared-cluster conditions change.
Full Paper and External Tests
Hugging Face says it plans to post a full version on arXiv, but has not given a date. That paper may clarify methodological details and provide additional evidence about how the model was evaluated. The preview does not say whether the researchers plan a deployment test.
For operators to judge whether the result applies beyond this pool, useful next evidence would include tests on runs collected at other times or on different clusters, followed by a live scheduler evaluation. Those checks could show whether the low prediction error persists under changing conditions and whether more accurate estimates translate into fewer idle intervals or delays.
Key Questions
How accurate was the CatBoost runtime model?
In five-fold cross-validation on the 272-job dataset, Hugging Face reports a 1.71-hour mean absolute error, a 0.51-hour median absolute error and an R² of 0.98. These are results on the study’s dataset, not a guarantee for other clusters.
How did it compare with the FLOP-based estimate?
The best tested FLOP baseline, at 30% MFU, had a 12.41-hour mean absolute error. Errors were 13.01 hours at 45% MFU and 14.62 hours at 60% MFU.
What job time does the study predict?
It predicts wall-clock time from when a job receives its GPUs until it finishes. Queue time is not included.
Does the study prove that GPU scheduling improved?
No. The preview reports prediction errors from cross-validation. It does not report a live scheduling trial or measured reductions in idle GPU time, cost or delays.
Can the result be assumed to apply to other GPU clusters?
Not based on the preview alone. The data came from one corporate production environment using shared H200 GPUs, and the model has not been shown to generalize across other clusters, hardware or software setups.
Primary source: Hugging Face · via ThorstenMeyerAI.com
