AIThis post was created with the assistance of artificial intelligence (AI).

🔍 Read the full analysis: Why Your Fine-tuning Job Takes Longer Than The FLOPs Say on ThorstenMeyerAI.com

Before you orderOffer from Amazon

Get the latest gadgets delivered free with Prime

  • Fast, free delivery on millions of items
  • Prime Video, Amazon Music and more included
  • Member-only deals all year
Start your free Prime trial Free trial for eligible customers · Cancel anytime
As an affiliate, we earn on qualifying purchases.

TL;DR

Hugging Face says a CatBoost model predicted the duration of 272 production fine-tuning jobs with a 1.71-hour mean absolute error in five-fold cross-validation. The best tested FLOP-based estimate had a 12.41-hour error, but the result comes from one production environment and has not yet been validated in a live scheduling trial.

Hugging Face reports that a CatBoost model predicted the wall-clock duration of 272 supervised fine-tuning jobs on shared NVIDIA H200 GPUs with a mean absolute error of 1.71 hours in five-fold cross-validation. The original analysis of why fine-tuning jobs take longer than FLOPs suggest compares this with the best tested FLOP-based estimate, which had a 12.41-hour error—a difference that could help teams plan GPU schedules, though the results come from one production pool and do not show savings from a deployed scheduler.

The jobs accounted for about 7,382 GPU-hours, with runtimes ranging from 1.6 to 172.4 hours. The dataset included model families up to 15 billion parameters, LoRA and full fine-tuning, different dataset sizes, and jobs assigned one, two or four GPUs. The H200s came from a shared pool of 8 to 32 GPUs across one to four nodes, where other training and inference workloads ran alongside the fine-tuning jobs.

Hugging Face defines runtime as the time from when a job receives its GPUs until it finishes; queue time is excluded. The predictor used information available before dispatch, such as model metadata, fine-tuning method, token volumes per GPU, evaluation cadence and checkpoint settings. It also included a FLOP-based duration estimate as one of its inputs. After feature selection and tuning, CatBoost used 24 features.

In the reported five-fold cross-validation, CatBoost had an R² of 0.98 and a root mean squared error of 3.52 hours; its median absolute error was 0.51 hours. The FLOP baseline’s best tested result was at 30% Model FLOPs Utilization, or MFU, with an R² of 0.31 and a 12.41-hour mean absolute error. At 45% MFU the error was 13.01 hours, and at 60% it was 14.62 hours. The preview says a wider 25% to 65% MFU sweep did not resolve the baseline’s shortcomings.

At a glance
reportWhen: Study preview reported; full paper fort…
The developmentHugging Face’s study preview reports that a model using job and configuration data predicted fine-tuning runtimes more accurately than tested FLOP-based estimates on 272 production jobs.
AI Training · Briefing

Why Fine-Tuning Takes Longer Than FLOPs Say

Hugging Face tested runtime predictions on 272 production fine-tuning jobs. Configuration-aware estimates were more accurate in this dataset, but results from one production pool do not establish performance elsewhere.

1.71 hours
CatBoost mean absolute error
12.41 hours
Best tested FLOP baseline error
272
Production jobs evaluated
7,382
Approximate GPU-hours

What the estimates capture

The target is time from GPU allocation to job completion; queue time is excluded.

Compute

The training loop

FLOP estimates approximate compute using an assumed Model FLOPs Utilization, or MFU.

Configuration

The whole job

CatBoost uses job settings and model metadata, including evaluations, checkpointing and token volumes.

Shared systems

Other runtime costs

Setup, data handling, network and storage can add time alongside training.

Prediction error in the tested data

Lower error means predicted duration was closer to observed duration on average.

CatBoost1.71 hours
FLOP baseline, 30% MFU12.41 hours
FLOP baseline, 45% MFU13.01 hours
FLOP baseline, 60% MFU14.62 hours

What this could mean for GPU scheduling

More useful duration estimates could help teams plan scarce GPU capacity.

When jobs finish early
  • GPUs may sit idle until another job starts.
  • Operators can misjudge when capacity will be free.
When jobs run over
  • Other work can be delayed.
  • Delivery dates may shift.

How far the result reaches

The preview reports cross-validation on one corporate production environment.

1
Tested on one pool

Runs used shared NVIDIA H200 GPUs and varied configurations.

2
Generalization remains open

Accuracy on other clusters, software stacks and workload mixes is not established.

3
Operational impact remains unmeasured

The preview reports prediction error, not savings in GPU time or scheduling delays.

What to watchThe full paper is forthcoming; validation on independent runs and a live scheduler trial would show whether the result carries over.
The takeaway
Configuration-aware prediction beat the tested FLOP estimates on these jobs, but teams need broader validation before relying on the same accuracy for their own clusters.

More Reliable GPU Job Scheduling

Runtime estimates influence when cluster operators dispatch queued work and how they allocate scarce accelerators. If a job finishes earlier than expected, its assigned GPUs can sit idle until another task starts. If it runs longer, it can delay other jobs and affect delivery schedules. A more accurate estimate could help operators plan around those mismatches and make workload timing less uncertain.

The size of the reported error gap is notable within this dataset, but it is a measure of prediction accuracy, not operational impact. The preview does not report that the model reduced idle GPU time, costs or delays after being put into a scheduler. Whether those benefits occur would depend on how predictions are used and how well they hold up as cluster conditions change.

Products Worth Considering

Logitech MX Creative Console with 9 Customizable Keys, Graphite

Logitech MX Creative Console with 9 Customizable Keys, Graphite

As an affiliate, we earn on qualifying purchases.

Why FLOP Counts Miss Runtime

A common training estimate approximates dense Transformer compute as six times the parameter count and training-token count, then divides by the number of GPUs, their peak throughput and an assumed MFU. MFU represents the share of theoretical peak throughput achieved in practice. The preview places typical MFU in a broad 30% to 60% range, making the choice of assumption one source of uncertainty in a compute-based estimate.

That formula focuses on training computation, while the study’s target covers a job’s GPU-assigned wall-clock duration. Setup, model loading, tokenization, scheduled evaluations, checkpoint uploads and final storage can all add time. Hugging Face says these costs vary with configuration and can compete for network and storage capacity on a multi-tenant cluster. The researchers included scheduled evaluation tokens in the FLOP comparison and tested multiple MFU assumptions, rather than relying on a single setting.

“Everything is known before dispatch.”

— Hugging Face, in the study preview

How Far the Results Travel

The reported scores come from cross-validation within one corporate production dataset, not tests on independently collected clusters or a later, unseen period. The preview does not establish accuracy on other GPU types, software stacks, workload mixes or organizations. It also does not report a live scheduling trial or quantify any resulting change in GPU utilization, cost or job delays.

The full paper has not yet been posted, and the preview gives no release date. Further details about the dataset, evaluation design and feature selection may appear there. It is also not clear how frequently the predictor would need retraining as job configurations and shared-cluster conditions change.

Full Paper and External Tests

Hugging Face says it plans to post a full version on arXiv, but has not given a date. That paper may clarify methodological details and provide additional evidence about how the model was evaluated. The preview does not say whether the researchers plan a deployment test.

For operators to judge whether the result applies beyond this pool, useful next evidence would include tests on runs collected at other times or on different clusters, followed by a live scheduler evaluation. Those checks could show whether the low prediction error persists under changing conditions and whether more accurate estimates translate into fewer idle intervals or delays.

Key Questions

How accurate was the CatBoost runtime model?

In five-fold cross-validation on the 272-job dataset, Hugging Face reports a 1.71-hour mean absolute error, a 0.51-hour median absolute error and an R² of 0.98. These are results on the study’s dataset, not a guarantee for other clusters.

How did it compare with the FLOP-based estimate?

The best tested FLOP baseline, at 30% MFU, had a 12.41-hour mean absolute error. Errors were 13.01 hours at 45% MFU and 14.62 hours at 60% MFU.

What job time does the study predict?

It predicts wall-clock time from when a job receives its GPUs until it finishes. Queue time is not included.

Does the study prove that GPU scheduling improved?

No. The preview reports prediction errors from cross-validation. It does not report a live scheduling trial or measured reductions in idle GPU time, cost or delays.

Can the result be assumed to apply to other GPU clusters?

Not based on the preview alone. The data came from one corporate production environment using shared H200 GPUs, and the model has not been shown to generalize across other clusters, hardware or software setups.

Primary source: Hugging Face · via ThorstenMeyerAI.com

You May Also Like

Tiny Eden – Official Release Date Announcement Trailer

The official release date for Tiny Eden has been revealed in a new announcement trailer, confirming when players can start exploring the game.

The Coding And Cowork Questions Behind Qwen3.8-Max

An IdeaNavigator AI brief frames Qwen3.8-Max as a test case for role-filtered AI monitoring, not as a verified model launch or benchmark report.

Step Into The Future Of AI With Grabette’s Open Robot Data System

Hugging Face introduces Grabette, an open-source handheld device for recording human manipulation demos into robot-ready datasets, aiming to lower data collection costs.

TikTok CEO Sees AI Advancing Science Over The Next Three Years

A CNBC headline attributes a three-year scientific-breakthrough forecast to TikTok CEO Shou Zi Chew, but the supplied material lacks details or supporting evidence.