Results Are In at GECCO 2026: The Benchmark Design Challenge for Continuous Optimisation

Results are in — GECCO 2026 Benchmark Design Challenge for Continuous Optimisation. This note has been updated from “entered” to “scored.” I participated as boris_almonacid, one of six entries. AC_TUWien was the outright winner; my score tied for second-best of the six, though only first place carried an official award — there’s no diploma or certificate for this one, just the result.

Most competitions in this corner of computer science ask you to solve an optimization problem. The BenchDesign competition, organized by Diederick Vermetten and Niki van Stein for GECCO 2026 (July 13–17, San José, Costa Rica, hybrid), flips that around: the task is to design one.

Entrants submit a set of 25 deterministic benchmark functions — bounded to [-5, 5], scalable across dimensions, written to a fixed function signature — and they’re scored not on how solvable the functions are, but on how well they separate five reference optimizers (CMA-ES, Differential Evolution, PSO, BFGS with restarts, and COBYLA), each working within a budget of 5,000 evaluations per problem. A good submission isn’t an easy problem or an impossible one — it’s one where the five algorithms actually disagree, revealing something real about what makes optimization landscapes hard, diverse, or distinctive for a given method. Full technical details live on the IOHprofiler BenchDesign track page, with the reference implementation in Vermetten’s BenchDesign repository.

The submission — Experiment: The Benchmark Design Challenge for Continuous Optimisation — is published on Figshare alongside the rest of the work on this site.

What the competition actually measures

The score is a rank-distribution metric: for each of the 25 submitted problems, the five reference solvers are ranked 1 (best) through 5 (worst), and a good suite is one where those ranks spread out — different algorithms winning and losing on different problems — rather than one solver dominating everything. Submissions are scored twice: once during training, against the exact solver configurations organizers published in advance, and once during a held-out test, where those same solver families reappear under different parameterizations the organizers kept hidden until after submissions closed — CMA-ES becomes a reconfigured ParametrizedCMA, DE becomes quasi-oppositional QODE, PSO becomes ConfiguredPSO, BFGS is swapped for the derivative-free Powell, and COBYLA becomes RCobyla with restarts. The gap between a suite’s train score and its test score is effectively a direct measurement of overfitting to the specific algorithm settings used during development.

Results

Team	Train	Test	Change
AC_TUWien (1st)	2.6	2.8	+0.2
boris_almonacid (tied 2nd-best score)	2.2	1.8	−0.4
AndrewGourley (tied 2nd-best score)	2.2	1.8	−0.4
rohit_salgata	0.6	1.4	+0.8
Ananta_Shahane	2.1	1.0	−1.1
jakub_kudela	2.8	1.0	−1.8

The competition only made one award, to the outright winner, AC_TUWien — there’s no runner-up prize or certificate here, just the scoreboard. By that scoreboard, my submission tied with AndrewGourley for the second-best test score: a 0.4-point drop from train to test, a moderate rather than catastrophic generalization gap. AC_TUWien is the standout result regardless: their score improved from train to test (2.6 → 2.8), meaning their suite kept — and arguably sharpened — its ability to tell algorithms apart even once the specific solver configurations changed underneath it. Two entries didn’t survive the transfer nearly as well: jakub_kudela had the best training score of anyone (2.8) and collapsed to 1.0 on test, and Ananta_Shahane fell from 2.1 to 1.0. Both look like suites tuned to the exact training configurations rather than to the underlying algorithmic behavior those configurations represent — which is exactly the overfitting failure mode this competition is designed to expose.

Why the submission held up reasonably well

Digging into the per-algorithm rank distributions (not just the aggregate score) explains where my 1.8 came from and why it didn’t collapse the way jakub_kudela’s or Ananta’s did:

• CMA-ES → ParametrizedCMA: in training, CMA-ES was spread across several rank positions. Under the reconfigured ParametrizedCMA, it concentrates heavily in rank 5 — many of the 25 functions turn out to be specifically unfavorable to that parameterization. Still a clear differentiation, just a less internally diverse one: ParametrizedCMA mostly just loses, rather than moving around.
• COBYLA → RCobyla: COBYLA sat mostly in ranks 3–4 during training, and RCobyla (with restarts) stayed in largely the same intermediate/poor band on test. A fairly stable family behavior — restarts shift the distribution somewhat but don’t transform its role.
• BFGS → Powell: BFGS was fairly spread out across ranks in training; Powell — a derivative-free local search swapped in for BFGS’s gradient-based one — stayed spread out too on test. That the suite doesn’t collapse this swap into a single rank suggests it’s distinguishing behaviors within local search, not just separating local from global methods.
• DE → QODE: DE and QODE are the one family that keeps a consistent visual identity between train and test charts. Both show a wide spread across good and bad ranks — this looks like the strongest-transferring component of the suite, since the evolutionary family never gets pinned to one position regardless of which specific variant is running.
• PSO → ConfiguredPSO: the biggest shift. PSO showed up in rank 5 often during training (though not exclusively); ConfiguredPSO on test concentrates much more tightly in ranks 1–3, almost disappearing from rank 5. Reconfiguring PSO changed its relationship to the suite substantially — the suite characterized the training PSO well, but that characterization didn’t fully carry over to the reconfigured version.

AC_TUWien’s winning edge looks structural rather than accidental: in their test-set distributions, each of the five reconfigured solvers occupies a more distinct profile — ParametrizedCMA heavy in rank 5, RCobyla dominant in rank 4, Powell strong in rank 1, ConfiguredPSO concentrated in ranks 2 and 4, QODE spread across extremes — five distributions that overlap less and cover more distinct roles between them. That’s a higher-complementarity suite, and it’s a reasonable explanation for why their score went up under reconfiguration while everyone else’s went flat or down.

The overall takeaway: this submission generalized reasonably from train to test and matched the second-best score in the field, but the gap to first wasn’t really about difficulty — AC_TUWien’s suite gave every reconfigured algorithm a more differentiated rank signature than this one did, and theirs was the only result the competition formally recognized.