Qwen3.8-Flash-Next as a coding agent on the Mac Studio M1 Ultra

The agent gets a repository at the state before the fix and the issue verbatim, plus one sentence: resolve the issue, you may run commands and the existing tests, stop when you are done. It does not get the tests used for scoring. Dependencies are installed; only the model server is reachable.

The tasks come from SWE-bench Verified, from its easiest level: issues that the dataset rates at under 15 minutes for an experienced developer. Eligible are the 87 tasks of this level from Django, sympy, Sphinx and pytest that run under Python 3.9 to 3.11. 40 of them were drawn at random, stratified by repository, with a fixed seed. Nothing was selected by solvability. One Sphinx task could not be checked in the scoring environment and was replaced by the next one drawn.

Scoring follows SWE-bench: the patch the agent leaves in the working tree is applied to a fresh checkout together with the task's tests. A task is solved when the tests for the bug pass (FAIL_TO_PASS) and no previously passing test fails (PASS_TO_PASS). Scoring runs without Docker in a Python environment following the SWE-bench harness, version 4.1.0. Each of the 40 tasks passed there with the reference fix beforehand.

Settings: context up to 262,144 tokens, output up to 32,768 tokens per request, temperature 1.0, top_p 0.95, reasoning effort xhigh, automatic compaction on, prompt cache on the SSD (up to 20 GB, cleared at the start). First all 40 tasks run once, then all of them a second time, in the same order, with one server start for all passes. Nothing else runs.

The agent solves 72 of 80 passes, that is 90.0 %. The 95 % range runs from 80.0 to 97.5 %; it is drawn over tasks, not passes, because two passes of the same task are not independent. The rate holds for the 87 tasks the draw came from, not for SWE-bench Verified as a whole.

The agent solves 35 tasks in both passes, 2 in one and 3 in neither.

G01 (django__django-15127): the agent ends both passes after 4.6 and 8.6 minutes with a patch under which all existing tests pass, but not the test for the bug.

G03 (django__django-15277): solved in the first pass after 3.9 minutes. In the second, the agent ends after 6.9 minutes with a patch under which all existing tests pass, but not the two tests for the bug.

G18 (sympy__sympy-15875): in the first pass the same picture as G01, after 33.5 minutes and 82 steps. The second pass reaches the 120-minute time limit after 131 steps; the patch written by then passes all tests.

M08 (sphinx-doc__sphinx-7462): of the two tests for the bug, one passes in the first pass and none in the second.

M04 (sphinx-doc__sphinx-8595): in both passes the agent creates a test file, tests/roots/test-ext-autodoc/target/empty_all.py, which the task's test patch also creates. The test patch therefore cannot be applied, and the pass counts as not solved.

Half of the passes finish within 10.4 minutes. Django and sympy take a median of 8.8, Sphinx and pytest 12.8 minutes. Some take much longer: the three longest needed 120.0, 91.9 and 59.5 minutes. All 80 passes ran in one go without intervention, in 20.6 hours including scoring.

Every agent step is one request to the server: it reads the new part of the history, usually the output of the last tool, and writes thinking and the next tool call. The passes themselves took 19.9 hours together. Of these, 159 minutes go to reading, 697 to output and 339 to tools, mainly the tests the agent runs itself, and to OpenCode. Output stays almost as fast as the context grows; reading at 64k to 128k is 8 % slower than below 32k.

A pass needs a median of 36 steps. The deepest session reached 146,869 tokens, just over half of the 262,144-token context window. Of 3,591 tool calls, 32 failed.

One pass ended at the time limit: the second pass of G18, after 120 minutes and 131 steps. The patch the agent had written by then passes all tests and therefore counts as solved, as in SWE-bench.

No pass established a connection other than to the model server. The operating system's sandbox blocks every other destination; a network monitor checks this for every pass.