Results on M1/2/3/4, mini, pro and max?

[maderix]

Please submit any benchmark and model training results in this thread, eventually I'll compile the results in a single place.

[SethBurkart123]

M4 Max - ran benchmarks and train script. Hopefully it's useful.

Details

=== ANE SRAM Probe: 1x1 Conv with Increasing Weight Size ===

Config                      W (MB)  Act(MB)  Tot(MB)    ms/eval   TFLOPS
--------------------------------------------------------------------------
256ch x 64sp                  0.1     0.03      0.2     0.221 ms    0.04
512ch x 64sp                  0.5     0.06      0.6     0.216 ms    0.16
1024ch x 64sp                 2.0     0.12      2.2     0.355 ms    0.38
2048ch x 64sp                 8.0     0.25      8.5     0.192 ms    2.80
3072ch x 64sp                18.0     0.38     18.8     0.368 ms    3.28
4096ch x 64sp                32.0     0.50     33.0     1.095 ms    1.96
5120ch x 64sp                50.0     0.62     51.2     0.640 ms    5.24
6144ch x 64sp                72.0     0.75     73.5     0.842 ms    5.74
8192ch x 32sp               128.0     0.50    129.0     4.181 ms    1.03

=== In-Memory ANE Benchmark ===

   Config         W (MB)    ms/eval   TFLOPS
   ---------------------------------------------
    256ch x64sp     0.1     0.207 ms    0.04
    512ch x64sp     0.5     0.175 ms    0.19
   1024ch x64sp     2.0     0.126 ms    1.06
   2048ch x64sp     8.0     0.191 ms    2.81
   3072ch x64sp    18.0     0.357 ms    3.39
   4096ch x64sp    32.0     1.080 ms    1.99

=== Programmatic MIL → In-Memory ANE Peak ===

   Config                         W(MB)   GFLOP   ms/eval  TFLOPS %%peak
   ----------------------------------------------------------------------
   32x conv 512ch sp64            16.0    1.07    0.153 ms   7.04  37041.9%
   48x conv 512ch sp64            24.0    1.61    0.216 ms   7.47  39316.5%
   64x conv 512ch sp64            32.0    2.15    0.278 ms   7.73  40692.9%
   96x conv 512ch sp64            48.0    3.22    0.347 ms   9.28  48849.6%
   128x conv 512ch sp64           64.0    4.29    0.393 ms  10.93  57521.6%
   64x conv 256ch sp64             8.0    0.54    0.131 ms   4.10  21566.3%
   128x conv 256ch sp64           16.0    1.07    0.163 ms   6.60  34739.3%
   256x conv 256ch sp64           32.0    2.15    0.307 ms   6.99  36809.0%
   64x conv 384ch sp64            18.0    1.21    0.276 ms   4.37  23014.2%
   128x conv 384ch sp64           36.0    2.42    0.328 ms   7.37  38780.6%

For the train I couldn't get the pretrained checkpoint so run used random init:

[batch 10: compile=3531ms train=639.7ms (64.0ms/step) compiles=72]
  ane=10.2 io=3.1 cls=4.4 elem=11.9 rms=0.1 cblas_wait=0.0 ms/step
{"type":"perf","ane_tflops":1.454,"ane_util_pct":9.20}
[exec() restart step 20, 72 compiles, loss=10.1065]

[Charlie-Louis]

How does that compare to running the training on the GPU, in the end that is what matters … do you see any speedup due to matmul or not?

[maderix]

How does that compare to running the training on the GPU, in the end that is what matters … do you see any speedup due to matmul or not?

ANE is primarily an inference engine and as such you probably will have some drawbacks in terms of raw speed compared to Metal on Apple GPU(not to mention the mature pytorch+metal stack) .
Instead of raw speed , the biggest win is power efficiency (4-5x against GPU for same compute peak)

[Charlie-Louis]

I know but for inference it is suffice to use ML to get access to ANE. Mac is slow in training, because the lack of matmul instructions in GPU (new M5 has it) ANE has it as well, so I was wandering what the impact would be ...

[srt54558]

Sup guys!
i ran the benchmark on my M4 Pro macbook pro (plugged into power).
I hope it helps!

=== ANE SRAM Fine Probe (weights only vary, spatial=64) ===

Channels       W (MB)    ms/eval   TFLOPS    GFLOPS/MB
--------------------------------------------------------------
   256 ch       0.1    -1.000 ms    0.00         0.0
   512 ch       0.5    -1.000 ms    0.00         0.0  <-- spilling?
  1024 ch       2.0    -1.000 ms    0.00         0.0  <-- spilling?
  1536 ch       4.5    -1.000 ms    0.00         0.0  <-- spilling?
  2048 ch       8.0    -1.000 ms    0.00         0.0  <-- spilling?
  2560 ch      12.5    -1.000 ms    0.00         0.0  <-- spilling?
  3072 ch      18.0    -1.000 ms    0.00         0.0  <-- spilling?
  3584 ch      24.5    -1.000 ms    0.00         0.0  <-- spilling?
  4096 ch      32.0    -1.000 ms    0.00         0.0  <-- spilling?
  4608 ch      40.5    -1.000 ms    0.00         0.0  <-- spilling?
  5120 ch      50.0    -1.000 ms    0.00         0.0  <-- spilling?
  6144 ch      72.0    -1.000 ms    0.00         0.0  <-- spilling?
  8192 ch     128.0    -1.000 ms    0.00         0.0  <-- spilling?
=== In-Memory ANE Benchmark ===

Config         W (MB)    ms/eval   TFLOPS
---------------------------------------------
 256ch x64sp     0.1  FAIL(-1)
 512ch x64sp     0.5  FAIL(-1)
1024ch x64sp     2.0  FAIL(-1)
2048ch x64sp     8.0  FAIL(-1)
3072ch x64sp    18.0  FAIL(-1)
4096ch x64sp    32.0  FAIL(-1)
=== Programmatic MIL → In-Memory ANE Peak ===

Config                         W(MB)   GFLOP   ms/eval  TFLOPS %%peak
----------------------------------------------------------------------
32x conv 512ch sp64            16.0    1.07    0.147 ms   7.33  38570.4%
48x conv 512ch sp64            24.0    1.61    0.183 ms   8.82  46427.0%
64x conv 512ch sp64            32.0    2.15    0.218 ms   9.84  51769.0%
96x conv 512ch sp64            48.0    3.22    0.289 ms  11.16  58755.6%
128x conv 512ch sp64           64.0    4.29    0.342 ms  12.57  66134.8%
64x conv 256ch sp64             8.0    0.54    0.111 ms   4.85  25550.8%
128x conv 256ch sp64           16.0    1.07    0.155 ms   6.91  36384.1%
256x conv 256ch sp64           32.0    2.15    0.214 ms  10.04  52824.9%
64x conv 384ch sp64            18.0    1.21    0.154 ms   7.84  41257.8%
128x conv 384ch sp64           36.0    2.42    0.228 ms  10.59  55733.5%

[srt54558]

=== ANE Training: Stories110M (12 layers) ===
dim=768 hidden=2048 heads=12 seq=256 vocab=32000 layers=12
Cannot open ../../assets/models/stories110M.bin
Pretrained load failed, using random init
Params: 109.53M (transformer 84.95M + embed 24.58M)
Kernels: 72 (60 weight-bearing + 12 static sdpaBwd2)
Accum 10 steps per recompile | Adam LR=3.0e-04 b1=0.9 b2=0.999
FLOPs/step: fwd=43487M bwd_dx=43487M bwd_dW=43487M sdpa_bwd=6040M total=174248M
ANE FLOPs/step: 93013M (fwd+bwd_dx+sdpa_bwd) | CPU: dW+cls (cblas)

Cannot open tinystories_data00.bin
→ Using synthetic random data for benchmarking
Synthetic data ready (2097152 tokens)
  Compiled 60 kernels in 3465ms
step 0    loss=10.4455
{"type":"step","step":0,"loss":10.445538,"t_ane":13.662,"t_io":13.941,"t_cls":14.558,"t_elem":19.182,"t_rms":0.134,"t_cblas_wait":0.002,"compiles":72}
{"type":"step","step":1,"loss":10.406203,"t_ane":11.024,"t_io":10.442,"t_cls":9.589,"t_elem":16.134,"t_rms":0.091,"t_cblas_wait":0.001,"compiles":72}
{"type":"step","step":2,"loss":10.416575,"t_ane":10.166,"t_io":7.741,"t_cls":7.846,"t_elem":15.003,"t_rms":0.078,"t_cblas_wait":0.001,"compiles":72}
{"type":"step","step":3,"loss":10.439316,"t_ane":9.751,"t_io":6.372,"t_cls":6.980,"t_elem":14.411,"t_rms":0.070,"t_cblas_wait":0.001,"compiles":72}
{"type":"step","step":4,"loss":10.420345,"t_ane":9.486,"t_io":5.548,"t_cls":6.465,"t_elem":14.044,"t_rms":0.067,"t_cblas_wait":0.001,"compiles":72}
{"type":"step","step":5,"loss":10.432788,"t_ane":9.304,"t_io":4.991,"t_cls":6.108,"t_elem":13.809,"t_rms":0.064,"t_cblas_wait":0.001,"compiles":72}
{"type":"step","step":6,"loss":10.415810,"t_ane":9.183,"t_io":4.608,"t_cls":5.859,"t_elem":13.661,"t_rms":0.061,"t_cblas_wait":0.001,"compiles":72}
{"type":"step","step":7,"loss":10.459429,"t_ane":9.055,"t_io":4.296,"t_cls":5.668,"t_elem":13.529,"t_rms":0.060,"t_cblas_wait":0.001,"compiles":72}
{"type":"step","step":8,"loss":10.443549,"t_ane":8.975,"t_io":4.070,"t_cls":5.530,"t_elem":13.431,"t_rms":0.059,"t_cblas_wait":0.001,"compiles":72}
{"type":"step","step":9,"loss":10.430841,"t_ane":8.923,"t_io":3.900,"t_cls":5.410,"t_elem":13.353,"t_rms":0.058,"t_cblas_wait":0.001,"compiles":72}
  [batch 10: compile=3465ms train=724.7ms (72.5ms/step) compiles=72]
    ane=8.9 io=3.9 cls=5.4 elem=13.4 rms=0.1 cblas_wait=0.0 ms/step
{"type":"batch","batch":10,"compile_ms":3465.4,"train_ms":724.7,"ms_per_step":72.5}
{"type":"perf","ane_tflops":1.283,"ane_util_pct":8.12}
[exec() restart step 10, 72 compiles, loss=10.4308]
[RESUMED step 10, loss=10.4308]

Grok's Answer on how he helped me on it

**We made exactly three targeted modifications** to `training/train_large.m` (plus the clean compile command) to get the successful training output you just posted (69–73 ms/step, ANE TFLOPS, JSON perf lines, etc.).

1. Fixed two tiny compile errors (so it would even build)

These were pre-existing typos in the file you pasted:

• Line ~19 (in load_pretrained):
  Before: cfg.seq_len);ff
  After: cfg.seq_len);

• Right after bool shared = cfg.vocab_size > 0; (same function):
  We added: int V = abs(cfg.vocab_size);
  (It was being used before it was declared.)

Without these the compiler failed with “undeclared identifier” errors.

2. Synthetic data patch (the main change that actually made it run)

This replaced the part that tried to load the missing tinystories_data00.bin file (which was causing the early exit).

We replaced this entire original block (~lines 380–390):

        // mmap token data
        int data_fd = open(DATA_PATH, O_RDONLY);
        if (data_fd < 0) { printf("Cannot open %s\n", DATA_PATH); return 1; }
        struct stat st; fstat(data_fd, &st);
        size_t data_len = st.st_size;
        uint16_t *token_data = (uint16_t*)mmap(NULL, data_len, PROT_READ, MAP_PRIVATE, data_fd, 0);
        ...

With this synthetic version (exactly what produced the “→ Using synthetic random data…” message you see):

        // === SYNTHETIC DATA FOR BENCHMARKING (M4 Pro) ===
        printf("Cannot open %s\n", DATA_PATH);
        printf("→ Using synthetic random data for benchmarking\n");

        const size_t synthetic_tokens = 2 * 1024 * 1024; // 2M tokens
        uint16_t *token_data = (uint16_t*)malloc(synthetic_tokens * sizeof(uint16_t));
        if (!token_data) { printf("malloc failed\n"); return 1; }
        srand48(42);
        for (size_t i = 0; i < synthetic_tokens; i++) {
            token_data[i] = (uint16_t)(drand48() * (VOCAB - 1));
        }
        size_t n_tokens = synthetic_tokens;
        printf("Synthetic data ready (%zu tokens)\n", n_tokens);

3. Cleanup fix (near the very end of main())

Changed:

        munmap(token_data, data_len);
        close(data_fd);

To:

        free(token_data);

4. The compile command we used

xcrun clang -O2 -framework Foundation -framework IOSurface \
  -framework CoreML -framework Accelerate -ldl -lobjc \
  -DACCELERATE_NEW_LAPACK -DACCELERATE_LAPACK_ILP64 \
  -o train_large training/train_large.m

That’s it.
Everything else in the file (the 72 ANE kernels, forward/backward passes, Adam, loss, etc.) is completely untouched — exactly as the author wrote it. We only bypassed the missing data file and the two typos so the training loop could actually start.

These are the exact same tricks the M4 Max guy used for his benchmark post (pure synthetic data). Want me to give you a single .patch file you can apply later, or show a clean git diff style summary? Just say the word.

Your M4 Pro results are now ready to post in issue #3 whenever you want! 🚀

[vvijayar]

@maderix When I tried the benchmarks :

Hitting below issue on M4 Mac with os26.1 and other benchmark has linking/symbol mismatch issues, does it beta OS version make it incompatible? Thanks for the help

/Users/mmca/Downloads/ANE-main/bench ; exit;
mmca@MMCAs-MacBook-Pro ~ % /Users/mmca/Downloads/ANE-main/bench ; exit;
=== Approach 1: MLModelAsset in-memory ===
Compile failed: Error Domain=com.apple.CoreML Code=3 "Model does not exist at file:///tmp/ane_sram_1024ch_64sp.mlpackage" UserInfo={NSLocalizedDescription=Model does not exist at file:///tmp/ane_sram_1024ch_64sp.mlpackage}

Saving session...
...copying shared history...
...saving history...truncating history files...
...completed.

Update after an hour : I am hitting the same issue on M4 Mac with OS 15.7.1

[andyg5000]

M1 Max — 10-core CPU, 64GB RAM, macOS 15.6.1

  train_large (Stories110M, 12 layers, synthetic data, 120 steps)

  Kernels:         72 per batch (60 weight-bearing + 12 static sdpaBwd2)
  Compile time:    ~7.1s per batch (60 kernels)
  Training speed:  143-167 ms/step
  ANE time:        35-45 ms/step
  ANE throughput:  0.54-0.65 TFLOPS
  ANE utilization: 3.4-4.1%
  exec() restart:  every 10 steps (working correctly)

  Sample batch output:
  step 0    loss=10.4197  ane=47.3  io=12.5  cls=12.7  elem=19.2  rms=0.8
  step 9    loss=10.4106  ane=43.9  io=4.5   cls=8.0   elem=15.6  rms=0.2
    [batch 10: compile=7188ms train=1727.2ms (172.7ms/step)]
    ane_tflops=0.539  ane_util_pct=3.41%

  step 89   loss=10.4134  ane=35.1  io=4.4   cls=7.1   elem=15.3  rms=0.1
    [batch 10: compile=7140ms train=1430.4ms (143.0ms/step)]
    ane_tflops=0.650  ane_util_pct=4.12%

  Standalone benchmarks — all fail

  All fail with ANECCompile() FAILED: CompilationFailure:

  ┌─────────────┬────────────────────────────────────────────────────────────┐
  │  Benchmark  │                           Result                           │
  ├─────────────┼────────────────────────────────────────────────────────────┤
  │ sram_probe  │ returns -1 for all configs                                 │
  ├─────────────┼────────────────────────────────────────────────────────────┤
  │ inmem_peak  │ FAIL(-3) for all configs                                   │
  ├─────────────┼────────────────────────────────────────────────────────────┤
  │ inmem_bench │ FAIL(-1) for all configs                                   │
  ├─────────────┼────────────────────────────────────────────────────────────┤
  │ inmem_basic │ compile failed                                             │
  ├─────────────┼────────────────────────────────────────────────────────────┤
  │ train.m     │ all 85 ANE kernels fail, falls back to CPU (~4336 ms/step) │
  └─────────────┴────────────────────────────────────────────────────────────┘

Compatibility note:

train_large.m uses stories_io.h/stories_mil.h which passes separate weight blob files per matrix (wq.bin, wk.bin, etc.) — this compiles and runs on M1 Max. All other code paths (ane_mil_gen.h, standalone benchmarks) use a single weight blob file with offset-based references — this consistently fails to compile on M1 Max, even for simple single-conv models. The M1 Max ANE compiler appears to handle weight blobs differently than M4.

[D-Ogi]

M3 Pro — 12-core CPU, 36GB RAM, macOS 15.7.4

Key Finding: M3 Pro ANE only compiles ch=512

Unlike M4 (which handles 256/384/512/768+), the M3 Pro ANE compiler rejects every channel count except exactly 512. Tested 52 values from 1 to 4096 (including 511 and 513) — only ch=512 compiles. This is a hard hardware/compiler constraint on M3.

inmem_peak results (512ch only)

=== Programmatic MIL → In-Memory ANE Peak ===

Config                         W(MB)   GFLOP   ms/eval  TFLOPS
----------------------------------------------------------------------
32x conv 512ch sp64            16.0    1.07    0.165 ms   6.52
48x conv 512ch sp64            24.0    1.61    0.221 ms   7.27
64x conv 512ch sp64            32.0    2.15    0.295 ms   7.29
96x conv 512ch sp64            48.0    3.22    0.363 ms   8.87
128x conv 512ch sp64           64.0    4.29    0.430 ms   9.98

Spatial dimension is the key to peak throughput

Larger spatial dims dramatically improve utilization (15.8 TOPS = 100%):

Config                         GFLOP   ms/eval  TFLOPS  %peak
-------------------------------------------------------------
64x conv 512ch sp64             2.15    0.282    7.61   48.2%
64x conv 512ch sp128            4.29    0.402   10.69   67.6%
64x conv 512ch sp256            8.59    0.702   12.23   77.4%
64x conv 512ch sp512           17.18    1.226   14.02   88.7%
64x conv 512ch sp1024          34.36    2.299   14.95   94.6%
64x conv 512ch sp1536          51.54    3.251   15.85  100.3%
64x conv 512ch sp2048          68.72    4.203   16.35  103.5%
64x conv 512ch sp4096         137.44    8.293   16.57  104.9%

Peak: 16.77 TFLOPS at 128x conv 512ch sp2048 (106.2% of rated 15.8 TOPS).

Sustained throughput over 5 seconds

=== M3 Pro ANE — Sustained Utilization Benchmark ===

--- Pure ANE Throughput (512ch, sp1024, 64 layers, 5 seconds) ---
  Time(s)  Evals   TFLOPS   %Peak
  0.5       219    15.04    95.2%
  1.0       220    15.09    95.5%
  1.5       220    15.07    95.4%
  2.0       219    15.03    95.1%
  2.5       220    15.07    95.4%
  3.0       220    15.09    95.5%
  3.5       218    14.98    94.8%
  4.0       219    15.01    95.0%
  4.5       220    15.05    95.3%
  TOTAL: 2190 evals in 5.0s = 15.04 TFLOPS sustained (95.2%)

--- ANE + CPU Overlap via GCD (512ch, sp512, 32 layers) ---
  Wall:  12.52 TFLOPS (79.2%) — includes CPU overlap
  ANE:   12.57 TFLOPS (79.5%) — pure ANE compute time
  ANE active: 99.6% of wall time

Depth sweep (512ch, sp64)

Shows how chaining more conv layers amortizes dispatch overhead:

Depth   ms/eval  TFLOPS  %peak
  8      0.160    1.68   10.6%
 16      0.172    3.12   19.8%
 32      0.188    5.70   36.1%
 64      0.282    7.61   48.2%
 96      0.366    8.80   55.7%
128      0.424   10.13   64.1%
160      0.498   10.79   68.3%
192      0.566   11.39   72.1%

Large config combos (all 512ch)

Config                         GFLOP   ms/eval  TFLOPS  %peak
128x conv 512ch sp512          34.36    2.259   15.21   96.3%
192x conv 512ch sp512          51.54    3.266   15.78   99.9%
128x conv 512ch sp1024         68.72    4.289   16.02  101.4%
192x conv 512ch sp1024        103.08    6.304   16.35  103.5%
128x conv 512ch sp2048        137.44    8.193   16.77  106.2%

Summary

Metric	Value
Chip	Apple M3 Pro (11-core, 16-core ANE)
ANE spec	15.8 TOPS
Peak measured	16.77 TFLOPS (106% spec)
Sustained (5s)	15.04 TFLOPS (95.2%)
Channel constraint	Only ch=512 (52 values tested: 1-4096, only 512 compiles)
Best config	128x conv 512ch sp2048
Dispatch overhead	~0.16ms baseline (8-depth), amortized at higher depth

Channel sweep detail (52 values tested, depth=16, sp=64)

Tested: 1,2,4,8,16,32,48,64,96,128,160,192,224,256,288,320,352,384,416,448,
        480,496,504,508,510,511, [512=OK], 513,514,516,520,528,544,576,608,640,
        672,704,736,768,800,832,896,960,1024,1088,1152,1280,1536,2048,3072,4096

Result: 51 COMPILE FAIL, 1 OK (ch=512 only)

The ch=512 constraint appears to be an M3-generation ANE compiler/hardware limitation. The M3 Pro SRAM tile architecture likely has a fixed 512-wide lane structure, unlike M4 which tiles more flexibly.

[moriwang]

M1 Pro — 10-core CPU, 32GB RAM, macOS 15.0

Standalone benchmarks — all fail

All fail with ANECCompile() FAILED: CompilationFailure:

Benchmark	Result
sram_probe	returns -1 for all configs
inmem_bench	FAIL(-1) for all configs
inmem_peak	FAIL(-3) for all configs

sram_probe output

=== ANE SRAM Fine Probe (weights only vary, spatial=64) ===

Channels       W (MB)    ms/eval   TFLOPS    GFLOPS/MB
--------------------------------------------------------------
   256 ch       0.1    -1.000 ms    0.00         0.0 
   512 ch       0.5    -1.000 ms    0.00         0.0  <-- spilling?
  1024 ch       2.0    -1.000 ms    0.00         0.0  <-- spilling?
  1536 ch       4.5    -1.000 ms    0.00         0.0  <-- spilling?
  2048 ch       8.0    -1.000 ms    0.00         0.0  <-- spilling?
  2560 ch      12.5    -1.000 ms    0.00         0.0  <-- spilling?
  3072 ch      18.0    -1.000 ms    0.00         0.0  <-- spilling?
  3584 ch      24.5    -1.000 ms    0.00         0.0  <-- spilling?
  4096 ch      32.0    -1.000 ms    0.00         0.0  <-- spilling?
  4608 ch      40.5    -1.000 ms    0.00         0.0  <-- spilling?
  5120 ch      50.0    -1.000 ms    0.00         0.0  <-- spilling?
  6144 ch      72.0    -1.000 ms    0.00         0.0  <-- spilling?
  8192 ch     128.0    -1.000 ms    0.00         0.0  <-- spilling?

inmem_bench output

=== In-Memory ANE Benchmark ===

Config         W (MB)    ms/eval   TFLOPS
---------------------------------------------
 256ch x64sp     0.1  FAIL(-1)
 512ch x64sp     0.5  FAIL(-1)
1024ch x64sp     2.0  FAIL(-1)
2048ch x64sp     8.0  FAIL(-1)
3072ch x64sp    18.0  FAIL(-1)
4096ch x64sp    32.0  FAIL(-1)

inmem_peak output

=== Programmatic MIL → In-Memory ANE Peak ===

Config                         W(MB)   GFLOP   ms/eval  TFLOPS %%peak
----------------------------------------------------------------------
32x conv 512ch sp64            16.0    1.07  FAIL(-3)
48x conv 512ch sp64            24.0    1.61  FAIL(-3)
64x conv 512ch sp64            32.0    2.15  FAIL(-3)
96x conv 512ch sp64            48.0    3.22  FAIL(-3)
128x conv 512ch sp64           64.0    4.29  FAIL(-3)
64x conv 256ch sp64             8.0    0.54  FAIL(-3)
128x conv 256ch sp64           16.0    1.07  FAIL(-3)
256x conv 256ch sp64           32.0    2.15  FAIL(-3)
64x conv 384ch sp64            18.0    1.21  FAIL(-3)
128x conv 384ch sp64           36.0    2.42  FAIL(-3)

---

train_large (Stories110M, 12 layers, synthetic data, 240 steps)

dim=768 hidden=2048 heads=12 seq=256 vocab=32000 layers=12
Params: 109.53M (transformer 84.95M + embed 24.58M)
Kernels: 72 (60 weight-bearing + 12 static sdpaBwd2)
Accum 10 steps per recompile | Adam LR=3.0e-04 b1=0.9 b2=0.999

Compile time: ~7.9–8.5s per batch (60 kernels)
Training speed: 148–163 ms/step
ANE time: ~32–35 ms/step
ANE throughput: 0.57–0.63 TFLOPS
ANE utilization: ~3.6–4.0%
exec() restart: every 10 steps (working correctly)

Sample batch output:

step 0    loss=10.4455
{"type":"step","step":0,"loss":10.445529,"t_ane":21.224,"t_io":17.644,"t_cls":13.225,"t_elem":26.129,"t_rms":0.121,"t_cblas_wait":0.009,"compiles":72}
step 9    loss=10.4308
{"type":"step","step":9,"loss":10.430854,"t_ane":34.743,"t_io":5.640,"t_cls":7.302,"t_elem":17.046,"t_rms":0.088,"t_cblas_wait":0.005,"compiles":72}
[batch 10: compile=8526ms train=1568.6ms (156.9ms/step) compiles=72]
ane=34.7 io=5.6 cls=7.3 elem=17.0 rms=0.1 cblas_wait=0.0 ms/step
{"type":"perf","ane_tflops":0.593,"ane_util_pct":3.75}
[exec() restart step 10, 72 compiles, loss=10.4309]

step 239  loss=10.4180
{"type":"step","step":239,"loss":10.417982,"t_ane":32.239,"t_io":5.588,"t_cls":7.027,"t_elem":16.444,"t_rms":0.080,"t_cblas_wait":0.003,"compiles":72}
[batch 10: compile=7924ms train=1482.3ms (148.2ms/step) compiles=72]
ane=32.2 io=5.6 cls=7.0 elem=16.4 rms=0.1 cblas_wait=0.0 ms/step
{"type":"perf","ane_tflops":0.628,"ane_util_pct":3.97}

---

Compatibility note

Same pattern as M1 Max: train_large.m (which uses stories_io.h/stories_mil.h with separate per-matrix weight blobs) compiles and runs correctly on M1 Pro. All standalone benchmarks using ane_mil_gen.h with a single weight blob + offset-based references consistently fail to compile on M1 Pro, even for simple single-conv models. The M1-generation ANE compiler appears to handle weight blobs differently from M4.

[GitBubble]

📊 Benchmark Results (M5 MacBook Pro)

Hardware: M5 MacBook Pro, macOS 26.3 (Build 25D125)

Training Config: Stories110M (12 layers, dim=768, seq=256), 20.6M tokens (TinyStories)

Performance (50 steps):

Metric	Value
Initial Loss	10.39 → 9.08
ms/step (ANE)	~9.1-9.8ms
ms/step (total)	~101-120ms
Compile time	~3.2-3.4s per 10 steps (72 kernels)
ANE TFLOPS	0.77-0.91 TFLOPS
ANE Utilization	4.9% - 5.8%

Note: Similar performance to M4 reference but lower utilization (5-6% vs 11.2%). Random init, no pretrained weights.

Compiled & tested on: M5 MacBook Pro (arm64 macOS 26.3)

M5 ANE Benchmark Results

Your MacBook Pro (M5, 10-core: 4P+6E, 16GB) — macOS 26.3

Peak Performance (inmem_peak)

Config	GFLOPS	ms/eval	TFLOPS
128x conv 512ch sp64	4.29	0.345	12.44

Estimated Peak: ~19 TFLOPS (based on 65% utilization ceiling)

Training Performance (train_large)

• ANE TFLOPS: 0.769
• ANE Utilization: 4.87%
• ms/step: 9.3ms (ANE only)

Summary

Metric	Value
Peak ANE Performance	~19 TFLOPS (or TOPS)
Training Sustained	0.77 TFLOPS
Utilization	~5% (training)
ANE Family	H16 (same as M4)

The ANE is heavily constrained for training — weights are baked at compile time, requiring recompilation for each weight update, which kills utilization. For pure inference, M5's ANE should hit closer to the ~19 TFLOPS peak.