Multiplication Without Transistors: An Audit of Photonic Matrix Processors as Accelerators for Large Language Model Training

Training a frontier LLM is, arithmetically, a very long sequence of matrix multiplications. The standard estimate for a dense transformer is that training cost ≈ 6·N·D floating-point operations, where N is parameter count and D is the number of training tokens — roughly 2ND for the forward pass and 4ND for backpropagation. For a model in the 10¹¹-parameter class trained on 10¹³ tokens, this is on the order of 10²⁵ FLOPs, executed almost entirely as general matrix–matrix multiplications (GEMMs) inside attention and feed-forward blocks. At current GPU efficiencies of order 10¹⁰–10¹¹ FLOP/J at training precision, the arithmetic alone implies gigawatt-hour energy scales before accounting for memory movement, which typically dominates.

Three physical facts frame the opportunity:

• —Digital multiplication is expensive by construction. A digital b-bit multiplier performs O(b²) full-adder operations, each of which charges and discharges transistor gates. The energy of an 8-bit MAC in modern CMOS is of order 10–100 fJ; the interconnect energy to move the operands often exceeds the arithmetic itself.
• —Linear optics performs linear algebra natively. Any lossless linear-optical network implements a unitary transformation on the optical field. Interference is addition of amplitudes; a beam splitter is a 2×2 unitary; modulation is multiplication by a scalar. The mathematics of the layer we are trying to compute and the physics of the medium are the same mathematics.
• —Propagation costs no switching energy. Once the field is launched, the "computation" — the redistribution of amplitude among output modes — dissipates no energy in the ideal lossless case. The energy cost concentrates entirely at the boundaries: encoding (modulation) and readout (detection and digitization).

The seduction is obvious: replace the O(b²·N²) transistor switching events of a digital N×N MVM with a single pass of light through a passive structure. The honest question — the one this paper is organized around — is what is actually being purchased, in what currency, and whether the purchase helps with training specifically, which is a substantially harder workload than the inference tasks on which nearly all published photonic demonstrations have been benchmarked.

Our thesis, stated up front: the prohibition "analog optics cannot do serious numerics" is not a prohibition; it is a specification. The specification has four requirements — sufficient effective precision, amortizable conversion cost, a weight-update path fast enough for stochastic gradient descent, and a scheme for dynamic (activation-by-activation) matrices. Each requirement is currently met in isolation by at least one experimental system. No system meets all four simultaneously. That conjunction is the open problem.

2.1 Coherent interferometric meshes

The canonical architecture, introduced for deep learning by Shen et al. (2017), encodes an input vector x ∈ ℝᴺ onto the complex amplitudes of N coherent optical modes and passes them through a mesh of Mach–Zehnder interferometers (MZIs). Each MZI, parameterized by two phase shifts (θ, φ), implements a programmable 2×2 unitary on its pair of modes. By the Reck (1994) and Clements (2016) decompositions, any N×N unitary U can be realized exactly by a mesh of N(N−1)/2 MZIs.

An arbitrary real matrix M is not unitary, but the singular value decomposition M = UΣV† factors it into two unitaries and a diagonal of singular values. The photonic implementation is therefore two MZI meshes sandwiching a column of amplitude modulators (with the matrix rescaled so all singular values are ≤ 1; the overall scale is restored digitally). The full MVM is performed in a single optical transit — roughly 100 ps for a centimeter-scale silicon photonic chip — regardless of N.

Two structural points deserve emphasis because they recur in the error analysis:

• —The computation is exact in the ideal limit. Unlike digital floating point, there is no rounding in the linear transformation itself; errors enter only through fabrication imperfection, phase noise, loss, and detection.
• —Errors compose multiplicatively along the mesh depth. An N×N Clements mesh has optical depth N, so component-level imperfections of magnitude ε produce output errors that grow with N — the central scaling obstacle for coherent meshes, mitigated but not eliminated by self-configuration and error-correction protocols (Bandyopadhyay, Hamerly & Englund 2021; Pai et al. 2023).

2.2 Incoherent crossbar arrays and WDM parallelism

The second scheme abandons phase and encodes information in optical intensity. Input elements are carried on distinct wavelengths (wavelength-division multiplexing, WDM); weights are set by tunable attenuations — microring resonator (MRR) weight banks (Tait et al. 2017) or crossbar arrays — and summation is performed by a photodetector, which naturally integrates total incident power. Multiplication is attenuation; addition is photocurrent summation. Because the scheme is incoherent, it is robust to phase noise, and because wavelengths are independent channels, a single spatial structure processes many vector elements — or many separate MVMs — in parallel. Demonstrations include the 11-TOPS convolutional accelerator of Xu et al. (2021) using a soliton microcomb as a many-wavelength source, and recent systems multiplexing across time, space, and wavelength simultaneously to reach hundreds of TOPS. Parallel processing across 100 wavelength channels has been demonstrated (Yu et al. 2025).

The cost of incoherence is that intensities are non-negative: signed weights require differential encoding (two detectors, subtracted photocurrents) or biasing schemes, doubling hardware or halving dynamic range.

2.3 In-memory photonics with phase-change materials

The third scheme makes the weights nonvolatile. Feldmann et al. (2021) demonstrated a photonic tensor core in which weights are stored in the optical transmission state of phase-change material (PCM) cells (GST) patterned on waveguides. The matrix is literally printed into the material's crystallinity; light passing through performs the MVM with no static power spent holding the weights — the photonic analogue of resistive in-memory computing. Magneto-optic in-memory cells with ultra-high write endurance have since been demonstrated (Pintus et al. 2025), addressing PCM's limited write-cycle budget.

This is the architecture most explicitly optimized for inference: weights that rarely change, held at zero static cost. Its relevance to training is inverted — the weight-write operation is the slowest and most energy-expensive step, which is precisely the step training exercises billions of times. Write endurance and write speed of the memory cell become the binding constraints (§6.2).

2.4 Free-space and diffractive processors

For completeness: free-space systems using spatial light modulators and diffractive surfaces achieve very large N (10⁵–10⁶ parallel modes) at the cost of bulk, alignment sensitivity, and slow reconfiguration. Recent single-shot matrix–matrix processors based on spatial–spectral hypermultiplexed diffraction, and coherent direct tensor processing schemes demonstrating full matrix–matrix products in one propagation (Nature Photonics, 2025), indicate that the matrix–matrix (not merely matrix–vector) primitive — the actual GEMM shape of transformer workloads — is now experimentally accessible in optics.

3.1 Where the joules go

Consider one photonic MVM of dimension N. The energy budget decomposes as:

• —Stage Cost scaling Typical magnitude
• —Laser (wall-plug, per symbol) O(N) set by SNR requirement (§4) DAC + modulator (encode x) O(N) ~0.1–10 pJ per element
• —Propagation (the multiplication) 0 (ideal); loss in practice —
• —Photodetection + TIA O(N) ~fJ–pJ per element ADC (digitize y) O(N) ~0.1–10 pJ per element, rising steeply with bits
• —Weight hold (MZI thermal / PCM) O(N²) static / 0 nonvolatile ~mW per thermal phase shifter

The MVM performs N² MACs, but the boundary conversions number only O(N). Therefore:

Energy per MAC ≈ (conversion energy per element) / N.

This is the correct, defensible form of the "free multiplication" claim: the multiplication is not free; it is amortized. At N = 1000, a 1 pJ/element conversion budget yields 1 fJ/MAC — one to two orders of magnitude below digital 8-bit MACs, before counting the digital system's dominant memory-movement costs. Larger N buys proportionally cheaper MACs, which is why photonic advantage is fundamentally an argument about scale: below N ≈ 100, conversion overhead eats the advantage entirely, and a GPU wins.

Two caveats sharpen the audit:

• —Weight-programming energy must be amortized over reuse. Thermal phase shifters consume ~10 mW each continuously; an N = 256 mesh spends ~300 W just holding its matrix. Nonvolatile weights (PCM, magneto-optic, MEMS) reduce this to zero but pay per-write. Either way, the accounting only closes if each programmed matrix is reused across many input vectors — trivially true for inference, structurally violated by attention (§6.3).
• —The ADC is the tax collector. ADC energy scales roughly exponentially with effective bits (a consequence of thermal-noise-limited comparator design). Demanding 12-bit readout can push the conversion cost per element into tens of pJ, destroying the amortization at practical N. Photonic systems are therefore driven toward low-precision readout — which collides with training's precision demands (§5). Integrated electro-optic DAC links (Nature Photonics, 2025) attack exactly this boundary cost.

3.2 The fundamental floor: photons per MAC

Is there a physical lower bound? Landauer's bound (kT ln 2 ≈ 3 zJ at 300 K) is irrelevant at current scales — we are six orders of magnitude away. The operative floor is shot noise: a coherent detection of a signal carrying n photons has amplitude SNR ~ √n. Achieving b effective bits at the output requires the accumulated photocurrent to carry ~2²ᵇ photons of signal. Because the dot product sums N terms, the required photon number per multiplication falls as 1/N for fixed output precision — the optical analogue of the amortization argument, now imposed by quantum statistics rather than engineering.

Wang et al. (2022) demonstrated an optical neural network operating below one photon per scalar multiplication (~0.001 photons/MAC in accumulation), with the sum still resolvable above shot noise. This is the cleanest experimental statement of the physics: individual products may be sub-shot-noise garbage; only the accumulated observable needs fidelity. Digital logic has no analogous mechanism — every partial product must be individually correct.

Ledger note (theorem-grade): for fixed output SNR, optical energy per MAC scales as O(1/N) down to a shot-noise floor of order 2²ᵇ/N photons per MAC. Ledger note (engineering fact): system wall-plug energy is currently dominated by lasers, DACs and ADCs, not by the optical field; demonstrated system-level efficiencies remain within one to two orders of magnitude of GPUs, not the four to six orders sometimes quoted from device-level projections.

Photonic MVM is analog computation, and inherits analog's classical pathology: precision is purchased in dB, and every dB is expensive.

Error sources stack as follows:

• —Static fabrication error in MZI splitting ratios and waveguide phases — correctable by calibration and self-configuration (progressive algorithms can configure meshes using only external measurements), and by hardware error-correction schemes that add redundant MZIs.
• —Thermal crosstalk and drift — phase shifters heat neighbors; matrices drift on ~ms timescales, requiring periodic recalibration.
• —Shot noise and detector noise — irreducible; sets the bits-per-readout budget.
• —Coherent mesh depth scaling — output error grows with optical depth N, biasing large coherent meshes toward block-partitioned architectures (tiles of N ≈ 64–256 composed digitally, which reintroduces conversion costs at tile boundaries).

The empirical state of the art is instructive: silicon microring weight banks have demonstrated better than 9-bit weight precision (Zhang et al. 2022) — a record, and achieved under laboratory conditions on small arrays. Typical operating points for full systems are 4–8 effective bits.

For inference, this is now understood to be sufficient: quantization research on transformers shows 8-bit and even 4-bit inference is achievable with negligible quality loss given proper calibration. Photonic inference at scale is therefore credible on precision grounds alone — consistent with commercial photonic processors (e.g., Lightmatter's 2025 Nature demonstrations of running modern AI workloads photonically) targeting inference first.

For training, the precision question is sharper and is the pivot of this paper, so we give it its own section.

Training differs from inference in three ways that each interact badly with analog photonics. We state them as constraints, then examine which are hard physics and which are soft engineering.

5.1 Gradient accumulation demands dynamic range

SGD updates are small: per-step weight changes are typically 10³–10⁵ times smaller than the weights themselves. Digital training handles this with mixed precision — low-precision (FP8/BF16) multiplications feeding high-precision (FP32) accumulators and master weights. The multiplications can be sloppy; the accumulation cannot.

This decomposition is actually favorable for photonics: the expensive O(N²) multiplication work can run at analog precision, provided accumulation and weight storage remain digital at high precision. The resulting architecture — analog photonic GEMM engine + digital optimizer state — concedes that photonics accelerates the FLOPs, not the whole optimizer. Since optimizer arithmetic is O(N) per step versus O(N·D_batch) for the GEMMs, this concession is cheap. Moreover, a body of results on training with noisy gradients shows SGD is remarkably tolerant of unbiased noise in gradient estimates — noise averages out over steps, acting like a smaller effective batch. The killer is not noise but bias: systematic analog errors (drift, crosstalk with fixed sign) do not average away and can steer training off course. The specification is therefore: analog errors must be zero-mean by construction or by dithering; bias, not variance, is the failure mode.

5.2 The weight-update bandwidth problem

A training step updates every weight. If weights live in thermal phase shifters (~µs–ms settling) or PCM cells (~100 ns–µs writes, finite endurance of ~10⁸–10¹² cycles), then reprogramming an N×N array every step at training step-rates (10–100 ms per step, but millions of steps) is either too slow, too energy-hungry, or burns out the memory. Quantitatively: 10⁶ training steps exceeds the endurance of most PCM cells outright.

Mitigations, in order of maturity:

• —Keep weights digital; stream them. Use the photonic engine as a stateless GEMM unit: weights are streamed in as one operand each cycle (like activations), stored nowhere in the optics. This sacrifices the in-memory advantage but matches how GPUs already work and sidesteps endurance entirely. Time-multiplexed and AWGR-based architectures are of this type.
• —Fast electro-optic phase shifters (lithium niobate, plasmonic, MEMS): ns-scale reconfiguration at fJ–pJ energies, replacing thermal tuning. This converts the weight-update problem from "impossible" to "an O(N²) conversion cost per step" — real, but amortized over the O(N²·D_batch) MACs each step performs.
• —Magneto-optic memory with >10¹² endurance (Pintus et al. 2025) — early, but directly aimed at the endurance wall.

5.3 Attention breaks the static-weight assumption

The transformer's defining operation — attention — multiplies activations by activations: QKᵀ and (softmax)·V. Both operands change every token, every layer, every step. Architectures whose weight-programming path is slow (meshes, PCM) simply cannot host attention; they are confined to the feed-forward and projection GEMMs (which are, to be fair, the majority of FLOPs at short-to-moderate context). Photonic attention requires architectures in which both operands enter optically at symbol rate — e.g., crossbars where one operand modulates wavelengths and the other modulates time slots, or the recent coherent direct matrix–matrix processors. This is an active frontier; surveys of photonic LLM acceleration identify dynamic-matrix mapping as a principal open challenge alongside memory for long context.

5.4 Backpropagation through analog hardware

Backprop requires multiplying by the transpose of each forward matrix and by the Jacobian of each nonlinearity. Options:

• —Digital twin / hybrid backprop: forward pass in optics, backward pass on a digital model of the hardware. Fails when the model and the physical device diverge (the "reality gap").
• —Physics-aware training (Wright et al. 2022, Nature): forward pass through the physical system, backward pass through a differentiable digital surrogate, with the surrogate's error absorbed because gradients only need to be directionally correct. Demonstrated on arbitrary physical systems, including optics.
• —In-situ adjoint backpropagation (Hughes et al. 2018; experimentally realized by Pai et al., Science 2023): exploit optical reciprocity — send the error signal backward through the same mesh, and the interference of forward and backward fields yields the gradient with respect to each phase shifter, measured locally. The hardware computes its own gradients. This is the most elegant known resolution, currently demonstrated at small scale.
• —Fully forward-mode training (Xue et al. 2024, Nature): gradient estimation using only forward physical propagations, avoiding the backward pass entirely; and forward-only on-chip training of photonic deep networks (Bandyopadhyay et al. 2024, Nature Photonics). These trade extra forward passes for architectural simplicity, and their scaling to 10⁹+ parameters is unproven.
• —The honest summary: every ingredient of photonic training exists as a published demonstration — sub-photon MACs, 9-bit weights, in-situ gradients, ns-scale weight updates, matrix–matrix single-shot processors. No published system composes even three of them at LLM-relevant scale. Composition, not existence, is the open problem.

6.1 Amdahl's law for a photonic GEMM engine

Suppose photonics makes GEMM energy negligible. Training energy on current systems splits roughly into arithmetic, memory movement (HBM reads/writes of weights, activations, optimizer state), and interconnect (gradient all-reduce). GEMMs are perhaps 40–70% of energy even when they are >95% of FLOPs, because memory traffic is so expensive per byte. A perfect photonic GEMM engine therefore yields at most a 2–3× system energy improvement unless the memory system is also rebuilt — for instance by keeping activations optical between layers, or by the in-memory weight schemes of §2.3. Surveys of photonic LLM hardware reach the same conclusion: memory, especially for long-context KV caches, is the co-equal bottleneck.

This is not an argument against photonic GEMM; it is the specification for what must accompany it. The corresponding digital-history precedent is exact: fast multipliers alone did not make GPUs dominant — high-bandwidth memory co-design did.

6.2 The nonlinearity gap

Between every pair of matrix multiplications sits a nonlinearity (GeLU, softmax, layernorm). All-optical nonlinearities at low photon numbers remain weak, noisy, or slow; nearly all systems detect, apply the nonlinearity electronically (or in the detector's own response), and re-modulate. Each optical–electronic–optical (OEO) hop is a conversion tax at every layer boundary. Softmax is particularly hostile: it is global (requires a reduction across the row), dynamic-range-heavy (exponentials), and precision-sensitive. A realistic near-term architecture keeps softmax/layernorm digital and pays the OEO tax, again bounding the achievable speedup rather than eliminating it. On-chip trainable diffractive systems with integrated nonlinear activation are appearing (Cheng et al. 2024), but at small scale.

6.3 A quantitative sketch

Take a deliberately concrete near-term design point and let it be wrong in public:

• —Photonic tile: N = 256 coherent block, 10 GHz symbol rate → 6.5 × 10¹⁴ MAC/s per tile.
• —Conversion budget: 1 pJ/element/boundary (aggressive but roadmapped with integrated EO-DAC links) → ~4 fJ/MAC amortized.
• —8 effective bits forward, stochastic rounding into FP32 digital accumulators; weights streamed, not stored optically.
• —1000 tiles per system → 6.5 × 10¹⁷ MAC/s ≈ 1.3 EFLOP/s (counting MAC = 2 FLOP) at ~5 kW of conversion power plus laser and digital overheads — call it 30–50 kW system.

A 10²⁵-FLOP training run on such a system: ~10⁷ s·EFLOP → order 100 days on one such system, at an arithmetic energy of order 10 MWh versus ~10³ MWh for the same arithmetic on current GPUs. The two-orders-of-magnitude arithmetic energy gap is real; the system gap after memory, interconnect, cooling, and the OEO tax plausibly compresses to 5–20×. That is not "free." It is, however, the difference between a gigawatt datacenter and a 50–200 MW one — which at frontier scale is the difference that decides where and whether training runs happen.

Every number in this sketch is a conjecture-grade engineering estimate, not a measurement. Its purpose is falsifiability: each line item names the benchmark that would confirm or destroy it.

The folk claim — "optics computes matrix products for free" — decomposes under audit into one theorem, one engineering fact, and one open problem:

• —Theorem-grade: linear optical propagation implements exact linear transformations with zero switching energy, and shot-noise statistics allow the energy per MAC to fall as O(1/N) at fixed output precision, demonstrated below one photon per multiplication.
• —Engineering fact: system energy is boundary-dominated (lasers, DAC, ADC, OEO hops); current full systems sit within 1–2 orders of magnitude of digital, with the gap set by conversion electronics that improve on ordinary CMOS scaling curves, not by any optical limit.
• —Open problem: the conjunction required for training — unbiased analog error statistics, ≥8-bit effective forward precision, weight-update paths compatible with 10⁶–10⁷ SGD steps, dynamic-operand capability for attention, and gradient computation that does not round-trip through a digital simulation — has never been demonstrated in a single system at even 10⁸ parameters.

Framed as a permission structure: physics does not forbid photonic training. It issues a specification with a price list. The multiplication itself is the one line item marked gratis; everything surrounding it is priced in photons, conversions, and calibration overheads, all of which amortize favorably with scale and none of which sits on a fundamental floor within three orders of magnitude of where engineering currently is. The pattern matches other cases where an apparent prohibition dissolves into a cost paid in a different currency: here the currency is not energy per operation but precision management and boundary bandwidth.

7.1 Falsifiable milestones (2026–2035)

The thesis of this paper should be considered confirmed in stages if the following occur, and weakened or falsified by their absence:

• —M1 (near-term): a photonic system trains a ≥10⁸-parameter transformer to within 1% of digital baseline perplexity, end-to-end, with all GEMMs optical. Falsifier: persistent bias-induced divergence at scale despite dithering.
• —M2: demonstrated attention (both operands dynamic) at ≥10 GHz symbol rate with ≥6 effective bits in a matrix–matrix photonic processor. Falsifier: dynamic-operand precision proves fundamentally worse than static-weight precision due to modulator nonlinearity.
• —M3: system-level (wall-plug, including lasers and conversion) energy per training MAC below 1 fJ at ≥8 output bits. Falsifier: ADC energy scaling fails to improve and pins the boundary cost.
• —M4 (decisive): a frontier-relevant training run (≥10²⁴ FLOP) completes on a photonic-majority system at ≥5× lower facility energy than the contemporaneous GPU baseline.

If M1–M3 land and M4 does not, the diagnosis will almost certainly be memory and interconnect, not multiplication — which would itself confirm the central accounting of this paper: multiplication was never the thing being paid for.

Photonic processors do not make matrix multiplication free. They make it amortized: the O(N²) arithmetic rides at zero marginal energy on O(N) boundary conversions, down to a shot-noise floor that experiments have already touched at sub-photon-per-MAC levels. For inference, this accounting already closes commercially. For LLM training, four additional requirements — unbiased low-precision statistics, fast unlimited-endurance weight paths, dynamic-operand attention, and physical gradient computation — are each individually demonstrated and jointly undemonstrated. The barrier is not a prohibition of physics but a conjunction of engineering specifications, every one of which is currently on a public roadmap. The cartoon — light doing math for nothing — closes. The real problem — composing five solved subproblems into one machine at 10¹¹ parameters — stays open.

Claim Layer Status

A1 Lossless linear optics implements exact unitaries; SVD extends to arbitrary matrices physics theorem (Reck 1994; Clements 2016) A2 Optical propagation dissipates no switching energy; costs concentrate at boundaries physics theorem (ideal case) / measured (real devices) A3 Energy per MAC amortizes as O(1/N) at fixed output SNR physics theorem-grade derivation, experimentally supported (Wang 2022) A4 Sub-1-photon-per-MAC operation is possible physics demonstrated (Wang et al. 2022) A5 ≥9-bit photonic weight precision achievable dynamics demonstrated at small scale (Zhang et al. 2022) A6 Coherent mesh error grows with optical depth N dynamics theorem-grade, drives tiling in practice A7 SGD tolerates unbiased analog noise; bias is the failure mode dynamics strong conjecture — solid theory + small-scale experiment; unverified at LLM scale A8 In-situ optical backpropagation via reciprocity dynamics demonstrated small-scale (Pai et al. 2023); scaling open A9 Attention (dynamic×dynamic) is photonically implementable at training rates dynamics open problem; single-shot MMM demos are existence proofs only A10 Weight-update endurance/bandwidth compatible with 10⁶⁺ SGD steps dynamics open; streamed-weight architectures dodge it, nonvolatile ones do not yet A11 System-level 5–20× training energy advantage by ~2035 dynamics conjecture — the §6.3 sketch; every line falsifiable A12 "Optics computes for free" myth retired — replace with A2 + A3 A13 "Photonics is 1000000× more efficient than GPUs" myth retired — device-level projection misread as system claim