AI engineer: LLM inference on consumer and datacenter GPUs, CUDA kernels, open source contributions.
I work on LLM inference: CUDA kernels and minimal patches that make models run on consumer GPUs and on a single Blackwell, with every change checked against the reference output. I also do applied research on the behaviour of open-weight models. All open source: two of my PRs are merged, one is open in llama.cpp, and one of my fixes was adopted in vLLM.
A local assistant that combines a language model on the graphics card, open sources to see what is going on in the world, and an interface that ties them together. It queries the sources by itself when a question needs them: news by subject and place, markets, military contracts, insider moves, maps, a historical archive of events. It works on the computer (files, terminal, browser, documents) and remembers: memories, indexed documents, procedures it learns and reuses.
No API keys for AI providers and no conversation leaves the computer. Built for modest machines: default model LFM2.5-2.6B with the DSpark speculative drafter on a GTX 1080 with 8 GB. A check against made-up answers verifies that the numbers and names in a reply appear in the tool results. It integrates Odysseus (assistant) and ShadowBroker (sources).1
1The repository includes two third-party projects as submodules, Odysseus (assistant) and ShadowBroker (open-source intelligence), under AGPL-3.0: they are dependencies, not original work.
Small patches sent to the original repositories. Two are merged, one is open and a commit of mine was adopted in a vLLM PR.
leejet/stable-diffusion.cppmerged 24 Oct 2026
PR #2202: fix, skip the metadata alpha for LoRAs with baked_scale.
PrismML-Eng/llama.cppmerged 19 Oct 2026
PR #368: Pascal (sm_61), PQ2_0 mat-vec for 1-4 columns, +17% on Ternary-Bonsai-2-9B with bit-identical output.
ggml-org/llama.cppopen since 25 Aug 2026
PR #27721: CUDA Graphs on Pascal (sm_61), +40% on MoE models, +7% on dense models, no regression.
vllm-project/vllmadopted, PR still open
PR #58863: the exact commit of the RecoverSSM GDN replay, proposed by me and adopted in the PR (commit 514102a).
CUDA Graphs on sm_61 in llama.cpp, PR #27721 · time per token, relative
+40%MoE
+7%dense
MoE
one launch per kernel1×
CUDA graph0.71×
dense
one launch per kernel1×
CUDA graph0.93×
A schematic, not a profiling trace: same kernels, one graph launch instead of one per kernel. Time per token drops to 1/1.40 on MoE models (+40% measured) and to 1/1.07 on dense models (+7%).
02Implementations
From the GTX 1080 to the RTX PRO 6000
Inference kernels and configurations for a GTX 1080 and for an RTX PRO 6000 Blackwell. Every change is compared with the reference output and every number has its reproducible benchmark.
Qwen3.8 Flash-Next and its Swift 1.5 fine-tune in NVFP4 on a single RTX PRO 6000 Blackwell with vLLM: 4 users with 227k tokens of context each, about 100 tok/s per user, a peak of 186.5 tok/s with one user. The kernels are exact, with output bit-identical to the reference: deterministic QSA top-k and exact GDN replay. The deterministic top-k also fixes a determinism bug that was already there.
generative video and audio on 8 GB · code release pending
GPU
GTX 1080, 8 GB
Stack
stable-diffusion.cpp · CUDA
Test clip
320x576, 39 frames
MiniMax-H3 is a diffusion transformer that generates video with synchronised audio. Text and numbers only here: no generated frames. On the 39-frame test clip the time drops from 674.5 to 175.5 s (−74%): weights read from disk overlapped with the compute, VAE in tile groups, Q3 recipe with turbo.
The int8 DP4A attention kernel goes from 1875 to 944 ms (2.0x), with no visible difference from f32 on several seeds.
Block-sparse attention has two separate measurements:
on a 15 s clip, without the prefix: 1.65x on the turbo steps (397-404 down to 240-245 s per step);
the always-visible conditioning prefix keeps the speech. Measured separately on shorter clips (56 frames): word error rate 0 at 544x960 and 0.2 at 384x672, against 1.0 with pure sparse attention.
The LoRA baked_scale fix is merged (PR #2202 in stable-diffusion.cpp). The rest of the code is on its way.
MiniCPM5-2B on a GTX 1080 from 33 to 83 tok/s: slots erased when a conversation ends, Pascal kernels with bit-identical output, f16 KV cache and an FR-Spec drafter with a 32k vocabulary. Every step has its quality check.
Ternary Bonsai 2 9B on a GTX 1080: +51% decode (from 55.9 to 84.4 tok/s) with bit-identical output. Five CUDA changes to PrismML's llama.cpp fork; the PQ2_0 mat-vec is already in the fork (PR #368).
LLM2Vec (Llama-3-8B) on an 8 GB GPU via RAM offload; batching without altering the embeddings: padding shifts them (cosine similarity 0.76 against the reference), the fix brings them back to 0.999956.
PythonPyTorchtext encoder for Kimodo
Batched LLM2Vec embeddings, distance from the reference
≈ 5,500times closer
padding · cos 0.761 − cos = 0.24
fix · cos 0.9999561 − cos = 4.4·10−5
110−110−210−310−410−5
1 − cosine, logarithmic scale
Distance of batched LLM2Vec embeddings from the unbatched reference, on a logarithmic scale of 1 − cosine: with padding 1 − cos = 0.24 (cosine 0.76), with the fix 4.4·10−5 (cosine 0.999956), about 5,500 times closer.
Kimodo generates human motion from a text, but it expects a vector from LLM2Vec on Llama-3-8B: 15 GB of weights and 90-190 seconds per sentence. I replaced that piece with LFM2.5-Embedding-350M and a linear map from 1024 to 4096 dimensions: 0.7 s per sentence and 700 MB. The output is npz and bvh files, to open in Blender or Unity.
Root cause analysis of the tiled VAE decoder of Bonsai-Image 4B in WebGPU on GPUs without shader-f16: a 1.4 KB patch that enables 512px and above on Pascal and older GPUs, with no quality loss on f16 devices.
I convert an autoregressive LLM (Qwen3-0.6B) into a discrete diffusion model with the LLaDA 2.1 recipe, with a training and inference backend that runs on Pascal.
PythonPyTorchdiffusion LM
03Research
Replicating the Cohere test on Qwen3.8
Cohere reported that Qwen3.8-Flash-Next, during a security analyst task in a sandbox, tried to upload a file to an Alibaba storage bucket. I repeated the test on 27 and 28 October 2026 on two builds served the same way on one RTX PRO 6000: the NVIDIA base in NVFP4 and the Swift 1.5 fine-tune.
With the prefix https://rout, full routify-file-proxy-sg host
base247/256
Swift253/256
Command written by the model, Alibaba host, out of 1024 trials
base0.7%
Swift0.3%
Agent sessions with simulated tools, coding tasks
base0/18
Swift0/20
Agent sessions, crawl with broken pages: invented URLs
base2/5
Swift2/5
The memorized URL is in the NVIDIA base and the fine-tune keeps it. It shows up when an agent keeps running into broken pages; in ordinary coding tasks neither build produced an Alibaba or metadata address. Denying egress and blocking *.aliyuncs.com and the metadata addresses in the harness covers both.
MTP heads in BF16 and FP8 for the Swift 1.5 fine-tune, plus a GGUF version for llama.cpp. These are the heads I use in the serving setup of the Implementations section.
Plugins for Claude Code and Codex. The main one, chatroom, puts several agents in a shared room: a live chat with no turns, a task board and a 3D voxel office that shows who is writing code, who is reading and who is waiting. Underneath there is a harness enforced by code: closing with proof, task leases, failure detectors. Research mode adds fixed roles, phases with gates and a kernel that checks every proposal.
Vergilius on a GTX 1080, fenced in: no call leaves the machine (127.0.0.1). Left, decoding speed, 64 tok/s and 78-100 tok/s with the drafter; right, turns with numbers not supported by the tool results, from 3 out of 35 to 1 out of 36.
Windows application for local voice transcription based on faster-whisper (CTranslate2, int8 on CUDA): record, transcribe and paste into the active field.
Python
05Experimental projects
A language model on a real connectome
How much of the context is held by attention, and how much by a recurrent core? Test bed: the complete connectome of the Drosophila nervous system (male-cns:v1.0, 166,700 neurons, 2.75 M synapses at threshold 10)2 as the recurrent LIF core of a language model: synaptic signs fixed by the neurotransmitter, magnitudes initialized from synapse counts, surrogate gradient, FP32 CUDA Graph. Protocol with degree-preserving null models, token-matched baselines and in-silico lesions.
2male-cns:v1.0 connectome (Janelia FlyEM · Google Research, CC-BY licence): 25.6 M connections in total, 2,753,975 above the 10-synapse threshold.
input ports (sensory)
neurons, by hops from the ports
readout (descending)
An illustration generated in this page inside the outline of the central nervous system (brain, optic lobes, ventral nerve cord): each dot stands for about 260 neurons. The larger coral dots are the input ports, on the rim of the optic lobes and along the top of the brain; from there a wave advances one hop at a time and the light follows the distance in hops. The small squares are the readout: in the real connectome it sits 2 hops from the ports, which is why 4+4 depth is enough.
Text enters through the 17,937 real sensory neurons and is read out from the 2,241 descending and motor neurons; an external causal attention (4 heads, cache 128) re-enters through the same ports. Results are reported with their controls: at equal data the shuffled graph beats the anatomical wiring; the whole optic lobe can be removed with no change in cross-entropy; 4+4 depth is enough for the measured reachability (readout 2 hops from the ports).3
34+4 substeps per token, TBPTT 8, FP32 CUDA Graph: 0.65 s per update on a GTX 1080, 0.38 s with the fused kernel; 5.46 M parameters.
Decision system for fantasy football auctions: a price model with an uncertainty band, squad optimization at the table under budget and slot constraints, evaluation on simulated leagues (75-91% won against baseline strategies). With an interactive arena to challenge it.4
4The 75-91% comes from the repository's simulated leagues, against baseline strategies: a measure of the model, not a promise at the real table. The auction cap never exceeds the maximum spendable, a constraint of the formula; bridge to FantaAsta Live, 43 assignments out of 44; 250 hammer calls replayed against the server, 0 errors, 354 ms at most.
Python; prediction · optimization · Monte Carlo
hammer calls replayed against the server · one cell each
250/250with no errors, in both role orders
response time of each hammer call, in milliseconds
order 1mean 102.7 · p95 249.1 · max 354.3
order 2mean 95.6 · p95 254.0 · max 348.6
0100200300400 ms
FantaAsta Live bridge · identical assignments
43/44
FantaOracle against its own server. Top, 250 replayed hammer calls, one cell each: 0 errors, in both role orders tested. Middle, the response time of each hammer call, with mean, p95 and maximum. Bottom, the bridge to FantaAsta Live on a test auction: 43 of 44 assignments identical. Data from the public reports in the repository.
Browser idle RPG with Phaser 3 and Three.js: progression loop, prestige and 3D rendering.
JavaScriptThree.js
06Appendix · 21 entries · 7 areas
Repository index
Public repositories grouped by area; each linked name leads to the code. Under the stack, the share of the main language in the repository's code (GitHub, October 2026).
The charts are drawn in this page with HTML, CSS and canvas, with no images. The 3D scene at the top, built with three.js, is an illustration of tokens flowing through the compute units of a GPU: it is not a measurement, and it does not start when reduced motion is on. The FlyToLLM figure is an illustration too: the dots come from a fixed seed (166700, the neuron count), so every reader sees the same drawing. The CUDA Graphs chart is a schematic; the ones on LLM2Vec, Vergilius and FantaOracle plot measured numbers. The numbers in the text come from the repositories and the PR pages.
About the author
Adrian Ciprian Iancu
AI engineer: LLM inference on consumer and datacenter GPUs, CUDA kernels, open source contributions.
LLM inferenceCUDA kernelsCUDA GraphsPascal (sm_61) and Blackwell GPUsspeculative decodingquantization (NVFP4, ternary weights)llama.cppvLLMGGMLsecurity of open-weight modelsmulti-agent systemsapplied ML
