NUL bytes in generated code: source files that compile, lint, and run — but are invisible to grep, VS Code search, and code review

Summary. A raw NUL byte (0x00, not the escape sequence) leaves a source file completely valid so long as it sits inside a string literal or a comment, which is where a generated one lands: TypeScript compiles the file, node runs it, ESLint passes it, and Prettier reformats around it while preserving the byte.

The same file is invisible to the tools that find and review code. Walking a directory, ripgrep skips it and exits 1 with no output at all; VS Code's search is built on ripgrep and behaves the same way. git diff reports Bin N -> M bytes with zero insertions and deletions, and GitHub's API serves no patch for it. Nothing warns you in either direction.

The consequence here: two modules that had to agree on a composite map key did not. One built it with a NUL separator, the other with a space. The lookup could never match, and because a missed Map lookup is a normal control path rather than an error, the dashboard rendered duplicate rows with their counts split. The file holding the correct implementation had been unsearchable — and its diffs unreviewable — since it was created.

The remedies, in short: add a CI check that fails when a tracked text file contains a control byte; give every source extension a diff line in.gitattributes (*.ts diff, *.astro diff, and so on) so git never silently stops diffing a source file; and when a search returns nothing you expected, re-run it with --text before concluding the code isn't there. Details, and the reason the byte was there at all, follow.

Versions, all measured rather than assumed: ripgrep 14.1.1, git 2.42.0 and BSD grep 2.6.0 on macOS, plus the ripgrep 15.0.0 that ships inside VS Code 1.134.0, run directly against the same files. All of it is documented default behaviour rather than a bug, and the rules turn on the bytes in a file rather than on the platform.

The symptom: one region, two rows

An analytics dashboard merged two sources — days already folded into pre-aggregated tables, plus the current day read live from the raw event table. The merge is a Map keyed by the fields that identify a row.

The geography table began showing the same country and region twice, with the day's views split between the two rows:

A duplicate here means the merge missed: either the stored side already had two rows, or the two sides disagreed about the key.

The strings were identical

The stored side was clean — GROUP BY country, region returns one Texas row. And the values really were the same bytes, checked rather than eyeballed:

SELECT country, region, hex(region) FROM wa_geo_daily WHERE region LIKE '%exas%';
-- US | Texas | 5465786173

SELECT country, region, hex(region) FROM wa_events   WHERE region LIKE '%exas%';
-- US | Texas | 5465786173

Same country code, same region name, same hex. Both sides derive the key from those two values, so the lookup should have matched.

The separator was a raw NUL byte

Forcing the live-side key into a failing assertion prints it verbatim:

AssertionError: expected [ 'US\u0000Texas' ] to deeply equal [ 'FORCE-FAIL-TO-PRINT' ]

US\u0000Texas, not US Texas. One module joined the parts with a NUL byte, the other with a space:

// rollup.ts — writing the aggregates
const key = `${country}<NUL>${region}`;

// query.ts — reading them back to merge
const key = `${r.country} ${r.region}`;

NUL is the better separator, and the module that used it was right to: region names and URL paths contain spaces, so a space separator is ambiguous — ["a b", "c"] and ["a", "b c"] produce the same key. The defect was that two files each invented the format independently, with nothing forcing agreement.

Critically, the mismatch cannot surface as an error. A Map lookup that misses is the ordinary path for "first time seeing this key", so the code created a second row and counted into it. Both modules were unit-tested and both were correct in isolation. The disagreement existed only in the seam between them.

The NUL in that file was not the six-character escape \u0000 that a developer would type. It was an actual 0x00 byte inside a template literal in a.ts file. Inside a string literal that byte is legal; between tokens the compiler would have rejected it outright. But its presence anywhere in the file is precisely what git, ripgrep, and grep each use as their definition of "binary".

Why the code could not be found

Searching for the key-building code the way code is normally searched:

$ rg -n "geoKey" src/
$ echo $?
1

Empty output and exit 1 — the same answer ripgrep gives for a string that genuinely is not in the tree. Any wrapper passing grep -I behaves identically. Plain BSD grep is marginally better: it prints Binary file … matches, naming the file while withholding every line.

ripgrep's rule is documented and deliberate: a file containing a NUL byte is binary, binary files are skipped, and -a/--text overrides it. Given an explicit path it does say so:

$ rg -n "aggKey" after.ts
after.ts: binary file matches (found "\0" byte around offset 11)

But walking a directory — how searching actually happens — it reports nothing at all.

VS Code's project-wide search is built on ripgrep, and the same result comes out of the ripgrep binary shipped inside VS Code itself: handed the file by name it answers binary file matches, and asked to search the folder containing it — the only mode the editor's search box offers — it returns nothing and exits 1. The request for a setting to search binary files (vscode#151456) is still open.

Why the code could not be reviewed

git applies the same heuristic to diffs. Changing exactly one character of a TypeScript file — a separator becoming a NUL — produces:

$ git diff --stat
 keys.ts | Bin 53 -> 53 bytes
 1 file changed, 0 insertions(+), 0 deletions(-)

Zero insertions, zero deletions, for a real change to source code. And git marks a diff binary when either side is binary, so this is not confined to the commit that introduced the byte: every subsequent change to that file is unreviewable until someone removes it.

That includes the commit which removes the byte: with the old side still binary, that diff is unreadable too, and only the commit after it returns to text.

In the repository where this happened, the rule meant every commit the file ever received:

Bin     0 ->  9826 bytes
Bin  9826 -> 12172 bytes
Bin 12172 -> 13230 bytes
Bin 13230 -> 13822 bytes
Bin 13822 -> 16916 bytes
Bin 16916 -> 17061 bytes
Bin 17061 -> 18062 bytes

Seven commits, no reviewable line. GitHub reports the same thing — for a pull request that certainly modified the file, its API returns:

query.ts        modified  +104/-25  patch: yes
rollup.test.ts  modified  +10/-0    patch: yes
rollup.ts       modified  +0/-0     patch: no

+0/-0, with no patch to serve: the file is listed as modified and carries nothing readable underneath — zero lines added, zero removed, no diff to review. A binary diff also does not look like a problem. It looks like an image.

Meanwhile the tools that execute the code had no complaint:

Valid, lint-clean, type-checked source that Prettier will happily format — and that no search could find and no review could read. Every one of these tools behaves exactly as documented. What no tool owns is the disagreement between them.

The two heuristics are not the same heuristic

git inspects the first 8,000 bytes for a NUL. ripgrep scans the whole file. So the position of the byte decides which half of the workflow breaks:

A byte near the top of a file costs code review. A byte near the bottom costs search while diffs continue to look completely normal. Neither state is announced, and no single tool observes both.

How the byte got there: JSON tool calls

This file was written by an AI coding assistant, and the byte came in through that interface. Tool calls are JSON — MCP is JSON-RPC, and function calling is JSON either way — so when a model writes a file, the contents travel as a JSON string. JSON has an escape for NUL: \u0000 is valid, and it decodes to the byte. Two payloads, one backslash apart, produce very different files:

{"content": "const SEP = \"\u0000\";"}     ← a raw NUL byte lands in the file
{"content": "const SEP = \"\\u0000\";"}    ← the six characters land in the file

Both are valid JSON. Neither errors. Nothing downstream objects, because the compiler accepts the result. To get an escape sequence into source, a model must escape its own escape — and nothing here is specific to one assistant or one harness, since every tool-calling interface moves file contents this way.

The mechanism is easy to repeat: the regression test written to prevent raw NULs in source was itself written with two raw NULs, inside the comment explaining why raw NULs do not belong in source. A byte scan caught them before the file was committed — the test itself could not have, since it walks tracked files and the new file was not yet one of them.

Why this is sharper for agents than for people

A person opening an unfamiliar codebase gets a file tree. They see the file whether or not search can read it, and opening it shows the contents.

An agent works mostly through search — in this workflow, grep and file reads are the whole interface to the repository. That changes what a skipped file means. For a person, an unsearchable file is a file with a quirk. For an agent, a file that search cannot see is a file that does not contain that code.

So when the next task was "merge the live events onto the stored totals", the available evidence said no key-building convention existed, and the reasonable action given that evidence was to invent one. That is the likeliest account of how a space ended up where a NUL belonged — it cannot be proven after the fact, but it fits what the tools would have returned. Either way the failure needs no hallucination and no carelessness to explain: every step is locally correct once a tool fails silently in the direction that resembles success.

The general form is worth stating carefully, because it is not limited to NUL bytes: anything that removes a file from search removes it from an agent's view of the codebase. An over-broad.gitignore entry will do it — ripgrep skips ignored files by default, so a checked-in-but-ignored file is invisible until someone passes --no-ignore. So will a generated file that never reaches the index, or a path the searcher declines to follow. A person routes around those gaps by accident, because they have a second channel. An agent working through search alone has no second channel, so a gap becomes a confident wrong answer rather than a moment of confusion.

Remedies, in order of how much they help

1. Prevent the byte (CI, deterministic). A test that walks tracked text files and fails when one contains a control byte, naming the file. A few dozen lines, runs in milliseconds, and worth confirming it is not vacuous by planting a byte in a tracked file and watching it fail. Note the scope: walking git ls-files means it catches the byte when the file is staged or in CI, not at the moment it is written. As a one-off audit, no test required:

$ git ls-files -z '*.ts' '*.tsx' '*.js' '*.mjs' '*.css' '*.md' '*.json' \
  | xargs -0 perl -0777 -ne 'print "$ARGV\n" if /\x00/'

2. Keep diffs readable (`.gitattributes`, one line). Declaring source extensions diffable stops git from silently giving up on a file:

*.ts diff

With that in place, the same NUL-bearing file that previously produced Bin 63 -> 67 bytes produces a normal diff, and git grep finds its contents again:

$ git diff --stat -- keys.ts
 keys.ts | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

This is the highest-value single line here, because it removes the review blindness permanently and requires no vigilance from anyone. It does not fix search — ripgrep and VS Code still skip the file — which is why the CI check comes first.

3. One habit for the search half. When a search returns nothing and something was expected, re-run it with --text before concluding the code is absent. This is a fallback, not a strategy: it depends on already being suspicious, which is exactly what the failure mode prevents.

And the design fix that actually mattered, separate from the byte: one exported function now owns the separator and both modules call it. Two files independently inventing the same key format was the defect. The byte only ensured nobody could see it.

How common is this?

Rare. A scan of one project's full dependency tree — roughly 60,000 text-extension files across some 900 packages — found no raw NUL bytes at all. In curated, published code this essentially does not occur.

Rarity is the mechanism rather than a defence. Nobody adds a check for a failure they have never seen, so when it does occur nothing bounds it, and this particular failure is silent in the one direction that resembles success.

Nor is it entirely new, though the older vectors are milder, and it is worth being precise about how. Windows PowerShell 5.1 writes UTF-16LE from Out-File and >, an encoding in which every other byte is NUL. Such a file is invisible to plain grep, and binary to git — but it is not invisible to ripgrep or to VS Code's search, because PowerShell writes a byte-order mark and ripgrep transcodes BOM-marked UTF-16 in either endianness. A.ps1 written that way is searchable in the editor and unreadable in the diff. Strip the BOM and search loses it too: the mark is what rescues it. Filesystem crashes are also reported to leave NUL-padded files behind, through ext4's delayed allocation.

A raw NUL in ordinary source is the strictly worse case: nothing marks it, and every searcher skips it. A workflow where code is found by searching is exactly where that costs the most.

The durable lesson is about the boundary rather than the byte. A toolchain has a layer that runs code and a layer that finds and reviews it, and those layers do not share a definition of "text". Usually that costs nothing. When it costs something, the finding layer fails by going quiet — and quiet is indistinguishable from correct.