Solveig

How work is decided and checked here

The practices this project actually runs on, which until now lived in nobody’s document. ROADMAP.md says what is outstanding, ideas.md says what was considered and refused, releasing.md says how a release is cut. This says how any of those get decided, and how a claim gets to be believed.

Every rule below was learnt by getting something wrong, and each one names the occasion. That is the only reason to trust them.

A program asks, not a document

Nothing is built until something asks for it, and the asking is done by a program rather than by a page. ideas.md records each deferred idea with a named trigger, and there are two dozen of them; the trigger is what fires the work.

The rule earns its keep in both directions.

It fired for startsWith and endsWith on 2026-08-30. They had been deferred the day before with one customer. A count then found three programs, nine call sites, and two independent implementations of endsWith — one of which carried a comment recording a defect the absence had already caused. Two copies of one function is the same trigger string:replace was built on.

And it did not fire for dictionary:of, built the same day because a dictionary literal would compile to it and because it was asked for. Every dictionary:new in the tree is an accumulator or a table of blocks, so nothing called it but its own demonstration. The entry says so rather than dressing it up, because a page that only records the times the rule was obeyed is not evidence of anything.

Why it holds: the one time the method was ignored, the page arguing for extensions kept getting longer instead of being tested. When it was finally acted on, an afternoon falsified two of its claims.

The throwaway comes before the design

Build the smallest thing that settles the question, and let it correct the design. Fifty lines, thrown away afterwards.

The extensions entry closed by advising exactly this and was then proved right by its own advice. The GTK canvas found that a cairo_t cannot be published to a program because it dangles the moment a draw callback returns — a design question no amount of reading would have raised, and one that decided the interface.

Scope before building, and the decision is separate

A language change is written into ideas.md first, with the analysis, a recommendation, and the calls only the author can make. Then it stops. Building is a separate instruction.

@expr was scoped on paper on 2026-08-28 and implemented the next day with almost nothing left to decide. The dictionary literal went the same way on 2026-08-30 — and the scoping is what found that half the proposal was unnecessary, which no implementation would have discovered because that half would simply have been built.

Scoping is also allowed to end in no. Named arguments were recommended in the morning and refused in the afternoon, once the options array turned out to catch every mistake it stood accused of passing.

Check the thing, not a picture of it

A graphics program’s output is a picture, so the check has to be the picture. On 2026-08-30 an SDL example drew bands of noise past three checks that had all passed: the arithmetic rendered as ASCII, the program run and timed, and the colours recorded as they were handed to sdl:fill. All three tested the inputs to the drawing calls and the defect was on the other side of them. Look at it — screencapture, or have the program write a PPM.

It is not only about pixels. The 0.40.0 release page rendered as a narrow ragged column because release notes are rendered with hard line breaks and every document here is wrapped at 79 columns. The markdown was correct; every check on it passed and every check was honest. There was nothing to find by reading, because nothing that was read was wrong.

A check that cannot fail is decoration

Two of these in one day, both the same shape, and the shape is worth more than the instances: a comparison whose two sides came from the same source is not a comparison.

The editor’s screen transcript was compared against a recording made at a path that no longer existed, so the comparison ran one binary twice. html.sol before and after a change was compared by recompiling a probe — except @include resolves while compiling, so the probe’s .sob had the old library baked into it and the same bytecode ran twice.

The question that catches both: what would have to be broken for this to fail? If the answer is nothing, the check is decoration.

And its mirror, from the same day: a test that fails for the wrong reason misleads exactly as far as one that cannot fail. Three findings against the 0.40.0 release page were artefacts of the checks rather than faults in the page, and reporting any of them would have been worse than not checking.

The hardest version of that is the one that misleads usefully. On 2026-09-01 a throwaway link checker reported one dead anchor in CHANGELOG.md. Chasing it found two real faults in that file — a paragraph wrapped so that ``` began a line, and an inline code span wrapped so that <if-statement> began one — which between them had left 64 of that page’s 327 headings on the published site for ten days. Then the reported anchor turned out to be an artefact of the throwaway’s own fence rule: under the rule the renderer keeps, that link was fine, and the checker that shipped reports nothing on a tree with both faults still in it.

A wrong check produced a wrong finding, and chasing the wrong finding found two right ones. That is luck, and the write-up has to say so — the temptation is to report the check as having worked, because something real came out of the run. Ask what the check would have said if the faults were the only thing wrong, and put the answer in the entry. Here it was nothing, which is how the artefact was found at all.

And a comparison whose two sides did not run alike is not one either

The other half of the rule above, and it cost an afternoon on 2026-09-01. A throwaway measured /usr/bin/tail -f and -F through a rotation, in one script, one after the other, and reported that both follow the name. That went into a roadmap entry in bold, including the man page is wrong about its own flag.

It is not. -f follows the descriptor, and lsof on the running process shows it holding the renamed file open. The measurement reproduced four times, at two timings, which is exactly what made it convincing; reducing the script to one flag made the wrong answer disappear.

What caught it was follow.sh — a harness that runs the two sides under one set of conditions — on the first run after a scenario for this went in. The rule had been applied to every check here and not to the throwaway that measured the oracle those checks compare against.

So a throwaway that measures something the documents will state is not a throwaway. It is a check, and it owes the same discipline: one harness, both sides, and direct evidence when a run disagrees with a published description. Distrusting the run rather than the page was the right instinct and was applied in the wrong direction — the run was re-run, agreed with itself, and was believed. Re-running a wrong experiment is not evidence.

Check against what ships, not against the working tree

On 2026-08-29 a throwaway reported sixteen .sob files in the tree differing from a fresh compile, and one of them shipping without an exports boundary its source demands. Written up as a defect. It was not one: *.sob is ignored, make install copies lib/*.sol, and nothing in lib/ is tracked or built. They were hand-compiled leftovers.

git ls-files takes four seconds, and it was skipped precisely because the finding looked interesting. Presenting a non-defect as a defect costs more than saying nothing.

And every check here enumerates by extension

Its sibling, found on 2026-09-02 while cutting 0.41.0. programs/:= had been in the tree since 2026-08-31 — 117 kilobytes, byte-identical to pascal.sol, committed by accident, and about to ship inside the tarball, since make dist archives HEAD.

It survived two days, fifty-four commits and a day spent auditing the documentation, because not one check here looks at the set of files. They look at files of a kind: expect.sol counts solFilesIn:value("programs") and reads .md, the link checker walks markdown, site.sh fetches published pages, and make test compiles what the Makefile names. A file with no extension is outside all of them at once — so nineteen programs stayed true, recounted on every build, while the directory held twenty files.

The remedy is one command and it is not git ls-files. That answers what is tracked, which this file was. git diff --name-status <last tag>..HEAD is the only view here that asks what a stretch of work added, as against what the tree says about itself, and it found this in four seconds while the release notes were being written rather than while anybody was looking. releasing.md runs it before the four files now.

This is the enumeration rule one level up. There it was the four states of a pipe against the three anybody thought of; here it is the kinds of file a repository can contain, and the answer came the same way — by asking the thing itself rather than listing what came to mind.

Hold it against something somebody else wrote

The strongest check available is an implementation whose author had no idea what this language can do. Every other check here is a transcript recorded by the person who wrote the code, and can only catch what that person thought to check.

It cost ninety seconds to prove on 2026-08-31. programs/sed.sol was held against /usr/bin/sed and the first run reported a defect in lib/pattern.sol that had been shipping for days: an empty match at the position where the previous match ended was being counted as a match, so s/o*/-/g over aoc answered -a--c- where every sed answers -a-c-. edit.sol’s :s had had it for as long as it existed.

The part that transfers is why nothing had caught it. The library’s header explains the neighbouring rule at length and demonstrates it with s/x*/-/g over abc — and that example is the one case that cannot show the difference, since the star never matches a character there, so no match has an end for a later empty one to land on. The documentation was careful, correct, and blind by construction. An example written by the author of the code shows the rule the author was thinking about; only a stranger’s program picks a case they were not.

It is not a text-tool trick and does not run out. sha256sum has published vectors, diff has diff, a matrix multiply has numpy, a Prolog has swipl. What the frontier loses is exactly the oracle, which is the argument for spending the cheap ones early — ideas.md surveys them.

And better than an implementation is a number somebody printed. The sentence above was written on 2026-08-31 and tested the same day. programs/sha256sum.sol is held against /sbin/sha256sum and against the digests in FIPS 180-4, and the second is the stronger of the two for a reason the first cannot fix: an oracle can be wrong in the same direction as anything derived from it. Two implementations of a hash that share an ancestor share its mistakes; a number printed in a standard before this language existed cannot have been influenced by anything here. programs/sha256sum/vectors.sh is the second check here that does not depend on another implementation being right, and where such a thing is available it should be the first one written.

The first was the NBS Minimal BASIC Test Programs, 208 programs written at the National Bureau of Standards in 1980 against ANSI X3.60-1978, which found seven defects in basic.sol that eighty-three claims written by its author had not. That is the same idea and it was here first; what vectors.sh adds is that a digest is a string, so the comparison is mechanical, where the NBS suite prints what a correct result looks like for a person to read and cannot be scored by a machine. A standard that prints answers is worth more than one that prints descriptions, and it is worth knowing which kind a direction offers before betting a check on it.

It also decides what the oracle is for. Held against the vectors, the algorithm is either right or wrong. What the oracle then checks is the plumbing — a NUL lost, a high byte sign-extended, a chunk boundary landing inside a block, a warning that says “1 line is” where the other says “2 lines are” — and those are the failures a standard says nothing about. Two checks that fail for different reasons are worth more than two that fail for the same one.

Two corpora, and the second is the point. agree/ must match byte for byte. differ/ must not, and each case says at the top what each side does and why this one is allowed to be different — so the list of divergences stops being prose and becomes something that fails. A case in differ/ that starts agreeing is news too.

Run every case both ways in. A named file and a pipe are different code paths here, and a program that answered two ways about the same bytes would be wrong where a single-route check cannot look. Where the two genuinely cannot agree, the case declares it and bounds it: pipediffers: means the pipe’s answer must be the file’s plus exactly one newline, which would fail if the difference grew.

programs/oracle.sh <name> is the harness; it was written for sed and generalised by its second caller rather than copied.

An author-written corpus tests what its author thought of

The oracle is the only check here that can find what nobody thought to look for — and the corpus it runs is written by the same person who wrote the program, so it inherits their blind spots exactly.

programs/sed/agree/ holds sixty cases and not one of them used \(. So sed.sol read a group as a literal parenthesis for the life of the file, and on a valid script came out inverted — substituting the line that contained the text (ab)c and leaving the line that contained abc alone, with no error and exit 0. Its own header had said such a script “will be refused rather than misread”. Nothing had run it.

That is not an argument against the corpus, which has earned its place several times over. It is the reason the corpus is not the last word:

A generator is a second author, and it has blind spots too

The remedy for an author-written corpus is a generator, and on 2026-09-02 the generator was blind in the same direction as the corpus.

diff.sol’s corpus was twenty-four hand-written cases and all of them passed a wrong rule for where an empty range is written in a unified header. Seven were the only ones that could have shown it – the rule lives in the unified format and the other seventeen never print a hunk header – and all seven put their empty range where the simple rule and the real one give the same answer. The size of the corpus was never the point; the shape of what its author reached for was. A random sweep caught that – 44 disagreements in 1,050 runs – and then reported 2,400 runs, zero after the fix.

The first pair of real files disagreed anyway, an hour later: docs/method.md at two revisions. Where a line inside an inserted block equals the line at the seam, an insertion can be placed as one run or split around that line for the same number of edits. One real pair in five does it. The sweep could not: it mutates one line at a time, and the shape needs a block inserted whole – which is what editing prose does every time and what nobody constructs on purpose.

So the ladder has three rungs, not two. An author’s cases test what the author thought of. A generator tests what its generator can produce, which is a different blindness and not a smaller one. Data somebody made for their own reasons tests neither – and this repository’s own git history is sixty pairs of it, free.

And when two answers are both right, byte equality is the wrong check. Neither answer above is a defect, so the oracle cannot settle it and the disagreement is not news. What settles it is the property: apply.sh hands our unified diff to patch(1) and compares the result with the second file – is this the diff from A to B, where the oracle asks is this the tool’s diff. A program with an oracle can still want a check that is not the oracle, and the moment two right answers exist it needs one.

A check too slow to finish is a defect report nobody reads as one

On 2026-09-02 a sweep ran for two hours and fourteen minutes without finishing its first half, and that was the finding. It was reported as still running three times before anybody asked why, because a check that has not answered looks exactly like a check that is being thorough.

What it had found was a defect in the program under test: sort’s k-way merge picked its winner by a linear scan over every run’s head, so the cost was lines x runs, and the sweep’s -S 16 over a 14,707-line file made some forty-nine thousand runs. A heap took the same case from did not finish to 3.88 seconds and the whole sweep to five and a half minutes.

The check was working. Nothing was reading its output, because its output was silence and silence is what a slow check and a passing check both look like. A program that does not stop can still be checked says to give an unbounded program a deadline; this is the same rule pointed at the check, and it is the harder half to remember because the check is the thing you trust.

So: know roughly what a check should cost before running it, and treat a large overrun as a result rather than as weather. The estimate does not have to be good – ten minutes was guessed here and two hours was the answer, and the factor of thirteen is what should have been read as news on the first check rather than the third.

A program that does not stop can still be checked — give it a deadline

tail -f was nearly left out on the grounds that an oracle cannot check a program that never finishes. That was true and was not a reason. Start both, feed the input on a schedule, stop them, compare what each managed to write: fifteen lines of shell, in programs/tail/follow.sh.

It earned itself on its fourth scenario, finding that BSD tail puts a blank line before the first heading when following and not when it is not — which nothing but a check running the real thing would have found.

The general form: when a check looks impossible, ask whether it is the shape of the check that is wrong rather than the thing being unchecked. A deadline, a scripted key sequence (edit.sol’s 181 sessions), a pseudo-terminal — each turned something interactive or unbounded into something with an answer.

A sentence that was true when written is not checked by anything

Four instances in one day, 2026-08-31, and the shape is worth more than any of them. Each was a statement that was true when it was written, stayed technically true, and became misleading because the world moved underneath it.

The habit that catches them is not a tool. It is going to check a sentence before repeating it, and preferring the check that could fail: making the file, running the case, moving the number. The 2026-08-30 postmortem said the same thing about five documented claims of which four were wrong, and this is the second day running.

An enumeration that looks complete is not a proof

Three cases, all correct, reads exactly like all the cases, and the second is a far stronger claim than the first. Nothing in the sentence marks where its edges are, so a reader — including the person who wrote it — cannot tell a survey from an argument.

On 2026-08-31 tail.sol and sha256sum.sol both told a person at a prompt from a pipe with keyWaiting(0.0), and the reasoning was written down in three places and called exact rather than approximate: an idle terminal answers false, a pipe with data answers true, a pipe at its end answers true. Each of those is true.

A pipe has four states. The fourth is open, empty and not yet finished, and it answers false — exactly as the idle terminal does, because is there a byte right now is equally false of both. So { sleep 1; echo hi; } | prog took the terminal branch, and both programs threw away the input of any pipeline slow to produce its first byte. For as long as either had existed.

Nothing here was going to catch it. A pipeline typed at a prompt or written into a corpus has its first byte ready before the program starts, so the missing case does not occur anywhere it would be looked for. It needs a slow writer, which is not a thing anybody constructs by accident.

The check is not more care with the prose. It is to go and ask what states the thing has from its own side — a pipe, not the list of pipes somebody thought of — and count them. Where that is not possible, say these are the cases I found rather than these are the cases, so the sentence carries its own uncertainty.

Replacing something that works is how you find out what it was doing

The remedy that actually fired here is worth naming separately, because it is cheap and nobody plans it.

The keyWaiting paragraph had been read many times and never audited — there was no reason to audit it, since the program worked. The audit happened only because 6.40 built a message answering the same question, and the two answers had to be compared. The comparison is what asked, for the first time, what the old spelling had actually been answering.

So a replacement that is only about spelling is still worth doing, and the argument for it is not tidiness: an old expression nobody has a reason to doubt is exactly an expression nobody checks.

A predicted limitation decides which implementation gets written

Three times an entry here has predicted that 3.5, the recursion limit, would bite a program about to be written. Three times it did not, and the fourth was written down before it could happen:

the program what the entry expected what was written
basic.sol deep nesting an argument that a line-numbered language never nests
check_syntax.sol a recursive tree walker LPeg’s instruction set, after the walker measured 19 levels
pascal.sol a recursive descent that recurses a compiler rather than a walker
diff.sol Myers’ divide-and-conquer recursing on halves of the edit graph the greedy forward pass, which is two loops

The prediction is not wrong about the limitation. It is wrong about the author. A limitation that is written down is known before the implementation is chosen, so the implementation that meets it is the one nobody writes. What the entry is really predicting is which of two algorithms gets used, and the answer is always the other one.

That makes such a prediction nearly unfalsifiable, and unfalsifiable is the one thing a prediction here may not be – the whole point of writing them down before the program is that it found nothing stays an available answer. So: when an entry predicts a documented limitation, it owes the shape of the program that would hit it, and whether anybody would write that shape for reasons other than the prediction. diff’s entry did name the variant – the divide-and-conquer form – and the naming is what made it possible to say afterwards that the variant was never the one to write.

The useful predictions in the same entries were the other kind: the output format is the hard part named a difficulty rather than a limitation, and it was right, and it could have been wrong.

An analogy to a measured case carries the mechanism, not the rate

A prediction on 2026-08-31 was right about an absence and wrong about its price, and the way it went wrong is the useful part.

tail -f needs to wait. The prediction was that shell:run("sleep 1") would do it and the finding would be the cost — reasoning from 6.34, where the terminal’s size was reachable through stty at 7 ms an ask and the price was what made it an entry.

A fork of /bin/sleep measured 2.23 ms, which at a one-second poll is 0.22%. Perfectly livable. stty was a fork per keystroke and this is a fork per second, and the entry reasoned from one to the other because both are a fork where a syscall would do — without noticing they differ by four orders of magnitude in how often they happen.

A cost is a property of an operation and a rate, and the prediction carried only the first half. When an entry argues by analogy to a measured case, the thing to check is whether the rate carried over, not whether the mechanism did.

system:sleep was built anyway, on a weaker and truer argument: waiting is one call to the kernel and a program should not start a process to do it. Being right for the reason expected would have been worth less than finding out the reason was wrong, and the entry keeps both halves.

A scoping can be wrong about the order, not only the answer

Scope before building says a scoping may end in no. On 2026-08-31 one ended in not yet, and not in that order.

tail was scoped to be written first, on the whole-file read, so that it could ask for a ranged one — because a program asks and a page does not, which is the rule. The evidence had already arrived without it: fileSize answered and readFile refused on a file made in four seconds.

What was wrong is sharper than being unnecessary. A tail on the whole-file read cannot call the thing it is meant to be asking about, so it would have re-proved a measured wall and said nothing about the shape of the fix. The program meant to inform the design was the one program guaranteed not to.

The question splits: whether is often settled by a measurement, and what shape wants a caller — and a caller has to come after the call exists. When the evidence is already in hand, build the thing and write the program against it, which is the throwaway rule with the order put right. Both recommendations are kept in the entry rather than the first being overwritten, which is what that page does with predictions.

How a feature ships

One unit, in this order:

  1. Implementation, then tests — tests/test_<feature>.c, including a GC-stress case if it allocates.
  2. Prove a new GC root is load-bearing by removing it and showing what breaks, then restore. This can end in deletion. dictionary:of’s root was removed on 2026-08-30 and 200 dictionaries plus a 120-pair one ran clean under SOLUM_GC_STRESS, because sol_dict_put grows with calloc and cannot collect — which object.c already said, beside the code doing it. A guard against a hazard that is not there is worse than no guard: it tells the next reader the hazard exists. Keep the root only when removing it breaks something, and leave a comment saying why there is none. A front-end-only change owes no GC proof; say so.
  3. The documentation tail — docs/REFERENCE.md, docs/CHEATSHEET.md, docs/GRAMMAR.md and programs/check_syntax/solum.bnf if the grammar moved, an example, and docs/ROADMAP.md if an entry closed.
  4. Re-sync the marked counts. Adding a claim to a document moves the totals the checker recounts.
  5. Then journal and changelog, as their own commit, and the changelog hash as a third — an entry names the commit it landed in, which cannot be known until that commit exists.

An oracle that compares answers cannot see a defect that is only slow

Every check in this repository asks whether the bytes came out right, and that is the strength of them: oracle.sh holds a program against one somebody else wrote, sweep.sh holds a decompressor against the tool that produced its input, expect.sol runs the documentation. None of them can see a program that is correct and unboundedly slow, and on 2026-09-04 sort had two such defects in nine lines of reader.

reader:fill’s loop condition searched the whole buffer for a newline on every read, and reader:next copied the whole tail for every line. Both are quadratic in the length of a line longer than one read. A 4 MB line took 7.71 s where the tool takes none; 16 MB would have taken about two minutes. The program had been held against /usr/bin/sort over 1,127 runs a day earlier and agreed on every one of them, because it agreed.

Three things follow, and each was paid for on the day.

A conversion can multiply a constant without touching the shape. The scan was quadratic before readUpTo and after it; what changed is that readKey accumulated 65,536 bytes before each rescan and readUpTo answers out of a 4,096-byte window, so it happened sixteen times as often. A 2 MB line went from 0.95 s to 1.90 s in the commit that was an improvement, and the review that found it was looking at something else.

The input shape that shows it is the one no corpus has. The longest line in anything this repository’s checks read was 1,711 bytes, against a 4,096-byte read — so the branch that fills twice was reached by nothing. A case with lines of 30,000 and 4,097 bytes was added the same day for correctness and did not show this either: three orders of magnitude short. Size is a dimension a corpus has to be told about twice — once for the file and once for the record.

And what raised it was a number that looked like a tuning question. A constant where the larger value was the slower one is not a constant to tune; it is a symptom, and asking why bought two defects where acting on it would have bought 16% and left both. See sort.sol for the measurements and performance.md for why neither --steps nor --memory=N reported any of it.

The checker checks what it can run

expect.sol executes the claims in the documentation — over a thousand of them — and that is why the numbers in these pages can be trusted. It only checks claims it can execute.

Nothing is slower is a claim. It is not one the checker can see, and it was wrong by three orders of magnitude for a day. On 2026-08-30 five sentences standing in the documentation were tested and four were wrong, and not one was found by a program failing. They were found by going to check a sentence before repeating it.

The habit that catches them is not a tool. It is refusing to write because X until X has been run once.

And a program that measures a class cannot load a library that extends it

Found on 2026-08-30 when expect.sol began reporting the wrong number of messages an integer answers. It includes text.sol now, and text.sol puts two methods on integer. The count is read before the includes for that reason.

Four of the nine libraries add methods to built-in classes and five do not, so the hazard is real and invisible until it fires.