How often a diary search finds a pattern that is not there
A test I ran on made up diaries. Published 23 September 2026. Not peer reviewed.
I built 400 diaries, 100 at each of four lengths, where nothing affected how the person felt. A plain search for correlations reported a pattern in 370 of them, and in 97 of the 100 longest. The method I built reported one in 9 of the 400, about 2 in 100. It also misses a real pattern five times in six on a two month diary, where a strictly corrected search finds it more often, and it breaks when the numbers drift slowly. All of it is here, with the code to check it.
Data: made up diaries, 100 at each of four lengths. No real person’s records.
Planned and later: only the first two tables come from the planned run. Every other number on this page comes from checks added afterwards, once those results were in. CHECKS.md lists them all.
Check it: download the bundle, open the folder inside, and run its two scripts. The first rebuilds the two main tables in about 90 seconds; the second rebuilds every other number in about nine minutes.
Corrections: before publishing, a review pass found 17 mistakes in my first writeup. They are listed below.
Outside review: none yet.
My interest: I am building a paid Mac app meant to do this kind of reading. It is not released, and this method is not in it yet.
The result
Each cell counts diaries, out of 100, where the method reported at least one pattern. Nothing in these diaries affects how the person felt, so every one of these is a false alarm about the person.
| Method | 60 days | 90 days | 180 days | 292 days | All 400 |
|---|---|---|---|---|---|
| A. Plain correlation search | 90 | 88 | 95 | 97 | 370 |
| B. The same, strictly corrected | 6 | 11 | 12 | 10 | 39 |
| C. The method I built | 3 | 4 | 2 | 0 | 9 |
The 0 is method C’s best cell, not its rate. Its rate is about 2 in 100: 9 of 400, with a 95% range of 1 to 4 in 100. Both methods ran on the same diaries: 34 were flagged by B alone and 4 by C alone, a gap far too large to be chance (exact McNemar test, p below 0.000001).
The half that keeps it honest
A method that never reports anything would score perfectly above and be useless. So the same three ran on 100 more diaries per length with one real pattern planted: steps raising the outcome two days later, a rank correlation of about 0.38.
| Method | 60 days | 90 days | 180 days | 292 days |
|---|---|---|---|---|
| A. Plain | 72 | 82 | 96 | 100 |
| B. Corrected | 29 | 38 | 80 | 97 |
| C. Mine | 16 | 29 | 71 | 96 |
At 60 days my method misses a real pattern five times out of six. It takes about six months of diary before it finds one this strong most of the time, and a weaker pattern takes longer. The corrected search finds the planted pattern more often at every length; my method gives some of that up to raise fewer false alarms.
| Method | 60 days | 90 days | 180 days | 292 days |
|---|---|---|---|---|
| A. Plain, 32 checks | 5 | 9 | 13 | 6 |
| B. Corrected, 32 checks | 24 | 36 | 64 | 80 |
| B. Corrected, held to my 21 checks | 31 | 37 | 70 | 86 |
| C. Mine, 21 checks | 16 | 28 | 68 | 95 |
Scored on the same 21 checks, the corrected search has more clean finds than mine up to six months, and mine has more at 292 days. The plain search finds almost everything and almost always names something false beside it: in 94 of the 100 planted diaries at 292 days.
Where my method breaks
These diaries have no slow trends. Real ones can: a long recovery, a slow decline, a change of medicine. So, in a later check, I added a slow, independent wander to every column and ran 100 empty diaries of 292 days at each size of wander.
| Daily wander added to every column | B. Corrected | C. Mine |
|---|---|---|
| None (the study) | 10 | 0 |
| Small, 0.03 | 14 | 2 |
| Mild, 0.05 | 28 | 5 |
| Clear, 0.10 | 62 | 35 |
| Strong, 0.15 | 89 | 49 |
The wander is a random walk: each day it moves by a random step with that standard deviation, on a scale where the day to day noise has a standard deviation of about 1.15. The 0 in the first row is my method’s best cell; without wander its rate is about 2 in 100, so the small and mild wanders did not clearly change it. The wander rows also use a slightly different random stream. The plain search flagged 97 of the diaries without wander and 98 to 100 of every set with it.
This is a well known trap: two series that each drift look linked when they are not. My method rebuilds each measure from randomly picked blocks of a week or more, which respects the way today leans on yesterday, but not a trend that runs for months. A long diary with a clear upward or downward drift is exactly where it will mislead you. Stronger day to day persistence did not hurt it: 1 or 2 in 100, where the corrected search reached 58 to 90.
Why the plain search fails
Most of it is arithmetic. Each diary gets 32 checks: eight measures, each at zero to three days before. At 1 in 20 each, chance alone would flag about 81 in 100. Here is the plain search at 292 days as the diaries get more realistic, from a later check:
The step from 79 to 93 is the part people miss. When both the measure and the outcome lean on the day before, as every column here does, the pair holds fewer independent days than it has rows, and the textbook p value for that pair comes out too small. From the leaning alone, each single check here is wrong about 11 times in 100 instead of the 5 it promises; with the weekend rhythm added it is about 12.
That is why a correction on its own does not fix it. Bonferroni promises at most 5 in 100, however the checks relate to each other, as long as nothing is there to find. The weekend rhythm puts a small, real calendar link into a few of the checks, so I also rebuilt the same diaries without it, where every check truly has nothing to find. In that later check, at 292 days, Bonferroni still flagged 19 of 100, almost four times its promise. Bonferroni is working; the p values going into it are wrong. The strict correction I used as method B flagged 10.
Method C replaces the textbook p value with one built by rebuilding the measure from random blocks of seven days or more, thousands of times, while the outcome stays put. Most of its advantage comes from that: held to the same 21 checks, the corrected search still flagged 31 of 400 empty diaries against my 9. Its weekday step barely matters on these diaries: without it, the engine flagged 2, 2, 4 and 1 in 100.
Resampling is not the only fix. The older one is to take out each column’s own day to day leaning first and then correlate what is left. I did not test that in the study. In a later check on the same empty diaries, a simple version of it, held to my 21 checks, also flagged about 2 in 100, but it found the planted pattern less often than my method: 57 against 71 in 100 at six months. So what my method adds over it is finds, not fewer false alarms.
What was tested
The diaries. One outcome, a daily score where higher is worse, and eight measures: steps, sleep hours, pain, caffeine, stress, screen time, minutes outdoors and resting heart rate. Every series leans on the day before (an AR(1) process with phi 0.5). The outcome and three of the measures share a weekend bump, which gives a small same day correlation of about 0.07 that belongs to the calendar, not the person. 15% of entries are missing at random. Nothing in a diary affects how the person felt. Lengths are 60, 90, 180 and 292 days, and every diary rebuilds from a fixed seed.
A. Plain search. Spearman correlation of every measure against the outcome, zero to three days before, keeping anything with p below 0.05. 32 checks per diary.
B. Strictly corrected. The same 32 checks, corrected with Benjamini and Yekutieli, keeping anything with q below 0.06, the bar my engine uses. On these diaries it flagged fewer empty diaries than Bonferroni or Benjamini and Hochberg did, so it flatters the correction.
C. The method I built. It removes each weekday’s own average, leaves out pain because it reads like a symptom rather than a cause, checks one to three days before (21 comparisons), builds each p value from thousands of block resamples of the measure, then applies the same correction. It lists same day links apart; counting them too changes each cell by at most 1. Three of its settings, the 0.06 bar, the rule for block length and the choice of correction, were picked earlier by testing the engine on made up diaries much like these, so its low false alarm rate here is partly by design. The slow wander test above is one it was not tuned for, and it fails there.
Same diaries for everyone. Every method ran on the same diaries. Held to method C’s 21 comparisons, the plain search still flagged 87 of 100 at 292 days and the corrected one 9.
What the first writeup got wrong
Before publishing, a second review pass, run with the same AI assistant that helped write the first one, reread everything written about this the way a hostile reviewer would. Every number reproduced. These sentences did not:
- I wrote that the planted pattern was a rank correlation of about 0.30. It is about 0.38, so every find rate above is for a stronger pattern than I claimed.
- I called the result for my method “0 of 100”. That is its best cell. Its rate is about 2 in 100.
- I called method B “the usual correction”. On these diaries it was the most cautious one; at 292 days, Bonferroni and Benjamini and Hochberg flagged 31 and 32 in 100.
- I called method C “what Arkhelion’s engine does”. The Mac app does not contain it yet, and the web page that ran the engine was withdrawn in September 2026.
- I wrote that the diaries held no relationship of any kind. They share a weekend bump, a small calendar correlation.
- I described method C as method B plus two steps. It also drops the pain column and counts 21 comparisons, not 32.
- I listed the diaries’ realism and left out slow trends, which is where my method breaks.
- I called the method written in advance. I committed it 93 seconds before the recorded run, with the study script already written. That is a record made in one sitting, not a preregistration.
The full list of 17, with the evidence for each, is ERRATA.md; item 8 above has its own section at the end of that file. The original method and results are left exactly as they were written.
What this does not show
- These are made up diaries, not people. They are closer to real logs than clean random numbers, but they are not the same thing.
- No app and no AI assistant was tested. Method A is the kind of search a spreadsheet makes easy.
- One size of planted pattern. A weaker one would be missed more often by every method.
- “Pattern” never means cause, in any method.
- Method C is not in any product yet.
Check it yourself
Download the bundle, unzip it, open the folder inside, and run:
node false-alarms.cjs --out rerun.json # the two main tables, about 90 seconds node checks.cjs --out rechecks.json # every later check, about nine minutes
Then compare rerun.json with results.json and rechecks.json with checks.json: they match on every line but the date. Node only, no packages, tested on Node 26. Or read the files one by one: the study script, the checks, the engine, the weekday step, results.json, checks.json, CHECKS.md, PROVENANCE.md.
The engine file’s sha256 is b8b41ec422ec7717a41b59b87fb19124a692e78810a0a63c7f45fa1b43d45be4, the same as the copy the withdrawn web page used.
Terms. The writing and the results are CC BY 4.0: quote them, chart them, publish what you find, including against me. If you quote a number, quote its limit beside it. The code is published so anyone can read it and run it to check these numbers; it is not licensed for reuse in other software.
How this was made. The test code and these writeups were written with Claude, an AI assistant. A second review pass, also run with Claude, found the mistakes listed above and checked each one against the code. The numbers come from the code in the bundle, and anyone can reproduce them.
Cite as: VomSteeg, B. (2026). How often a diary search finds a pattern that is not there. https://arkhelion.ai/study/
Review
No one outside has reviewed this yet. If you check it, write to me at brock@arkhelion.ai. I will add your name here, including if you found a mistake.