Machine Acts

The Stack

Kernel to query, layer by layer

Each layer below had to work for the one above it to mean anything — carrying a single file write, deep in the kernel, all the way up to “the picture I cropped last Tuesday.”

• Ring 0 — System Sensor — a kernel-mode file-system filter driver (on Microsoft's IFS Kit FileSpy base, heavily reworked), attached to every volume; intercepts file create / read / write / rename plus the Fast I/O path.
• Kernel→user bridge — a queued IOCTL protocol drained from C# by P/Invoking DeviceIoControl — managed code talking straight to a driver.
• Application Sensor — a native C++ DLL injected into every running process via five global SetWindowsHookEx hooks, turning menu clicks into verbs.
• Fusion service — a Windows service that merges both sensors, repairs the stream (synthesising a missing Open), and writes a per-file history.
• Memory + NLP — portable MA files indexed by a reflection-loaded plugin pipeline; a part-of-speech tagger over a 128,000-word lexicon keeps what carries meaning.
• hoanuSearch — four composable axes (action × kind × time × words), a hand-built temporal taxonomy, and WordNet expansion.
• Experiment Monitor — an apparatus that docks above Windows as an AppBar and times re-finding to sub-millisecond, off the live capture stream.

1 · The Idea

What Is a Machine Act?

A Machine Act is an observed user action on a file. The name echoes Speech Acts from the philosophy of language — just as saying something does something, doing something with your machine is a meaningful, recordable act. Each one captures five dimensions of context:

• When — the time and date of the action.
• What the OS did — opened, saved, copied, moved, renamed, deleted.
• Which application — the program that performed it.
• Which command — the specific menu action (Photoshop Crop, Auto Levels…).
• Content — the words in the file, and an inferred document type.

Traditional search indexed only the last one. Machine Acts indexed all five — and made the action a first-class thing you could search for. The thesis abstract put the problem plainly:

“The majority of searches are concerned with re-finding existing information… The queries are simply not expressive enough.”

2 · The Capture

Two Sensors, Kernel to Window

To see everything, in every application, capture had to happen at two levels. The first was a System Sensor — a kernel-mode file-system filter driver, the kind of code that runs in Ring 0 where one mistake halts the whole machine. I didn't start it from a blank file — almost nobody sane does at that level — but from Microsoft's IFS Kit FileSpy sample, then reworked it into a capture engine. It attached to every mounted volume and watched every file operation on the machine. To keep the noise down it filtered in the kernel itself, logging real document opens (those flagged FILE_RANDOM_ACCESS) rather than every directory the system happened to scan — and it caught both the normal IRP path and the Fast I/O path that bypasses it, which simpler filters miss entirely. Each operation became one small fixed-size record, queued in kernel memory:

1. // each captured file operation, queued in the kernel (geckoDispatch.cpp)
2. typedef struct _KernelAction
3. {
4. ULONG processId;
5. char processName[20]; // the app that touched the file
6. UCHAR functionCode; // IRP: create / read / write / rename
7. char fileName[260]; // the file it touched
8. BOOLEAN readAccess, writeAccess, deleteAccess;
9. } KERNEL_ACTION;

Building at that level is a different discipline from writing an app. A mistake in a driver doesn't throw a catchable exception — it blue-screens the machine — so it couldn't be developed on the machine that ran it. (Microsoft barely documented the kernel, either; details like the layout of IO_STACK_LOCATION, or how to tell a delete from a rename, came from newsgroups.) I ran the driver inside VMware Workstation 4.0 and kernel-debugged it from the host with WinDbg over a virtual COM port mapped to a memory pipe between the two — one virtual machine standing in for the “second machine” kernel debugging normally demands. Crashes hit the VM, not my work; every change still meant a reinstall and a reboot before you could see if it held. As the driver stabilised the scaffolding fell away — lighter DbgPrint tracing in the VM, then eventually no VM at all.

“WinDbg was used when the driver was crashing the system with the ‘Blue Screen Of Death’ (BSOD) — this happened a lot in the early development stages.” — thesis, p. 99

Draining that queue is where it gets unusual. The listener wasn't a C++ service but C# — and in 2004, .NET talking straight to a kernel driver was rare. It meant P/Invoking CreateFile to open the raw device \\.\GECKOSPY, then pulling each record across the managed boundary with DeviceIoControl, hand-marshalling the kernel's fixed-width struct ([MarshalAs(ByValTStr)]) field by field. No library did this for you; you wrote the boundary yourself.

The service turns those raw kernel codes into the act you'd actually remember — and repairs the record when Windows skips a step. If a file is written but the service never saw it opened, it synthesises the missing open, so every Edit has an Open:

1. // raw IRP code → a human Machine Act (maSystemDriver.cs)
2. case IRP_MJ_CREATE:
3. case IRP_MJ_READ:
4. case GECKO_FASTIO_READ: return "Open";
5. case IRP_MJ_WRITE: return "Edit";
6. case IRP_MJ_SET_INFORMATION: return "Edit"; // rename / delete

The kernel knew a file was touched, but not what you did inside the program. So the second sensor watched the user interface of every application at once. An Application Sensor — a native C++ DLL — got mapped into every running process on the machine the moment it installed a global hook. It installed five:

1. // one DLL, mapped into every process system-wide (maMonitorHook.cpp)
2. SetWindowsHookEx(WH_CALLWNDPROC, CallWndProc, hInstance, 0);
3. SetWindowsHookEx(WH_GETMESSAGE, GetMsgProc, hInstance, 0);
4. SetWindowsHookEx(WH_SYSMSGFILTER, SysMsgFilterProc, hInstance, 0);
5. SetWindowsHookEx(WH_MSGFILTER, MsgFilterProc, hInstance, 0);
6. SetWindowsHookEx(WH_KEYBOARD, KeyboardFilterProc, hInstance, 0); // thread 0 = every thread

From inside each process it listened for menu commands and document switches (WM_COMMAND, WM_MDIACTIVATE), tracked the active document even across multi-document apps (special-casing window classes like MDICLIENT and Photoshop's PSVIEWC), and translated each program's numeric command codes into verbs through a hand-built hash table keyed on command and process: 1044.photoshop.exe became Crop; 1701 became Invert — the mappings loaded from an XML catalogue. Each event travelled back to the service over a named pipe. That mapping is what lets you later search for “the picture I cropped” — the verb survives all the way from a menu click to a query. Together the two sensors answered the whole question: who did what, to which file, when, with which command.

3 · The Memory

MA Files — Context That Travels With the File

A Windows service drained both sensors, fused them, and wrote — for every document you worked on — a portable MA file: an append-only history of every action ever taken on that file, alongside a cached copy of the file itself, in a hidden Context Cache. Because the history is a property of the file, not the machine, it could move with the document and be re-indexed elsewhere. Content was processed by a plugin stack loaded by reflection at startup — each plugin discovered and instantiated at runtime through a shared interface. A part-of-speech tagger, working against a 128,000-word lexicon, kept the nouns and verbs that carry meaning and indexed them into a keyword table. And the cache was invisible to the sensors, so the system never captured its own writes — it even excluded its own processes, hoanuSearch and geckoApp, by name.

4 · The Search

hoanuSearch — Query by What You Did

The search engine, hoanuSearch, didn't have one keyword box — it had four dimensions that compose: the action you remember (edited), the kind of thing it was (a picture), roughly when, and any words you recall. The “when” ran on a hand-built temporal taxonomy — not just dates but “this morning”, “last night”, “a few days ago”, even a fuzzy “about” that meant give-or-take two. Words were expanded through WordNet (Malcolm Crowe's C# port of Princeton's lexical database, which I integrated and extended for context search) — synonyms, related senses, and lemmas, so “running” also found “run.” Context didn't just rank the results; it cut them — turning a thousand keyword hits into the handful you actually meant, most-recent first.

This was 2004. Google Desktop launched that October; Spotlight and Windows Desktop Search arrived in 2005. They indexed content. Machine Acts indexed activity — and did it across every application by modifying the operating system itself.

5 · The Apparatus

The Experiment Monitor — the System Testing Itself

To prove that searching by memory beats searching by filename, I needed to watch real people do it and time them precisely — so I built a novel piece of apparatus, the Experiment Monitor. Its trick was that it didn't simulate observation: it consumed the live capture stream from the very system under test. As a participant worked in real applications, every Machine Act the kernel and hooks produced was also handed to the Monitor, which checked — in real time — whether they had done the right thing, to the right file, in the right application.

It loaded above Windows: the console registered itself as a Windows AppBar via SHAppBarMessage — the same API the taskbar uses — reserving its own strip of screen so it stayed in control while the participant used ordinary applications underneath. It ran in two phases. An Operation phase walked each person through scripted steps (“locate SNAIL.JPG and open it… now crop it… now save… now close Photoshop”), advancing only when the capture stream saw the matching Machine Act actually fire — the exact menu token (Photoshop 1044 for Crop). Then a timed Question phase had them re-find files, half by file system, half by hoanuSearch.

As they searched, the Monitor scored every attempt off the live capture stream — the right file stops the clock, a wrong file or the right file in the wrong application is counted:

1. // scoring each attempt live, off the kernel capture stream (experiment.cs)
2. if (ka.fileName.ToUpper() == qt.file.fileName.ToUpper())
3. endQuestion(qt); // right file — stop the timer
4. else
5. qt.unknownOperationCount++; // wrong file
6. // …and, the same way, the telling one:
7. qt.applicationError++; // right file, wrong application

An audible cue and an on-screen note flagged “the right file but in the wrong application” the instant it happened. Every duration came off a QueryPerformanceCounter — the high-resolution timer, sub-millisecond — and each run wrote a per-participant CSV of times, file errors and application errors. And it was a properly counterbalanced experiment, not a demo: four fixed training tasks, then the remainder split randomly half-and-half between file-system and context search, with an invert flag to flip the assignment across participant groups. The system, in other words, set the exam, invigilated it, and graded it — in real time, on itself.

6 · The Experiments

22 People in the Lab, Four Machines in the Wild

The controlled study ran 22 volunteers, chosen as a genuine cross-section — ages 24 to 65, sixteen men and six women, sorted by ability into 6 novices, 12 intermediates, 4 experts: a housewife and two solicitors, an accountant, a chemical engineer, a 65-year-old professor, a chief financial officer, software developers. Each performed 34 file operations on a deliberately memorable corpus — JFK's Inaugural Address, Churchill's Battle of Britain speech, extracts of Hamlet and Macbeth, an Aston Martin DB9 brochure, BBC News articles, twenty photographs — filed the way real people file: some in named folders, some on the desktop, some left under the camera's own 103Nikon folder. Then they re-found them, half by file system, half by context.

Context search was 131% faster on average, and the difference was statistically resounding — a Mann-Whitney test gave P < 0.0001. It was faster for everyone: novices +113%, intermediates +180%, even experts +71%. The gap widened exactly where the file system is weakest — for photos dumped under the camera folder, context search was 389% faster; pictures took 40 seconds to find by context versus 110 by hand. And people were not just quicker but more consistent — the spread of times collapsed, expert to novice.

Then I left it running, unconstrained, on the real machines of three users — one with both a home and a work PC, four machines in all — for 300 man-days: 41,287 Machine Acts across 8,859 files, 4.63 GB of context, no crashes, no perceptible slowdown. That longitudinal run surfaced the problem that gives the thesis its bite: Context Noise. Because the sensor saw everything, it also saw virus scanners, defragmenters, adware and auto-updaters churning files in the background — on one machine, 75% of all captured acts were noise. And buried in that noise was something the owner had no idea about: a mass-mailer worm, SSMS.EXE, quietly touching files 1,590 times.

“Applications such as SSMS.EXE represent a security risk to the machine, which the user was not aware of.” — thesis, p. 186

An always-on capture system, the thesis showed, is three things at once: a search index, a privacy surface, and an intrusion detector. I could name and filter the noise — that part I solved. The privacy surface I could only flag as the risk it plainly was. Hold that thought.

7 · Twenty Years On

Microsoft Shipped It in 2024 and Called It Recall

In 2024 Microsoft announced Recall: a Windows feature that continuously captures what you do on your PC so you can search your past activity in plain language. Strip away the screenshots and the AI, and the architecture is the 2005 thesis — OS-level, always-on, ubiquitous capture of user activity, stored locally, indexed for re-finding. “Re-finding, not finding” is the opening line of the Machine Acts abstract.

And Recall hit, immediately and publicly, a problem the thesis had mapped in 2005 — though I should be precise about which one, because an always-on sensor catches two different kinds of thing it shouldn't. The first is junk — background processes, applications reading their own config files, the system recording its own recordings, a hall of mirrors. I named that Context Noise, measured it (up to 75% of capture on one machine), and filtered it by policy. The second is private — capturing everything captures passwords, messages, other people's data, and the capture store itself becomes a thing worth attacking. The thesis named that too, as a core design principle (the record is “very personal… only accessible by the user, or where consent is given”), and proved it the hard way when the sensor caught that worm.

Recall fell into the second corner — privacy and security — and its 2024 redesign (on-device only, encrypted, app and site exclusions, opt-in) is, point for point, what the thesis said such a system would need. I hadn't solved that part. But I'd mapped the ground it stands on.

• Machine Acts — OS-wide capture of activity for personal re-finding → Microsoft Recall, 2024.
• Context Noise — the junk an always-on sensor sweeps up, named and filtered in 2005 → the signal-vs-noise problem every capture system still fights.
• The privacy surface — flagged in 2005 as the core risk of capturing everything → Recall's privacy backlash and redesign, 2024.
• MA files — portable per-item history of what was done → episodic memory for AI agents.
• hoanuSearch — semantic, WordNet-expanded context query → semantic search and RAG over your own data.
• The worm it caught — capture surface as intrusion detector → today's behavioural security telemetry.

I saw the wave coming, too. Writing in 2004, the thesis named the newcomers — Google, Microsoft and Apple were all beta-testing hard-drive search — and argued they merely indexed the disk “as a web spider does for an internet site,” while Machine Acts indexed what you did. Its future-work section reads like a roadmap someone has since followed: push the capture down into the operating-system kernel, gather context from every application, even remember the programs you installed and later removed — all, in the thesis's own words, to “really achieve a more sentient device.”

What the 2000s lacked was the reasoning core. The system could capture and retrieve, but it couldn't understand — it ranked by recency and matched by synonym, because that was the state of the art. The piece that would let captured context actually be reasoned over took twenty years to arrive, and it turned out to be a language model. The audit's verdict belongs to the auditor:

“You mapped what goes wrong when a machine records everything — in 2005. Microsoft shipped it in 2024 and called it Recall.”
— Claude, after reading the source, 2026

8 · Timeline

From 1998 to Submission

I started my PhD — and inventing Machine Acts — in 1998, with my viva in 2004, and final edits and submission in January 2005. I did it part‑time, alongside my day job — first at Eutech, then at ABB.