How it works

How real-time file detection works

A scanner that runs nightly finds yesterday’s shell. A scanner that runs on every write has to solve four problems the nightly one never meets. Here they are, with the trade-offs visible.

The problem with scheduled scanning

A cron job that walks /home at three in the morning has one virtue: it is simple. Everything else about it is a compromise.

A webshell uploaded at 09:15 has eighteen hours to be used before anything notices. In those eighteen hours it can be used to send mail, host a phishing page, pivot to another account or install a second backdoor somewhere the next scan will not look. By the time the scan reports, the file it finds is often not the interesting one any more — it is the one the attacker left behind on purpose.

Walking a terabyte of customer data also costs real I/O. On a shared host that is felt by every customer on the machine, which is why so many scheduled scans end up throttled to the point of irrelevance, or quietly disabled.

Real-time detection changes the question from “what is on this disk” to “what just changed”, which is a much smaller question and a much more useful one.

Two kernel interfaces, and why the choice matters

Linux offers two ways to be told about file changes.

inotify watches directories. It is available almost everywhere and needs no special privileges, and that is the end of the good news. Every directory needs its own watch, watches are a limited per-user resource, and recursion is your problem: a new subdirectory needs a new watch, and between its creation and your watch there is a gap. On a host with hundreds of accounts and deep WordPress trees, you either exhaust the watch limit or raise a global sysctl that affects everything else on the machine.

fanotify can mark an entire mount. One mark covers every file under it, including directories that do not exist yet. That removes the recursion problem and the watch-exhaustion problem in one move. The price is CAP_SYS_ADMIN, and — for the modes that report parent directory and name — a recent enough kernel with exportable file handles.

Shelltrap prefers fanotify with FAN_REPORT_DFID_NAME and marks per mount, and falls back honestly when it cannot have it:

Capability is probed on the running host. A kernel version number is a hint; the probe is the answer.

The important design decision there is not which tier is used. It is that the tier is reported. A container that can only manage delayed detection says so in the product rather than presenting itself as real-time.

What counts as an event

The obvious event is “a file was written”. Restricting yourself to that misses two of the most common persistence techniques on shared hosting, so the mask is wider:

CLOSE_WRITE — a write finished. The usual moment to scan, because the file is complete.
MOVED_TO and MOVED_FROM — a rename in or out. Attackers stage a file elsewhere and move it into place precisely because a naive watcher only looks at writes.
CREATE and DELETE — appearance and disappearance.
ATTRIB — attribute changes. A file uploaded as harmless data and then made executable, or given a setuid bit, is an attack that involves no write at all.

There is also a subtraction. A filesystem-level mark on the root filesystem delivers every open on the host, including the database server’s data directory and every log write. Those paths are marked ignored on purpose. Watching everything is not more secure; it is slower, noisier and more likely to be switched off by an exasperated administrator.

The privilege split

Here is the uncomfortable truth about any file scanner: it is a program that opens attacker-chosen files and feeds them to archive, image, document and executable parsers. That is the most dangerous code in the system. A parser bug in an archive library is an attacker-controlled memory-safety bug, and if that parser runs as root, so does the attacker.

So Shelltrap splits the job:

Privileges on the left of the SCM_RIGHTS line, parsers on the right. Neither side has both.

The broker has the privileges and no parsers. It holds the fanotify groups, opens files with openat2 and RESOLVE_NO_SYMLINKS (or a component-wise openat walk with O_NOFOLLOW where that syscall is unavailable), and passes a read-only descriptor onwards. It contains no YARA code, no ClamAV code, and no archive, Office or ELF parsing. It accepts opaque object IDs from the panel and the UI, never a path.

The worker has the parsers and no privileges. It runs as its own user in its own mount and PID namespace, with seccomp, Landlock, rlimits and cgroup limits — and no network at all. It receives descriptors; it never resolves a path itself, so a symlink race in a customer directory has nothing to race against.

That split is the single most consequential design decision in the product, and it is worth checking for in any scanner you evaluate: what runs as root, and does it parse?

When the queue overflows

Real-time systems have a failure mode that batch systems do not: the events arrive faster than you can process them. A customer restores a backup, a deploy writes ten thousand files, an attacker unpacks an archive on purpose. The kernel queue fills and the kernel tells you it dropped events.

There are three possible responses and only one of them is honest.

You can ignore it, and quietly have a gap in coverage nobody knows about. You can crash, and have no coverage at all. Or you can count it, alarm on it, and reconcile: run a bounded scan over the affected subtree to find what was missed.

Shelltrap does the third. The same reconciliation runs after a reboot, a crash, an upgrade, and whenever a new mount appears under a customer root — all situations where events happened that nobody was listening for. Reader and scan queue are separate goroutines with a persistent job journal in SQLite, so a restart resumes rather than restarts.

There is a cost, and it is worth knowing before you install: a first start without a clean shutdown marker plans a full crawl. On a busy host that backlog is real work, queue age climbs while it drains, and health stays degraded until it has. That is the correct behaviour — the scanner really is behind — but it is much nicer if you set cgroup limits before the first start rather than after.

Why “not scanned” is its own answer

Every scanner has limits: a maximum file size, a maximum scan time, a maximum archive recursion depth, a maximum expansion ratio for compressed data. Without them, a single crafted archive takes the machine down.

The question is what you report when a limit is hit. A worrying number of tools quietly count the object as clean, because the alternative is an ugly number on a dashboard. That turns a resource limit into a detection gap that is invisible in exactly the place you would look for it.

Shelltrap has five verdicts instead of two for this reason:

Verdict	Means
`clean`	scanned by every engine in the active profile, nothing found
`suspicious`	several independent signals, below the action threshold
`malicious`	a signature or hash hit, or a heuristic score above the threshold
`unscanned`	a limit was hit; the reason is recorded and it is queued for a bounded deep scan
`degraded`	the scanner itself was impaired — clamd gone, a worker dead, a rule set missing

unscanned and degraded are never displayed as clean. This makes some dashboards less green. It also makes them true.

The cache, and why a rule update invalidates it

Scanning the same unchanged file repeatedly is waste, so verdicts are cached. What goes into the cache key decides whether the cache is a performance win or a security hole.

The key contains the mount ID, the inode, high-resolution mtime and ctime, the size, the content hash — and the rule set, engine and policy generations. That last part is what makes a signature update invalidate old clean verdicts automatically: a file that was clean under yesterday’s rules has not been scanned under today’s.

Path-dependent policy verdicts live in a separate cache, because context can change without the content changing at all. The same inode can appear in a new place through a hard link or a rename, and “PHP in an uploads directory” is a statement about the place, not the bytes.

What real-time detection is not

It is not prevention. A file that is written is written; detection happens after the write completes, in milliseconds rather than hours, but after. Blocking the write itself at the kernel level is possible with fanotify permission events and is prototyped for a later release, deliberately not promised today — because a scanner holding a permission decision on every file open is a scanner that can hang a server if it stalls.

There is one path where prevention exists today, and it is the one that matters most for a web host: the synchronous upload gate , which judges a web upload before your application ever sees it.