Linux Persistence: ld.so.preload and Generators

pajus 2026-10-10 12 min read

After an attacker gets root, the next thing they want is a way to survive a reboot, a password change and your first look around. Cron jobs and a new .service file are the obvious places, and most admins check them. This post covers Linux persistence in three places that get checked less often: systemd generators, /etc/ld.so.preload together with LD_PRELOAD, and unit files that no package installed.

Most write-ups on this topic are detection rules for Elastic, Splunk or an EDR, and they assume you already collect that telemetry. This one sticks to a single Debian 13 host and plain tools: dpkg -S and debsums to sort packaged files from planted ones, /proc to see what is actually loaded, and auditd rules to catch the next attempt. It contains no exploit code. Run these checks only on systems you own or are authorised to investigate.

Versions referenced: Debian 13 (trixie) ships systemd 257.13-1~deb13u1 and busybox-static 1:1.37.0-6+b9.

How Linux persistence via systemd generators works

Generators are small executables that systemd runs before it loads unit files. They run very early at boot and again on every systemctl daemon-reload. Their job is to turn non-native configuration (for example /etc/fstab) into units. According to systemd.generator(7), system generators are looked up in:

/run/systemd/system-generators/
/etc/systemd/system-generators/
/usr/local/lib/systemd/system-generators/
/usr/lib/systemd/system-generators/

User generators have matching user-generators directories. A generator writes its output to one of three directories, and they do not all carry the same weight:

Output directory	Precedence
`/run/systemd/generator.early`	Overrides everything, including admin units in `/etc`
`/run/systemd/generator`	Overrides vendor units, but not admin units
`/run/systemd/generator.late`	Only adds units, never overrides

That makes a generator a strong foothold. An attacker drops an executable into one of the generator directories, and on every boot it writes a service into /run and links it into multi-user.target.wants. Nothing new appears in /etc/systemd/system, because the unit is recreated in tmpfs every time. Pepe Berba's research also points out two worse details. Generators run before auditd starts, so the boot-time execution itself is not logged. And a generator writing to generator.early can replace auditd.service or your EDR agent's unit with a harmless stub.

Generators run in a sandbox and cannot call systemctl. They can only rely on /usr, /run, /proc, /sys and /dev, because at boot they run before non-essential filesystems such as /var and /home are mounted. None of that stops them from writing a unit file, which is the only thing the attacker needs from them.

How ld.so.preload and LD_PRELOAD hijack processes

The dynamic linker ld.so loads extra shared objects before a program's normal libraries if they are listed in LD_PRELOAD or in /etc/ld.so.preload. LD_PRELOAD libraries are loaded first, then the ones in the file. The file is a whitespace-separated list, and ld.so(8) notes that it affects every program run on the system and is normally only used as an emergency workaround. A healthy server usually has no such file, and no Debian package ships it.

A preloaded library can define functions with the same names as libc functions, so its versions run instead. MITRE ATT&CK tracks this as T1574.006 and lists malware that hooks readdir and execve to hide files and processes from ls, ps and similar tools. Examples include Rocke and Hildegard, which edited /etc/ld.so.preload, and Ebury, which used LD_PRELOAD on OpenSSH servers.

This changes how you investigate. If /etc/ld.so.preload is rigged, every dynamically linked tool you run on that host may be lying to you, including ls, cat, find, dpkg and debsums. Statically linked binaries never go through ld.so, so the hook doesn't apply to them. Inspecting the disk from rescue media gets around the hook as well. A kernel-level rootkit is a different matter: it can lie to static binaries too, and only an offline check will catch it.

For setuid binaries, ld.so runs in secure-execution mode. It strips LD_PRELOAD from the environment and ignores preload paths that contain slashes. So a user-level LD_PRELOAD on its own is not a privilege escalation path. The system-wide file, which needs root to write, is the persistence risk.

Detecting the footholds on Debian 13

Step 0: get a trustworthy toolset

Ideally, bring a known-good static busybox on read-only media. If you install it on the suspect host, keep in mind that apt itself is dynamically linked:

apt install busybox-static
file /bin/busybox
busybox ls -la /etc/ld.so.preload
busybox cat /etc/ld.so.preload

Then compare that with what the normal tools report. If busybox ls shows the file and /usr/bin/ls doesn't, you have already confirmed a userland rootkit:

diff <(busybox ls -1a /etc) <(ls -1a /etc)

Step 1: what is actually loaded right now

/proc/PID/maps shows every library mapped into a process, whichever way it got there. Read it with busybox and check who owns each library. On Debian 13 with merged /usr, the dpkg database can record a path as either /lib/... or /usr/lib/..., so the loop tries both:

busybox cat /proc/[0-9]*/maps 2>/dev/null | busybox awk '$6 ~ /\.so/ {print $6}' | sort -u > /root/mapped-libs.txt
while read -r lib; do
  dpkg -S "$lib" >/dev/null 2>&1 || dpkg -S "${lib#/usr}" >/dev/null 2>&1 || echo "UNOWNED: $lib"
done < /root/mapped-libs.txt

Expect a few legitimate hits from software you installed by hand under /opt or /usr/local. Anything in /tmp, /dev/shm, /var/tmp or a hidden directory needs a closer look.

The loop above only sees the path, so it misses one important case: a library that was deleted from disk after loading. In maps that shows up as a separate (deleted) marker after the path, and a deleted library at a path dpkg owns would pass the ownership check. Check for it separately:

busybox grep ' (deleted)$' /proc/[0-9]*/maps 2>/dev/null | busybox grep '\.so'

A deleted library can be left behind by a normal package upgrade that ran without restarting the service, so compare against your recent upgrades before you panic.

Step 2: LD_PRELOAD in process environments

/proc/PID/environ holds the environment the process was started with, separated by NUL bytes:

for e in /proc/[0-9]*/environ; do
  if tr '\0' '\n' < "$e" 2>/dev/null | grep -q '^LD_PRELOAD='; then
    pid=${e#/proc/}
    pid=${pid%/environ}
    printf '%s: ' "$pid"
    tr '\0' ' ' < "/proc/$pid/cmdline"
    echo
  fi
done

One caveat from proc_pid_environ(5): the file shows the environment at execve() time and does not update if the process later calls unsetenv(). Malware that clears the variable after loading won't show up here, which is why Step 1 is the more reliable check. Then look for where the variable gets set:

grep -rns 'LD_PRELOAD' /etc/environment /etc/profile /etc/profile.d /etc/bash.bashrc /etc/systemd /usr/lib/systemd 2>/dev/null
grep -ns 'LD_PRELOAD' /root/.bashrc /root/.profile /home/*/.bashrc /home/*/.profile /home/*/.ssh/environment 2>/dev/null
systemctl show-environment

Step 3: generators nobody installed

Every file in a generator directory should belong to a package. This loop reports the ones that don't:

for d in /run/systemd/system-generators /etc/systemd/system-generators /usr/local/lib/systemd/system-generators /usr/lib/systemd/system-generators /run/systemd/user-generators /etc/systemd/user-generators /usr/local/lib/systemd/user-generators /usr/lib/systemd/user-generators; do
  [ -d "$d" ] || continue
  for f in "$d"/*; do
    [ -e "$f" ] || [ -L "$f" ] || continue
    dpkg -S "$f" >/dev/null 2>&1 || dpkg -S "${f#/usr}" >/dev/null 2>&1 || echo "UNOWNED: $f"
  done
done

Ownership alone isn't enough, because an attacker can also overwrite a packaged generator. debsums compares installed files against the package's stored checksums:

debsums -s systemd udev
debsums -c

debsums: changed file /usr/lib/systemd/system-generators/systemd-fstab-generator (from systemd package)

Know the limits. The checksums live in /var/lib/dpkg/info/*.md5sums on the same disk, and the debsums man page itself says the tool is of limited use for security. Root can rewrite those files. Treat a clean result as a weak signal and a dirty one as a strong one.

Next, check what the generators actually produced, and confirm that your security services still load from their real unit files:

ls -la /run/systemd/generator.early /run/systemd/generator /run/systemd/generator.late
systemctl show -p FragmentPath auditd.service ssh.service

FragmentPath=/usr/lib/systemd/system/auditd.service
FragmentPath=/usr/lib/systemd/system/ssh.service

A FragmentPath under /run/systemd/generator.early for auditd means someone is suppressing your logging.

Step 4: unit files with no package

/etc/systemd/system is full of enablement symlinks that dpkg doesn't track. Regular files are what you want to look at:

find /etc/systemd/system /usr/local/lib/systemd/system /run/systemd/system /usr/lib/systemd/system -type f \( -name '*.service' -o -name '*.timer' -o -name '*.socket' -o -name '*.path' -o -name '*.conf' \) 2>/dev/null | while read -r u; do
  dpkg -S "$u" >/dev/null 2>&1 || dpkg -S "${u#/usr}" >/dev/null 2>&1 || echo "UNOWNED: $u"
done
systemd-delta --type=extended,overridden

Your own units and drop-ins will show up in the list. Read each one with systemctl cat and check ExecStart=, ExecStartPre= and Environment= for anything you don't recognise.

Step 5: check from rescue media

When the online checks contradict each other, boot a Debian live or rescue image, mount the root filesystem read-only, and run the same checks against the dpkg database on that disk. A plain -o ro mount is not enough for evidence: on ext4 and other journalled filesystems it still replays a dirty journal, which writes to the disk. Set the block device read-only first and, on ext4, add noload to skip journal replay:

blockdev --setro /dev/sda2
mount -o ro,noload /dev/sda2 /mnt
cat /mnt/etc/ld.so.preload
ls -la /mnt/etc/systemd/system-generators /mnt/usr/local/lib/systemd/system-generators
dpkg-query --admindir=/mnt/var/lib/dpkg -S /usr/lib/systemd/system-generators/systemd-fstab-generator
debsums --root=/mnt -s

Replace /dev/sda2 with your actual root device (check it with lsblk). noload is an ext4 option; for other filesystems check their own option for skipping log recovery.

Defending: auditd rules and integrity baselines

Generators run before auditd, so you won't see them execute at boot. You will see them being written, along with writes to the preload file and new units. The auditctl(8) man page marks -w watches as deprecated and slow, so these rules use the syscall form. Create the optional directories first so the rules have something to attach to. An empty generator directory is harmless:

mkdir -p /etc/systemd/system-generators /usr/local/lib/systemd/system-generators

-a always,exit -F arch=b64 -F path=/etc/ld.so.preload -F perm=wa -F key=ldpreload
-a always,exit -F arch=b64 -F dir=/etc/ld.so.conf.d/ -F perm=wa -F key=ldpreload
-a always,exit -F arch=b64 -F dir=/etc/systemd/system-generators/ -F perm=wa -F key=sd_generator
-a always,exit -F arch=b64 -F dir=/usr/local/lib/systemd/system-generators/ -F perm=wa -F key=sd_generator
-a always,exit -F arch=b64 -F dir=/usr/lib/systemd/system-generators/ -F perm=wa -F key=sd_generator
-a always,exit -F arch=b64 -F dir=/etc/systemd/system/ -F perm=wa -F key=sd_unit
-a always,exit -F arch=b64 -F dir=/usr/local/lib/systemd/system/ -F perm=wa -F key=sd_unit

If your rule set ends with -e 2 (immutable mode), new rules only take effect after a reboot. Otherwise:

augenrules --load
auditctl -l | grep -E 'ldpreload|sd_generator|sd_unit'
ausearch -k ldpreload -i
ausearch -k sd_generator -i

Package upgrades will trigger sd_generator and sd_unit events from dpkg. That noise is expected. Events from a shell, curl, or a web server user are not. My auditd guide for Debian 13 explains how to keep a rule set like this quiet and how to report on it with aureport.

Audit logs record changes as they happen. An integrity baseline catches changes you missed. Add the generator directories, /etc/ld.so.preload and the unit directories to your AIDE configuration, and store the database off-host, as described in AIDE file integrity monitoring on Debian 13. Ship audit logs to a remote collector as well, so an attacker with root can't quietly delete them.

The best defence is still not giving anyone root in the first place. These footholds usually show up after a service was exposed and then abused. How exposed Redis gets rooted walks through one common chain.

Triage and cleanup order

Order matters here, because some of the obvious first moves make things worse.

Don't reboot and don't run systemctl daemon-reload yet. Both rerun every generator, which means running the attacker's code again.
Collect evidence first. Copy suspicious files, their hashes, /proc/PID/maps and /proc/PID/environ of affected processes, and the output of ls -la /run/systemd/generator* to another machine.
Neutralise /etc/ld.so.preload first, because it hides everything else. Remove the malicious line, or the whole file if you never created it, and only then delete the library. Then run your checks again with normal tools and compare with busybox.
Remove unowned generators. Don't reinstall packages yet.
Remove unowned units and disable them.
Only after steps 3 to 5, restore modified packaged generators and reload. Run apt install --reinstall systemd. Its maintainer scripts normally reload or re-execute the manager, which reruns all generators, so this step must come after everything unowned is gone. Then run systemctl daemon-reload and confirm that /run/systemd/generator.early contains nothing unexpected.
Rotate every credential the host held: SSH keys, service passwords, API tokens.

Warning: killing processes and removing units can stop production services. Plan a maintenance window if you can. If the online and offline checks disagree, or you find a kernel module you can't explain, stop cleaning and rebuild from known-good media. You can't trust a cleaned host as much as a fresh one.

Takeaways

Check /etc/ld.so.preload with a static binary. It shouldn't exist unless you created it.
Read /proc/*/maps to see loaded libraries, including ones marked (deleted), and use /proc/PID/environ to look for LD_PRELOAD. Remember that the second one can be evaded.
Every file in the generator directories and every regular unit file should be owned by a package (dpkg -S) and unmodified (debsums -c).
Confirm that auditd.service still loads from /usr/lib/systemd/system.
Add syscall-form audit rules for the preload file, the generator directories and the unit directories, plus an AIDE baseline that you store off-host.
Clean up in this order: evidence, then preload, then generators, then units, then package reinstall and daemon-reload, then credentials. Rebuild if you're in doubt.