smartd on Debian 13: SATA and NVMe Disk Alerts

12 min read

Disks rarely die without warning. Usually there are signs first: pending sectors on a SATA drive, a rising media error count on an NVMe drive, or a self-test that fails at one LBA. The problem is that nobody is watching. smartmontools gives you two tools for this. smartctl reads the drive's own health data when you ask for it. smartd polls that data every 30 minutes, runs scheduled self-tests and alerts you when something changes. This post sets up smartd on Debian 13 for a mixed SATA and NVMe box, explains which numbers matter, and shows how to prove that alerts actually reach you. That last part is the one most setups skip.

What smartd and smartctl are for

Use smartctl for ad-hoc inspection: before trusting a used drive, after an I/O error in the kernel log, or when a RAID member gets kicked out. Use smartd for continuous monitoring on any machine with physical disks. Neither one replaces backups or filesystem checksums. SMART tells you what the drive firmware is willing to admit, and that is less than the full picture (see the pitfalls below).

By default smartd checks devices every 1800 seconds (-i changes this). It logs to the syslog daemon facility, which on Debian 13 means the journal. It can save state under /var/lib/smartmontools/, so it remembers attribute values and when it last sent an alert across restarts.

Installing smartmontools on Debian 13

Trixie ships smartmontools 7.4-3. Upstream released 7.5 on 2025-04-30, and Debian has 7.5-2~bpo13+1 in trixie-backports (accepted 2025-10-26). Testing and unstable carry 7.5-2. For SATA-only machines, 7.4 is fine. For NVMe, 7.5 is worth the backport. Its smartd adds NVMe self-test scheduling, an NVMe attribute log, a bitmask for -H, and checks for Available Spare, Percentage Used and Media Errors.

apt update
apt install smartmontools
smartctl --version | head -n 1
apt policy smartmontools

To get 7.5, enable backports and install only this package from there:

Types: deb deb-src
URIs: http://deb.debian.org/debian
Suites: trixie-backports
Components: main
Enabled: yes
Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg
apt update
apt install smartmontools/trixie-backports
systemctl status smartmontools

The package installs smartmontools.service, /etc/smartd.conf, /etc/default/smartmontools and a hook directory /etc/smartmontools/run.d/. It does not depend on a mail command or an MTA. mailx is only a Suggests (met by bsd-mailx or mailutils). That matters later.

Reading smartctl output: the numbers that matter

Start by asking smartctl what it can see:

smartctl --scan-open
smartctl -x /dev/sda
smartctl -a /dev/nvme0

For ATA drives, use -x. The man page no longer recommends -a for ATA, because -a skips the 48-bit logs: extended error and self-test logs, device statistics and SCT temperature history. For NVMe, -a and -x are the same thing. A trimmed example of the SATA attributes that matter:

SMART overall-health self-assessment test result: PASSED
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       8
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       8

Note the PASSED next to eight pending sectors. The overall verdict only flips when a normalized value drops to its threshold, and by then the drive is often already gone. Look at the raw values of 5, 197 and 198 instead. Backblaze's fleet analysis also tracks 187 (Reported Uncorrectable) and 188 (Command Timeout).

For NVMe, the health log is the whole story:

SMART overall-health self-assessment test result: PASSED
Critical Warning:                   0x00
Temperature:                        41 Celsius
Available Spare:                    100%
Available Spare Threshold:          10%
Percentage Used:                    7%
Media and Data Integrity Errors:    0
Error Information Log Entries:      12

Critical Warning is a bitfield. Bit 0 means spare below threshold, bit 1 a temperature threshold, bit 2 reliability degraded, bit 3 media read-only, bit 4 volatile memory backup failed. Newer specs add a persistent memory region read-only flag. These bits show the current state and are not persistent. Percentage Used is the vendor's endurance estimate. It can go above 100, and 100 does not by itself mean failure. Media and Data Integrity Errors counts unrecovered integrity errors such as uncorrectable ECC. Error Information Log Entries often climbs because of harmless rejected commands, so don't panic over it alone.

Attribute-to-action table

SignalWhat it meansAction
ATA 197 Current_Pending_Sector > 0Sectors that could not be read and are waiting for reallocationRun a long test, check backups, plan replacement if it grows
ATA 198 Offline_Uncorrectable > 0The offline scan found unreadable sectorsSame as 197. Data in those sectors may already be lost
ATA 5 Reallocated_Sector_Ct risingThe drive is using spare sectorsWatch the trend. Steady growth means replace it
Self-test log shows a failure with an LBAThe drive's own test hit an unreadable blockReplace the drive. Restore the affected files from backup
NVMe Critical Warning bit 0, 2 or 3Spare exhausted, reliability degraded, or read-onlyReplace now. Bit 3 means writes are already refused
NVMe Media and Data Integrity Errors > 0Unrecovered integrity errorsRun a scrub and a backup check, then replace if it grows
NVMe Percentage Used near 100Rated endurance is used upBudget a replacement. This alone is not a failure
-H FAILEDThe drive has failed or predicts failure within 24 hoursCopy the data off immediately

Writing a smartd.conf for SATA and NVMe

First see what Debian enabled by default:

grep -Ev '^[[:space:]]*(#|$)' /etc/smartd.conf
systemctl cat smartmontools

The stock config is a single DEVICESCAN line. Once smartd reaches that line it ignores everything after it, so either put your explicit device lines above it or replace it. Use stable /dev/disk/by-id/ paths so a renumbered sdX doesn't make you monitor the wrong disk. This config targets smartmontools 7.5:

# SATA HDD: all checks, short test daily 02:00, long test Saturday 03:00
/dev/disk/by-id/ata-EXAMPLE_HDD_SERIAL1 -d sat -a -n standby,24,q -s (S/../.././02|L/../../6/03) -W 4,45,55 -m root -M exec /usr/local/sbin/smartd-alert

# NVMe (7.5): health minus temperature bit, logs, usage/spare, self-tests
/dev/nvme0 -d nvme -H 0xfd -l error -l selftest -l selfteststs -t -s (S/../.././03|L/../../7/04) -W 5,65,75 -m root -M exec /usr/local/sbin/smartd-alert

What the directives do:

  • -a is shorthand for -H -f -t -l error -l selftest -l selfteststs. On ATA it also turns on -C 197 -U 198, which report non-zero pending and offline-uncorrectable counts.
  • -s takes a regex matched against T/MM/DD/d/HH, where weekday 1 is Monday. S means short and L means long. On NVMe these map to the short and extended tests, but only from 7.5 onwards. Always use two-digit fields. smartd does not run scheduled tests during its first poll after it starts.
  • -H 0xfd masks out bit 1 (temperature) of the NVMe Critical Warning byte, so that a brief temperature spike during a backup doesn't page you. -W covers temperature instead. -H 0xff is the same as a plain -H.
  • -t means -p plus -u. Since 7.5 these watch NVMe Available Spare, Percentage Used and Media Errors.
  • -n standby,24,q (ATA only) skips checks while the disk is spun down, forces a check after 24 skips, and keeps the skip messages out of the log.
  • -W 4,45,55 logs changes of 4 °C, logs at info level from 45 °C, and treats 55 °C as critical, which also sends mail.

Before you restart anything, check the self-test schedule:

smartd -q showtests

Then reload. HUP rereads the config, and a config with a syntax error is ignored while the daemon keeps running with the old one, so check the journal afterwards:

systemctl reload smartmontools
journalctl -u smartmontools -b --no-pager | tail -n 30

Still on 7.4? Schedule NVMe tests with a timer

smartd 7.4 can't schedule NVMe self-tests. smartctl 7.4 can start them, though, so a timer covers the gap:

[Unit]
Description=NVMe short self-test

[Service]
Type=oneshot
ExecStart=/usr/sbin/smartctl -t short /dev/nvme0
[Unit]
Description=Daily NVMe short self-test

[Timer]
OnCalendar=*-*-* 03:00:00
Persistent=true

[Install]
WantedBy=timers.target

Enable it with systemctl enable --now nvme-selftest.timer. There are two caveats on 7.4. First, the upstream 7.5 NEWS lists a fix for smartctl -t self-tests on single-namespace NVMe devices, and most consumer drives have one namespace. Second, NVMe self-test support for smartd's -l selftest only arrived in 7.5, so smartd 7.4 won't alert you when one of these tests fails. After the first run, confirm that the test actually ran and passed, and keep checking the log yourself:

smartctl -l selftest /dev/nvme0

Keeping attribute history

The state and attribute logs are what turn single readings into trends. 7.5 adds NVMe support to the -A attribute log. You usually don't need to set -A on Debian. The package is built with attribute logging on, and Debian's smartd(8) says that without -A, attribute logs still go to /var/lib/smartmontools/ as attrlog.MODEL-SERIAL.ata.csv files. Check that directory for attrlog.*.csv files after smartd has run for a while.

Alerting that actually arrives

-m root makes smartd send mail with the system mail command. If mail isn't installed, or no MTA can deliver the message, you get a journal line and nothing else. Debian's stock line routes alerts through /usr/share/smartmontools/smartd-runner, which run-parts every script in /etc/smartmontools/run.d/. Read 10mail there before you rely on it. Then make sure local mail works at all:

apt install bsd-mailx
echo "test from $(hostname)" | mail -s "mail path test" root
journalctl -t postfix/local -t postfix/smtp -t postfix/qmgr -n 20 --no-pager

If Postfix refuses or bounces that message, fix it first. The Postfix relay access denied walkthrough covers the usual causes. For a push channel that doesn't need mail, use a small -M exec script. smartd passes the alert details in environment variables, and it blocks until the script exits, so put a timeout on any network call:

#!/bin/sh
# Called by smartd -M exec; details arrive in SMARTD_* variables.
subject="SMART ${SMARTD_FAILTYPE} on $(hostname): ${SMARTD_DEVICE}"
logger -t smartd-alert -p daemon.crit "${subject}: ${SMARTD_MESSAGE}"
if [ -n "${SMARTD_ADDRESS}" ]; then
    printf '%s\n' "${SMARTD_FULLMESSAGE}" | mail -s "${subject}" "${SMARTD_ADDRESS}"
fi
curl -fsS --max-time 10 -H "Title: ${subject}" -d "${SMARTD_MESSAGE}" https://ntfy.example.net/disk-alerts || true
chmod 0755 /usr/local/sbin/smartd-alert

Now prove the whole chain. Temporarily add -M test to each device line, so that smartd sends one test alert per device at startup:

/dev/nvme0 -d nvme -H 0xfd -l error -l selftest -l selfteststs -t -m root -M test -M exec /usr/local/sbin/smartd-alert
systemctl restart smartmontools
journalctl -u smartmontools -b --no-pager | grep -i test
journalctl -t smartd-alert -n 5 --no-pager

Your config is only done once the message shows up in your inbox or on your phone. Then remove -M test and reload. If state persistence is on, repeat alerts default to daily. Without it they default to once. -M diminishing backs off to 1, 2, 4… days between reminders.

Pitfalls, limitations and alternatives

  • RAID controllers and USB bridges. Behind a MegaRAID controller, /dev/sda is the logical volume. Address each member with -d megaraid,N. USB-SATA enclosures usually need -d sat. USB-NVMe enclosures need -d sntjmicron, -d sntasmedia or -d sntrealtek, depending on the bridge chip. Run smartctl --scan-open and check that smartctl -x returns real attributes before you trust any line.
  • Removable disks. Without -d removable, a missing device at startup is an error. With it, smartd carries on and stays quiet when the disk is unplugged.
  • Standby. -n works only on ATA. Some drives wake up anyway, and autodetection can itself spin a disk up, so set an explicit -d.
  • VMs. The upstream systemd unit carries ConditionVirtualization=no, so in a guest the service is skipped without an error. Monitor the disks on the host.
  • SMART says PASSED until it doesn't. In Backblaze's 2016 analysis, 23.3% of failed drives showed no warning in any of the five attributes they track. SMART is an early-warning system, not a guarantee.
  • Drive database. If a newer drive has unnamed attributes, update-smart-drivedb fetches a GPG-verified database. Debian's README advises doing this only when you actually need it.

Alternatives and complements:

  • nvme-cli (2.13-2 in trixie): nvme smart-log /dev/nvme0 reads the same health page and gives you vendor plugins and firmware tools. It doesn't do monitoring.
  • Scrutiny: a web dashboard with history and notifications (ntfy, Gotify, email and others). It runs smartctl under the hood and usually ships as a container. Use it alongside smartd's alerts, not instead of them.
  • btrfs and ZFS scrubs: btrfs scrub start and zpool scrub read every block and check it against its checksum. On RAID1, mirror or raidz they repair from a good copy. They catch silent corruption that SMART never reports. Debian's ZFS man page shows timers such as zfs-scrub-weekly@rpool.timer.

Combine them. smartd warns you, a monthly scrub confirms the data is intact, and a hardened restic backup with a regular check is what you restore from when the drive dies anyway. A dying disk can also cause odd write failures. If you see those, rule out the filesystem-level causes in No space left on device with free space in df before you blame the hardware.

Takeaways and cheat sheet

TaskCommand or directive
List devices and typessmartctl --scan-open
Full ATA reportsmartctl -x /dev/sda
NVMe reportsmartctl -a /dev/nvme0
Start a long testsmartctl -t long /dev/sda
Read self-test resultssmartctl -l selftest /dev/sda
Preview the smartd schedulesmartd -q showtests
Disk behind USB-SATA-d sat
Disk behind MegaRAID-d megaraid,N
Ignore NVMe temperature bit-H 0xfd
Do not wake sleeping HDDs-n standby,24,q
Test alert delivery-M test
Custom notifier-M exec /usr/local/sbin/smartd-alert
  • Use smartmontools 7.5 from trixie-backports if you have NVMe drives. 7.4 can't schedule NVMe self-tests from smartd.
  • Use explicit by-id device lines with -s schedules, and keep them above or instead of DEVICESCAN.
  • Alert on the raw values of 197, 198 and 5, on failed self-tests, and on NVMe spare, read-only and media errors. Don't wait for FAILED.
  • Prove delivery with -M test. An alert that only reaches the journal doesn't count.
  • Run scrubs and tested backups alongside smartd. SMART misses a real share of failures.

Sources

Comments

Latest alerts