Disks rarely die without warning. Usually there are signs first: pending sectors on a SATA drive, a rising media error count on an NVMe drive, or a self-test that fails at one LBA. The problem is that nobody is watching. smartmontools gives you two tools for this. smartctl reads the drive's own health data when you ask for it. smartd polls that data every 30 minutes, runs scheduled self-tests and alerts you when something changes. This post sets up smartd on Debian 13 for a mixed SATA and NVMe box, explains which numbers matter, and shows how to prove that alerts actually reach you. That last part is the one most setups skip.
What smartd and smartctl are for
Use smartctl for ad-hoc inspection: before trusting a used drive, after an I/O error in the kernel log, or when a RAID member gets kicked out. Use smartd for continuous monitoring on any machine with physical disks. Neither one replaces backups or filesystem checksums. SMART tells you what the drive firmware is willing to admit, and that is less than the full picture (see the pitfalls below).
By default smartd checks devices every 1800 seconds (-i changes this). It logs to the syslog daemon facility, which on Debian 13 means the journal. It can save state under /var/lib/smartmontools/, so it remembers attribute values and when it last sent an alert across restarts.
Installing smartmontools on Debian 13
Trixie ships smartmontools 7.4-3. Upstream released 7.5 on 2025-04-30, and Debian has 7.5-2~bpo13+1 in trixie-backports (accepted 2025-10-26). Testing and unstable carry 7.5-2. For SATA-only machines, 7.4 is fine. For NVMe, 7.5 is worth the backport. Its smartd adds NVMe self-test scheduling, an NVMe attribute log, a bitmask for -H, and checks for Available Spare, Percentage Used and Media Errors.
apt update
apt install smartmontools
smartctl --version | head -n 1
apt policy smartmontools
To get 7.5, enable backports and install only this package from there:
Types: deb deb-src
URIs: http://deb.debian.org/debian
Suites: trixie-backports
Components: main
Enabled: yes
Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg
apt update
apt install smartmontools/trixie-backports
systemctl status smartmontools
The package installs smartmontools.service, /etc/smartd.conf, /etc/default/smartmontools and a hook directory /etc/smartmontools/run.d/. It does not depend on a mail command or an MTA. mailx is only a Suggests (met by bsd-mailx or mailutils). That matters later.
Reading smartctl output: the numbers that matter
Start by asking smartctl what it can see:
smartctl --scan-open
smartctl -x /dev/sda
smartctl -a /dev/nvme0
For ATA drives, use -x. The man page no longer recommends -a for ATA, because -a skips the 48-bit logs: extended error and self-test logs, device statistics and SCT temperature history. For NVMe, -a and -x are the same thing. A trimmed example of the SATA attributes that matter:
SMART overall-health self-assessment test result: PASSED
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0
197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 8
198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 8
Note the PASSED next to eight pending sectors. The overall verdict only flips when a normalized value drops to its threshold, and by then the drive is often already gone. Look at the raw values of 5, 197 and 198 instead. Backblaze's fleet analysis also tracks 187 (Reported Uncorrectable) and 188 (Command Timeout).
For NVMe, the health log is the whole story:
SMART overall-health self-assessment test result: PASSED
Critical Warning: 0x00
Temperature: 41 Celsius
Available Spare: 100%
Available Spare Threshold: 10%
Percentage Used: 7%
Media and Data Integrity Errors: 0
Error Information Log Entries: 12
Critical Warning is a bitfield. Bit 0 means spare below threshold, bit 1 a temperature threshold, bit 2 reliability degraded, bit 3 media read-only, bit 4 volatile memory backup failed. Newer specs add a persistent memory region read-only flag. These bits show the current state and are not persistent. Percentage Used is the vendor's endurance estimate. It can go above 100, and 100 does not by itself mean failure. Media and Data Integrity Errors counts unrecovered integrity errors such as uncorrectable ECC. Error Information Log Entries often climbs because of harmless rejected commands, so don't panic over it alone.
Attribute-to-action table
| Signal | What it means | Action |
|---|---|---|
| ATA 197 Current_Pending_Sector > 0 | Sectors that could not be read and are waiting for reallocation | Run a long test, check backups, plan replacement if it grows |
| ATA 198 Offline_Uncorrectable > 0 | The offline scan found unreadable sectors | Same as 197. Data in those sectors may already be lost |
| ATA 5 Reallocated_Sector_Ct rising | The drive is using spare sectors | Watch the trend. Steady growth means replace it |
| Self-test log shows a failure with an LBA | The drive's own test hit an unreadable block | Replace the drive. Restore the affected files from backup |
| NVMe Critical Warning bit 0, 2 or 3 | Spare exhausted, reliability degraded, or read-only | Replace now. Bit 3 means writes are already refused |
| NVMe Media and Data Integrity Errors > 0 | Unrecovered integrity errors | Run a scrub and a backup check, then replace if it grows |
| NVMe Percentage Used near 100 | Rated endurance is used up | Budget a replacement. This alone is not a failure |
-H FAILED | The drive has failed or predicts failure within 24 hours | Copy the data off immediately |
Writing a smartd.conf for SATA and NVMe
First see what Debian enabled by default:
grep -Ev '^[[:space:]]*(#|$)' /etc/smartd.conf
systemctl cat smartmontools
The stock config is a single DEVICESCAN line. Once smartd reaches that line it ignores everything after it, so either put your explicit device lines above it or replace it. Use stable /dev/disk/by-id/ paths so a renumbered sdX doesn't make you monitor the wrong disk. This config targets smartmontools 7.5:
# SATA HDD: all checks, short test daily 02:00, long test Saturday 03:00
/dev/disk/by-id/ata-EXAMPLE_HDD_SERIAL1 -d sat -a -n standby,24,q -s (S/../.././02|L/../../6/03) -W 4,45,55 -m root -M exec /usr/local/sbin/smartd-alert
# NVMe (7.5): health minus temperature bit, logs, usage/spare, self-tests
/dev/nvme0 -d nvme -H 0xfd -l error -l selftest -l selfteststs -t -s (S/../.././03|L/../../7/04) -W 5,65,75 -m root -M exec /usr/local/sbin/smartd-alert
What the directives do:
-ais shorthand for-H -f -t -l error -l selftest -l selfteststs. On ATA it also turns on-C 197 -U 198, which report non-zero pending and offline-uncorrectable counts.-stakes a regex matched againstT/MM/DD/d/HH, where weekday 1 is Monday.Smeans short andLmeans long. On NVMe these map to the short and extended tests, but only from 7.5 onwards. Always use two-digit fields. smartd does not run scheduled tests during its first poll after it starts.-H 0xfdmasks out bit 1 (temperature) of the NVMe Critical Warning byte, so that a brief temperature spike during a backup doesn't page you.-Wcovers temperature instead.-H 0xffis the same as a plain-H.-tmeans-pplus-u. Since 7.5 these watch NVMe Available Spare, Percentage Used and Media Errors.-n standby,24,q(ATA only) skips checks while the disk is spun down, forces a check after 24 skips, and keeps the skip messages out of the log.-W 4,45,55logs changes of 4 °C, logs at info level from 45 °C, and treats 55 °C as critical, which also sends mail.
Before you restart anything, check the self-test schedule:
smartd -q showtests
Then reload. HUP rereads the config, and a config with a syntax error is ignored while the daemon keeps running with the old one, so check the journal afterwards:
systemctl reload smartmontools
journalctl -u smartmontools -b --no-pager | tail -n 30
Still on 7.4? Schedule NVMe tests with a timer
smartd 7.4 can't schedule NVMe self-tests. smartctl 7.4 can start them, though, so a timer covers the gap:
[Unit]
Description=NVMe short self-test
[Service]
Type=oneshot
ExecStart=/usr/sbin/smartctl -t short /dev/nvme0
[Unit]
Description=Daily NVMe short self-test
[Timer]
OnCalendar=*-*-* 03:00:00
Persistent=true
[Install]
WantedBy=timers.target
Enable it with systemctl enable --now nvme-selftest.timer. There are two caveats on 7.4. First, the upstream 7.5 NEWS lists a fix for smartctl -t self-tests on single-namespace NVMe devices, and most consumer drives have one namespace. Second, NVMe self-test support for smartd's -l selftest only arrived in 7.5, so smartd 7.4 won't alert you when one of these tests fails. After the first run, confirm that the test actually ran and passed, and keep checking the log yourself:
smartctl -l selftest /dev/nvme0
Keeping attribute history
The state and attribute logs are what turn single readings into trends. 7.5 adds NVMe support to the -A attribute log. You usually don't need to set -A on Debian. The package is built with attribute logging on, and Debian's smartd(8) says that without -A, attribute logs still go to /var/lib/smartmontools/ as attrlog.MODEL-SERIAL.ata.csv files. Check that directory for attrlog.*.csv files after smartd has run for a while.
Alerting that actually arrives
-m root makes smartd send mail with the system mail command. If mail isn't installed, or no MTA can deliver the message, you get a journal line and nothing else. Debian's stock line routes alerts through /usr/share/smartmontools/smartd-runner, which run-parts every script in /etc/smartmontools/run.d/. Read 10mail there before you rely on it. Then make sure local mail works at all:
apt install bsd-mailx
echo "test from $(hostname)" | mail -s "mail path test" root
journalctl -t postfix/local -t postfix/smtp -t postfix/qmgr -n 20 --no-pager
If Postfix refuses or bounces that message, fix it first. The Postfix relay access denied walkthrough covers the usual causes. For a push channel that doesn't need mail, use a small -M exec script. smartd passes the alert details in environment variables, and it blocks until the script exits, so put a timeout on any network call:
#!/bin/sh
# Called by smartd -M exec; details arrive in SMARTD_* variables.
subject="SMART ${SMARTD_FAILTYPE} on $(hostname): ${SMARTD_DEVICE}"
logger -t smartd-alert -p daemon.crit "${subject}: ${SMARTD_MESSAGE}"
if [ -n "${SMARTD_ADDRESS}" ]; then
printf '%s\n' "${SMARTD_FULLMESSAGE}" | mail -s "${subject}" "${SMARTD_ADDRESS}"
fi
curl -fsS --max-time 10 -H "Title: ${subject}" -d "${SMARTD_MESSAGE}" https://ntfy.example.net/disk-alerts || true
chmod 0755 /usr/local/sbin/smartd-alert
Now prove the whole chain. Temporarily add -M test to each device line, so that smartd sends one test alert per device at startup:
/dev/nvme0 -d nvme -H 0xfd -l error -l selftest -l selfteststs -t -m root -M test -M exec /usr/local/sbin/smartd-alert
systemctl restart smartmontools
journalctl -u smartmontools -b --no-pager | grep -i test
journalctl -t smartd-alert -n 5 --no-pager
Your config is only done once the message shows up in your inbox or on your phone. Then remove -M test and reload. If state persistence is on, repeat alerts default to daily. Without it they default to once. -M diminishing backs off to 1, 2, 4… days between reminders.
Pitfalls, limitations and alternatives
- RAID controllers and USB bridges. Behind a MegaRAID controller,
/dev/sdais the logical volume. Address each member with-d megaraid,N. USB-SATA enclosures usually need-d sat. USB-NVMe enclosures need-d sntjmicron,-d sntasmediaor-d sntrealtek, depending on the bridge chip. Runsmartctl --scan-openand check thatsmartctl -xreturns real attributes before you trust any line. - Removable disks. Without
-d removable, a missing device at startup is an error. With it, smartd carries on and stays quiet when the disk is unplugged. - Standby.
-nworks only on ATA. Some drives wake up anyway, and autodetection can itself spin a disk up, so set an explicit-d. - VMs. The upstream systemd unit carries
ConditionVirtualization=no, so in a guest the service is skipped without an error. Monitor the disks on the host. - SMART says PASSED until it doesn't. In Backblaze's 2016 analysis, 23.3% of failed drives showed no warning in any of the five attributes they track. SMART is an early-warning system, not a guarantee.
- Drive database. If a newer drive has unnamed attributes,
update-smart-drivedbfetches a GPG-verified database. Debian's README advises doing this only when you actually need it.
Alternatives and complements:
- nvme-cli (2.13-2 in trixie):
nvme smart-log /dev/nvme0reads the same health page and gives you vendor plugins and firmware tools. It doesn't do monitoring. - Scrutiny: a web dashboard with history and notifications (ntfy, Gotify, email and others). It runs
smartctlunder the hood and usually ships as a container. Use it alongside smartd's alerts, not instead of them. - btrfs and ZFS scrubs:
btrfs scrub startandzpool scrubread every block and check it against its checksum. On RAID1, mirror or raidz they repair from a good copy. They catch silent corruption that SMART never reports. Debian's ZFS man page shows timers such aszfs-scrub-weekly@rpool.timer.
Combine them. smartd warns you, a monthly scrub confirms the data is intact, and a hardened restic backup with a regular check is what you restore from when the drive dies anyway. A dying disk can also cause odd write failures. If you see those, rule out the filesystem-level causes in No space left on device with free space in df before you blame the hardware.
Takeaways and cheat sheet
| Task | Command or directive |
|---|---|
| List devices and types | smartctl --scan-open |
| Full ATA report | smartctl -x /dev/sda |
| NVMe report | smartctl -a /dev/nvme0 |
| Start a long test | smartctl -t long /dev/sda |
| Read self-test results | smartctl -l selftest /dev/sda |
| Preview the smartd schedule | smartd -q showtests |
| Disk behind USB-SATA | -d sat |
| Disk behind MegaRAID | -d megaraid,N |
| Ignore NVMe temperature bit | -H 0xfd |
| Do not wake sleeping HDDs | -n standby,24,q |
| Test alert delivery | -M test |
| Custom notifier | -M exec /usr/local/sbin/smartd-alert |
- Use smartmontools 7.5 from trixie-backports if you have NVMe drives. 7.4 can't schedule NVMe self-tests from smartd.
- Use explicit by-id device lines with
-sschedules, and keep them above or instead ofDEVICESCAN. - Alert on the raw values of 197, 198 and 5, on failed self-tests, and on NVMe spare, read-only and media errors. Don't wait for
FAILED. - Prove delivery with
-M test. An alert that only reaches the journal doesn't count. - Run scrubs and tested backups alongside smartd. SMART misses a real share of failures.
Sources
- Debian packages: smartmontools in trixie
- Debian tracker: smartmontools
- smartmontools NEWS (upstream)
- smartd.conf(5), smartmontools 7.5 (Debian unstable)
- smartd.conf(5), smartmontools 7.4 (Debian trixie)
- smartd(8) (Debian trixie)
- smartctl(8) (Debian trixie)
- smartmontools file list (trixie, amd64)
- Debian smartmontools README.Debian
- Debian smartmontools defaults file
- Upstream smartd.service.in
- Debian Backports instructions
- Microsoft Learn: NVME_HEALTH_INFO_LOG structure
- libnvme: nvme_smart_crit(2)
- libnvme: nvme_smart_log(2)
- Backblaze: What SMART Stats Tell Us About Hard Drives
- Debian packages: nvme-cli in trixie
- Scrutiny on GitHub
- btrfs-scrub(8) (Debian trixie)
- zpool-scrub(8) (Debian trixie)
- Postfix logging (MAILLOG_README)
Comments