What Is /var For on Embedded Linux?

Short answer: /var is where a running system keeps data that changes: logs, caches, service state, queued work, and crash dumps. /usr holds what ships in the image and never changes at runtime; /var holds what the system writes while it runs. On a board, that split matters because it lets you keep the root filesystem read-only and funnel every write into one place where you can reason about flash wear, power loss, and updates.

The catch is that /var is not one thing. /var/log and /var/cache can usually live in RAM, but /var/lib almost always holds state that has to survive a reboot. This post walks through the standard FHS and systemd paths and gives you a way to decide, per subdirectory, what goes on tmpfs and what goes on persistent storage.

References: Filesystem Hierarchy Standard 3.0, file-hierarchy(7), journald.conf(5), systemd.exec(5).

How does /var differ from /usr and /tmp?

In one line: FHS sorts files along two axes, shareable vs. unshareable and static vs. variable. /usr is shareable and static; /var is variable. /tmp is scratch space that may be wiped at boot.

FHS 3.0 defines /var as the home for “variable data files”: spool directories, administrative and logging data, and transient files. It is also explicit about why /var exists: so that /usr can be mounted read-only. Everything that gets written during normal operation is moved out of /usr and into /var, which is exactly the property an embedded image with a read-only root depends on.

PathNatureAfter rebootWhat it means on a board
/usrStatic, shareableKept (part of the image)The bulk of your firmware. Read-only by default
/etcHost-specific configKeptPer-device config. A read-only root may need an overlay or a separate partition for it
/varVariable dataDepends on the subdirectoryWhere runtime writes land. The part you actually have to design
/tmpTemporary filesMay be wipedUsually tmpfs
/var/tmpTemporary filesExpected to be keptFHS says it survives reboots. On embedded, pick a policy and write it down
/runRuntime stateAlways wipedtmpfs: PID files, sockets, locks

The one that trips people up is /var/tmp. FHS says files there are preserved across reboots. Mounting it as tmpfs is a common embedded choice, but it quietly breaks that contract. That is fine as long as it is documented in your image spec, so you can catch applications that park something important there.

What goes in each subdirectory under /var?

In one line: The name tells you the lifetime. log and cache can be lost and rebuilt, lib must not be lost, and run and lock were never meant to persist.

SubdirectoryMeaning per FHS/systemdIf it is lostTypical placement on a board
/var/logLog filesYou lose diagnostics; the system keeps runningtmpfs, or a size-capped persistent partition
/var/cacheRegenerable cacheSlower, but no data loss (an FHS requirement)tmpfs is fine; exclude it if it gets large
/var/libApplication and service stateConfig, pairing, databases, and package DBs may vanishPersistent storage
/var/spoolWork queued for later processingUnsent work disappearsPersistent if the queue matters
/var/runRuntime dataVolatile by designSymlink to /run
/var/lockLock filesVolatile by designSymlink to /run/lock
/var/tmpTemp files kept across rebootsDepends on your policyPersistent or tmpfs (document it)
/var/crashSystem crash dumps (optional in FHS)Post-mortem data is gonePersistent, with a hard size cap

A few of these deserve more than a table row:

  • /var/cache: FHS requires that an application can regenerate this data without loss if it is deleted. That is what makes it safe to put on tmpfs or to wipe when space runs low. The flip side: anything you cannot afford to lose does not belong here.
  • /var/lib: Package manager databases (/var/lib/dpkg, /var/lib/opkg), systemd state (/var/lib/systemd), network connection profiles, and local agent databases all live here. A large share of “settings reset on every reboot” bugs on embedded devices come down to /var/lib sitting on tmpfs.
  • /var/run and /var/lock: FHS 3.0 introduced /run and moved the role of /var/run there, allowing /var/run to be a compatibility symlink. systemd-based distributions and default Yocto images do exactly that, which is why old daemons that still write PID files to /var/run keep working.
  • /var/spool: Mail, print jobs, cron/at jobs, anything that is “handle later.” On devices it often ends up holding telemetry that could not be sent while offline. Whether that queue must survive a power cut decides where it lives.

Why does /var need special care on embedded systems?

In one line: On a desktop /var is just a directory on a big disk. On a board, limited flash endurance, small partitions, sudden power loss, and a read-only root all converge on it.

Flash wear and write amplification

NAND and eMMC blocks tolerate a limited number of erase cycles. eMMC controllers do wear leveling, but a steady stream of small writes still gets amplified, so the flash sees more writes than the application issued. Line-by-line logging with frequent fsync, and state files rewritten every few seconds, are the usual culprits, and almost all of that traffic hits /var/log and /var/lib.

Tiny root filesystems

Root partitions are often sized to fit the image with little headroom. If /var shares that partition, a few megabytes of logs can fill it. And when root fills up, it is not just logging that stops; every service that saves state under /var/lib starts failing too.

Read-only root with a writable area

A read-only root cuts the risk of filesystem corruption on power loss and makes A/B updates and image signature checks simpler. The price is that you have to open up writable locations explicitly. The three common patterns:

  1. Mount parts of /var as tmpfs — keep volatile data (/var/log, /var/cache, /var/tmp) in RAM.
  2. Mount /var or /var/lib from a separate rw partition — put persistent state on a dedicated data partition.
  3. Overlay the root with overlayfs — a read-only lower layer with a tmpfs or persistent upper layer. The trade-offs are covered in overlayroot vs A/B updates.

Yocto enables VOLATILE_LOG_DIR by default, which points /var/log at /var/volatile/log on tmpfs, and the read-only-rootfs image feature makes root read-only. Buildroot lets you choose whether root is remounted read-write at boot. Whatever your build system, check at least once that its defaults match what your product actually needs.

Where volatile ends and persistent begins

The real design work is answering one question per subdirectory: is it acceptable for this to disappear when power is cut? Skip that question and put all of /var on tmpfs, and you lose state. Put all of it on flash, and you get wear and capacity problems.

Where should logs, the journal, and crash dumps go?

In one line: Keep logs in RAM by default and persist only what you need. Control journald with Storage= and size caps, text logs with logrotate, and crash dumps with count and size limits.

journald

journald picks its location from Storage=. With volatile it writes only to /run/log/journal (RAM); with persistent it writes to /var/log/journal, creating it if needed. The default, auto, persists only if /var/log/journal already exists, so whether your image ships that directory silently decides the behavior.

If you do persist the journal on small flash, always cap it:

# /etc/systemd/journald.conf.d/10-embedded.conf
[Journal]
Storage=persistent
SystemMaxUse=16M
SystemMaxFileSize=4M
RuntimeMaxUse=8M
MaxRetentionSec=7day

SystemMaxUse= limits the persistent journal in /var/log/journal; RuntimeMaxUse= limits the volatile one in /run/log/journal. To shrink a journal that has already grown, see journald vacuum. A common alternative to persisting everything is to run volatile in production and switch to persistent only in field-debug images or under specific conditions.

Text logs and logrotate

For syslog output or applications writing their own /var/log/*.log, set up logrotate. On devices, size-based rotation (size, maxsize) with a fixed count (rotate) is more predictable than daily. It is also easy to forget that logrotate only runs if a cron job or systemd timer invokes it. For fstab examples and caveats when /var/log is on tmpfs, see putting /var/log on tmpfs.

Crash dumps

systemd-coredump stores core dumps in /var/lib/systemd/coredump by default, and coredump.conf lets you cap them with MaxUse=, ProcessSizeMax=, and friends. A single core from a large process can run to tens or hundreds of megabytes, so on a small data partition it is safer to disable storage (Storage=none) or set very low limits. Kernel panic records are better captured through pstore (/sys/fs/pstore) and then collected into /var/lib or /var/log after the next boot.

Where should services and agents keep their state?

In one line: Anything a service must remember across reboots goes in /var/lib/<service>, anything it can rebuild goes in /var/cache/<service>, and anything needed only while running goes in /run/<service>.

systemd encodes this split as unit options. StateDirectory= creates a directory under /var/lib/, CacheDirectory= under /var/cache/, LogsDirectory= under /var/log/, and RuntimeDirectory= under /run/, with ownership set for the service user.

[Service]
ExecStart=/usr/bin/fleet-agent
User=fleet
StateDirectory=fleet-agent
CacheDirectory=fleet-agent
RuntimeDirectory=fleet-agent
ProtectSystem=strict

With ProtectSystem=strict, most of the filesystem is read-only to the service, except the directories declared above. On a board with a read-only root this pattern pays off twice: the unit file itself documents where the service writes, which makes it much easier to decide which paths need persistent storage.

Be especially careful with software whose state is large and important: OTA clients, device management agents, and container runtimes (/var/lib/docker, /var/lib/containerd). On tmpfs, they re-enroll or re-download images on every boot. On a small shared root partition, they run out of space.

Package managers follow the same rule. If you run opkg or dpkg in the field, the package database (/var/lib/opkg, /var/lib/dpkg) must be persistent, while downloaded package caches under /var/cache can be volatile. If you only ever update whole images, keeping the package database inside the read-only image is the more consistent choice.

Practical checklist: what goes on tmpfs, what must persist?

In one line: For each path, ask whether it can be lost, how big it can get, and how often it is written. Then pin the answers down in fstab, journald.conf, and unit files.

PathDefault recommendationWhen it must persistWhat to check
/var/logtmpfsProducts that require field post-mortemsSize cap, rotation, write rate if persisted
/var/log/journalAbsent (= volatile)Need to trace logs across bootsStorage=, SystemMaxUse=
/var/cachetmpfsCaches that are very expensive to rebuildSize; no secrets mixed in
/var/libPersistent partitionAlmost alwaysPower-loss resilience, filesystem choice
/var/spoolDepends on requirementsOffline queues that must not be lostCap and eviction policy for old entries
/var/tmpDecide and documentApps that assume it survives rebootsWhich applications actually use it
/var/run, /var/lockSymlink to /runNeverThe symlinks actually exist
Core dumpsOff or tightly cappedDebug buildsMaxUse= in coredump.conf

A verification pass might look like this:

1. Check how each part of /var is mounted: findmnt -R /var
2. Find the paths that actually take heavy writes: watch /proc/diskstats or iotop over time
3. List the files that must survive a reboot, reboot, and confirm they did
4. Check journald's Storage= value and whether /var/log/journal exists
5. Confirm logrotate really runs on schedule (timer or cron)
6. Fill the data partition on purpose and confirm boot and key services still work
7. Pull power mid-operation and confirm /var/lib state is consistent afterwards

Steps 6 and 7 are the ones teams skip, and the ones that fail most often in the field. A device that will not boot because its data partition is full may be impossible to recover remotely.

What mistakes come up most often?

In one line: Most of them come from carrying over the desktop assumption that /var is always writable, always roomy, and always safe.

  1. Writing verbose logs straight to flash. Debug logging left on during development ships in the production image and comes back months later as worn flash or a full partition. Make log level and log destination a build-time decision.
  2. Putting secrets in /var/cache. Tokens, private keys, and device credentials sometimes land in /var/cache because they are “cached” from a server. By FHS definition, cache can be deleted at any time, and it is also the kind of path that backup, debug-dump, and support-bundle scripts happily sweep up. Keep credentials under a tightly permissioned /var/lib/<service> path, or better, in a hardware-backed store such as a TPM, secure element, or TEE.
  3. Assuming /var is always writable. With a read-only root where only parts of /var are writable, an application that tries to write to /var/opt or some ad-hoc /var/<name> will simply fail. Declare write paths with StateDirectory= and friends, or list the writable paths explicitly in your image design doc.
  4. Moving /var/lib to tmpfs along with everything else. The classic version: “We put all of /var on tmpfs to stop log writes, and now Wi-Fi settings and the device ID reset on every boot.” Apply tmpfs per subdirectory.
  5. Leaving tmpfs unsized. Without size=, tmpfs can grow to half of RAM by default. On a small-memory board, a log storm turns into an out-of-memory event.
  6. Keeping /var/run as a real directory. If /run and /var/run diverge, a PID file or socket written to one cannot be found through the other.

FAQ

QuestionShort answer
Can I put all of /var on tmpfs?Not recommended. State that must survive a reboot, such as /var/lib, will be lost. Split by subdirectory.
What is the difference between /tmp and /var/tmp?Per FHS, /tmp may be wiped at boot, while /var/tmp is expected to survive reboots.
Are /var/run and /run the same place?On current standard setups, /var/run is a symlink to /run.
Why do my journald logs disappear after a reboot?Most likely Storage=auto with no /var/log/journal directory, or Storage=volatile set explicitly.
Is it safe to delete /var/cache?Per FHS, applications must be able to rebuild it, so there should be no data loss. If there is, the application is misusing the path.
What about /etc on a read-only root?That is a separate problem from /var. Either bake config into the image or move only the files that need to change onto an overlay or data partition.

Bottom line: /var exists to gather every runtime write in one place. On embedded systems, splitting it into volatile and persistent parts per subdirectory, and putting a size cap on each, is most of the design work.

Sources