Boot latency analysis - BoxLite Documentation

The dominant cost of a cold start is not the sandbox policy but the macOS kernel validating code signatures page by page on freshly copied binaries. Once the cache is warm, a start drops to tens of milliseconds. Numbers below come from one Apple Silicon reference machine; the proportions matter, not the absolutes.

Executive Summary

One sample, measured on a single Apple Silicon machine (APFS, alpine:latest). Read the proportions, not the absolute numbers — they will differ on your hardware.

handle.start() is about ~2.1s with the jailer on, and about ~0.7s with the jailer off.
This ~1.4s difference is almost entirely not caused by sandbox-exec (the sandbox policy) — policy compilation measured ~5–10ms across runs.
The root cause is the macOS kernel’s per-page code-signature validation of freshly copied dylibs: an unavoidable kernel-level cost when executing a binary from a new inode.
Code-signature validation accounts for roughly 70% of the total startup time (~1450ms).

Measure it yourself

The code below directly measures the per-stage timing of one “cold start” and separately prints the kernel-code-signature-related stages. Note: metrics() is on the native Box and is async; the Boxlite runtime is entered with a synchronous with (and Boxlite.default() is synchronous), but its methods are async; Box is an async context manager.

# Requires: pip install boxlite   (latest published version)
# Platform: Linux+KVM or macOS (Apple Hypervisor.framework)
import asyncio

from boxlite import Boxlite, BoxOptions

async def main() -> None:
    # The Boxlite runtime is a [synchronous] context manager —— do not await it
    with Boxlite.default() as runtime:
        # runtime.create(...) is async (returns an awaitable); await it to get the native Box handle
        box = await runtime.create(BoxOptions(image="alpine:latest"))
        try:
            # Box is an async context manager; start() triggers the actual microVM startup
            await box.start()

            # box.metrics() is async and returns a BoxMetrics with stage timings
            m = await box.metrics()

            print("total_create_duration_ms :", m.total_create_duration_ms)
            print("guest_boot_duration_ms   :", m.guest_boot_duration_ms)
            print("stage_image_prepare_ms   :", m.stage_image_prepare_ms)
            print("stage_guest_rootfs_ms    :", m.stage_guest_rootfs_ms)
            print("stage_box_spawn_ms       :", m.stage_box_spawn_ms)
            print("stage_container_init_ms  :", m.stage_container_init_ms)
        finally:
            # Remove the box at the runtime layer (note: runtime.remove(...), not box.remove())
            # runtime.remove(...) is also async —— must be awaited (even though Boxlite itself is a synchronous CM)
            # force=True stops then removes; under default config the box may already be reclaimed, so tolerate not-found on removal
            try:
                await runtime.remove(box.id, force=True)
            except RuntimeError:
                pass  # box already reclaimed (box not found); ignore during cleanup

if __name__ == "__main__":
    try:
        asyncio.run(main())
    except RuntimeError as exc:
        # Missing virtualization or image pull failure raises a standard RuntimeError (process stays alive, can retry)
        print(f"boot failed: {exc}")

On the first run (cold copy, empty kernel signature cache), total_create_duration_ms is large; running it again on the same machine drops the stage timings markedly because the kernel’s per-page validation cache hits by inode. That is exactly this page’s core finding.

Full Startup Timeline (jailer on)

Latency Breakdown

Stage	Time	Share	Root cause
Pipeline setup	244ms	12%	image_prepare (237ms) + rootfs (7ms)
spawn → main()	1000ms	48%	Code signature: shim + libkrun + libgvproxy
krun FFI	450ms	22%	Code signature: dlopen(libkrunfw 22MB)
VM boot + guest	286ms	14%	Kernel boot + guest agent + vsock notification
Container init	86ms	4%	gRPC to the guest for OCI initialization

Root Cause: macOS Kernel Per-Page Code-Signature Validation

The mechanism

The jailer copies the shim plus dylibs (~36MB total) into each box’s own directory, producing a new inode (a new inode even with APFS reflink / copy-on-write). On macOS, when dyld loads an executable image via mmap(), the kernel validates its ad-hoc code signature per page:

dyld maps each library’s __TEXT segment via mmap(MAP_PRIVATE).
On the first page fault of each page, the kernel:
- Reads the page contents (4KB on x86, 16KB on ARM)
- Computes the SHA-256 hash
- Compares it against the embedded CodeDirectory hash slot
The result is cached by inode — subsequent executions of the same inode are therefore faster.

For 20.7MB of pre-main dylibs, that is about 1,300 per-page validations at ~0.77ms each = ~1000ms.

Comparison by scenario

The following scenarios all use the same shim binary (5.4MB) plus dylibs (libkrun 4.4MB, libgvproxy 10.9MB, 20.7MB total), varying only copy freshness and whether the sandbox is applied:

Scenario	Startup (ms)	Notes
Original binary, no sandbox	13	dyld shared cache is already hot
Cold copy, no sandbox	850	New inode, full validation
Warm copy, no sandbox	10	Kernel cache already populated
Warm copy + sandbox	14	Sandbox adds only +4ms
Cold copy, `cat` pre-warm	830	File I/O cache ≠ mmap cache
Cold copy, `codesign --verify`	850	Userspace validation ≠ kernel cache
Cold copy, pre-warm by exec-ing once	34	Only dyld’s mmap can pre-warm the kernel cache

Key conclusions

sandbox-exec is not the bottleneck — policy compilation measured ~5–10ms regardless of complexity (5 rules and 200 rules showed the same latency).
cat pre-warm has no effect — file I/O populates the buffer cache, but dyld’s mmap() goes through the kernel code-signature pager, a different cache path.
codesign --verify has no effect — it validates in userspace using its own file reads; the kernel maintains a separate validation cache that can only be populated via the mmap pager.
Only actually executing the binary warms the cache — the kernel’s code-signature validation cache is populated only when dyld maps executable pages via mmap().

Jailer On vs Off

Metric	Jailer ON	Jailer OFF	Delta
handle.start()	2067ms	696ms	+1371ms
spawn → main gap	1000ms	10ms	+990ms
krun FFI (engine.create)	450ms	23ms	+427ms
sandbox-exec overhead	~5–10ms	N/A	~5–10ms

The measured 1371ms delta is dominated by two components, both code-signature validation: the spawn→main gap (990ms) and the krun FFI call (427ms). The sandbox policy itself is negligible by comparison. (The two components were timed in separate runs, so they do not sum exactly to the end-to-end delta.)

The jailer / sandbox is controlled by the security options: BoxOptions(advanced=AdvancedBoxOptions(security=SecurityOptions.maximum())). AdvancedBoxOptions is not exported at the top level, so use from boxlite.boxlite import AdvancedBoxOptions; the SecurityOptions presets are development() / standard() / maximum() (there is no .minimum()).

Pipeline Stage Metrics (from `box.metrics()`)

The field names below are the readable attributes on the Python BoxMetrics (all with the _ms suffix; the stage_* fields are mostly Optional and are None when unavailable):

Field	Jailer ON	Jailer OFF
`total_create_duration_ms`	2039ms	695ms
`stage_filesystem_setup_ms`	0ms	—
`stage_image_prepare_ms`	237ms	—
`stage_guest_rootfs_ms`	7ms	—
`stage_box_spawn_ms`	12ms	11ms
`stage_container_init_ms`	86ms	63ms
`guest_boot_duration_ms`	~1697ms	~621ms

BoxMetrics also includes runtime fields such as cpu_percent, memory_bytes, commands_executed_total, exec_errors_total, and network_* (not startup-related). For runtime-level metrics use runtime.metrics(), with the fields num_running_boxes / boxes_created_total / boxes_failed_total / total_commands_executed / total_exec_errors.

What you can do about it

The kernel’s code-signature cache is keyed by inode and is populated only when a binary is actually executed. For callers that means one thing: Start one box before your batch begins. The first box pays the validation cost; every box after it on the same machine reuses the warm cache. Approaches that do not help, so they are not worth trying:

Attempt	Why it does not help
Pre-reading the binary with `cat`	Fills the buffer cache, but dyld’s `mmap()` goes through a different kernel path
Running `codesign --verify`	Userspace validation; it does not populate the kernel’s cache
Simplifying the sandbox policy	Policy compilation measured ~5–10ms regardless of complexity
Narrowing the FD-cleanup range	4092 `close()` calls take only 0.9ms in total

Methodology: host and shim wall-clock timestamps (chrono::Utc::now()) were correlated through the timing_profile.rs integration test (jailer ON/OFF), plus standalone benchmarks isolating one variable at a time — sandbox-exec policy compilation, FD cleanup, cold vs warm binary startup, and the pre-warm strategies above.

Reproduction

The repository contains a Rust timing baseline test (src/boxlite/tests/timing_profile.rs, targeting alpine:latest) that re-runs the data on this page:

# Run from the root of the BoxLite source repository
# 1) Rebuild the instrumented shim
./scripts/build/build-shim.sh

# 2) Run the timing profile test (--nocapture is required to see per-stage eprintln output)
cargo test -p boxlite --test timing_profile -- --nocapture

# 3) Run only the jailer case to avoid parallel contention skewing the timing
cargo test -p boxlite --test timing_profile boot_timing_profile \
  -- --exact --nocapture

The repository CLAUDE.md recommends preferring make targets (which encapsulate the correct build/cross-compilation flags). The cargo test commands above are run directly only for a local one-off timing experiment.

Troubleshooting

Symptom / error	Cause	Resolution
`RuntimeError`, `start()` fails immediately	The current environment has no hardware virtualization (no KVM; non-macOS without a hypervisor)	Linux needs KVM with the user in the `kvm` group; WSL2 needs KVM enabled; macOS uses Hypervisor.framework (no `/dev/kvm` required, supports Apple Silicon). This is an environment constraint — the exception can be caught and the process does not exit.
The first box is slow, later boxes are markedly faster	The kernel code-signature validation cache is populated by inode: cold on the first start, hot on reuse	Expected behavior. When starting many boxes in a batch, warm the first one first, or adopt the “shared bin/ directory” approach above.
`await box.metrics()` raises `'coroutine' was never awaited` or similar	`Box.metrics()` is async and must be awaited	Use `await box.metrics()`; the `Boxlite` runtime is a synchronous CM and `Box` is an async CM — do not confuse the two await conventions.
`await box.info()` behaves unexpectedly / errors	`Box.info()` is a synchronous method and must not be awaited	Call `box.info()` directly; it returns `BoxInfo`, and the status is in `box.info().state.status`.
`AttributeError: ... memory_usage_bytes` / `cpu_time_ms` / `active_boxes`	An old/wrong field name was used	Use the real fields: `BoxMetrics.memory_bytes`, `cpu_percent`, `total_create_duration_ms`, `stage_*_ms`; for the runtime use `RuntimeMetrics.num_running_boxes`.
Some `stage_*_ms` is `None`	That stage was not timed or that path did not run (for example, some stages are not populated with the jailer off)	The `stage_*` fields are `Optional`; check for None before reading, and do not perform arithmetic directly.
`ModuleNotFoundError` / import failure (Node: `Cannot find module 'boxlite'`)	Wrong package name	The Python package is `boxlite`; the Node package is `@boxlite-ai/boxlite` (not `boxlite`, not `@boxlite/sdk`).
Image pull timeout / network hiccup → `RuntimeError`	A failed image pull raises a standard `RuntimeError` (not `BoxliteError`)	Catch with `try/except RuntimeError` and retry; the first image pull also raises `stage_image_prepare_ms` markedly, which is unrelated to boot latency analysis.

​Executive Summary

​Measure it yourself

​Full Startup Timeline (jailer on)

​Latency Breakdown

​Root Cause: macOS Kernel Per-Page Code-Signature Validation

​The mechanism

​Comparison by scenario

​Key conclusions

​Jailer On vs Off

​Pipeline Stage Metrics (from box.metrics())

​What you can do about it

​Reproduction

​Troubleshooting

​Related Reading