App database¶
ziv.sqlite3 in the data folder is one shared SQLite file for features that keep data between runs. utils/app_db.py is stdlib only, so the CLI and the web layer can both use it. It holds no lock and no module-level connection, and it never creates, alters or drops a feature's tables itself. This page is the one home of what is stored in it and why.
How a feature uses it¶
- Components. A feature declares
Component(name, version, migrate). Its tables are named<name>_*, and it never reads or writes another component's tables.connect(components=(...))brings each listed component to its version before it returns. - Versions only go up.
schema_versionsstores, per component, the highest version that has run on this file, and that number is never lowered. When the stored version differs from the component's,connectruns the migration in oneBEGIN IMMEDIATEtransaction; if it raises, the transaction rolls back and the error propagates. A migration also receives a stored version higher than its own, which means a newer ziv wrote the file. A data component raises, so an older ziv leaves a newer component alone. A cache component keeps a table per version instead, so two ziv versions sharing a data folder (a checkout and an installed copy) never rebuild each other's tables. - Connections. Open one short-lived connection per operation,
with closing(app_db.connect(components=(MY_COMPONENT,))) as conn:. Never keep a module-global connection and never share one across threads. Run every write insidewith transaction(conn):and keep it short and free of slow I/O. A lock belongs to the feature that needs it.app_db.runis the shared connect, recover, close and error-wrapping step. - Settings of the file. WAL, a 5 s
busy_timeout,foreign_keys=ON,synchronous=NORMALand autocommit. The switch to WAL is retried for up to the busy timeout, because SQLite does not apply the timeout to that switch. - Corruption.
recover_if_corrupt(conn, exc)is the one place the policy lives. A file that is damaged (SQLITE_CORRUPT,SQLITE_NOTADB) is quarantined: renamed, with its-waland-shm, toziv.sqlite3.corrupt-<YYYYmmdd-HHMMSS>, and only if the file at the path still is the one this connection opened, so a file another process already replaced is left alone. The app never deletes the database. Errors that do not mean a damaged file (cannot open, read-only, busy, a directory at the path) never rename anything. Pass the exception torecover_if_corruptbefore deciding what to do with a failed operation. - Tests. The autouse
_isolated_app_dbfixture points the database at a throwaway folder; see Testing.
Two kinds of component exist, and the difference decides how a change is made:
| Kind | Components | If the data is lost |
|---|---|---|
| Cache, rebuildable from files | gallery, loras |
Re-read from disk; nothing is lost |
| Data, kept for the user | jobs, recent_prompts, lora_sources, assets |
Lost; a migration must copy the rows it keeps, and a quarantine keeps them in the .corrupt-* file |
Job log¶
web/job_log.py is the jobs component: one row per finished job in jobs_entries_v1, ordered by seq. server.py builds it as JobLog(app_db.app_db_path), and WebRunner writes nothing without one. Each row holds the job id and the entry's JSON exactly as written, so a listing returns what was written.
- Which jobs. Only image, video, audio and upscale jobs that started. A job removed from the queue never started, and
dummyjobs are not generation jobs. - Once only. A job is logged when it first reaches a terminal status. A later terminal event changes the in-memory status but writes no second entry, so the log keeps the first outcome. A job whose runs partly failed keeps its
completedorcancelledstatus and counts them infailed_runs, which thefailedfilter matches. - The write never reaches the job.
_publish_eventcopies the facts under the record's lock and writes last, holding no lock, on the worker thread. Building and writing share onetry/except Exceptionthat only emits aUserWarning, so the terminal event, the release of the claim and the next queued job do not depend on the log. If a future change logs queue removals, the write must move off the event loop. - Best effort. Rows are never trimmed; Delete and Clear log remove them. A server stopped while a job runs may exit before its entry is written. Two
ziv uiservers on one data folder share the table safely. - Routes.
GET /api/jobs/log,DELETE /api/jobs/log/{job_id}andDELETE /api/jobs/logare declared before the/api/jobs/{job_id}/...routes. A storage failure, or a schema newer than this ziv (JobLogError), is a 500 withCould not read or write the job log: <reason>.
Recent prompts¶
web/recent_prompts.py is the recent_prompts component: one row per distinct mode, format, prompt, scene and negative prompt in recent_prompts_entries_v1, newest by seq. It is separate from the job log on purpose: the log writes a job only after it ended, truncates the prompt, and its Delete and Clear would otherwise remove recents.
- What is recorded.
_generate_from_formcalls_remember_promptafterPOST /api/generateaccepted a job.recent_prompt_entry(pure) builds the entry for an inline submit and returns nothing for a prompt file or a blank prompt. A refused submit never reaches it. The text recorded is what the form sent. - Modes.
image,video, and for audio the kind (music,speech,sound), each with a list of its own. A song keeps its lyrics inscene. - Dedupe. The key is
(mode, format, prompt, scene, negative_prompt). Recording an existing one gives the row a new highestseqand the newused_at. Nothing is capped or trimmed. - Failures never reach the submit.
_remember_promptonly emits aUserWarning. The routes (GETandDELETE /api/recent-prompts,moderequired) turn a storage failure or a newer schema (RecentPromptsError) into a 500.
Gallery index¶
web/gallery_index.py is the gallery component, a cache. list_gallery_assets calls it on every listing, so /api/gallery and /api/workspace?include_history=true read a file's embedded metadata once per file version instead of once per request, including the first request after a restart.
- Raw facts, not derived fields. A row stores what was read from the file (the embedded settings and the image, video or audio facts), not the derived
GalleryAsset. The derivation (recorded settings, filename parsing, prompt fallback) runs on every request, so fields that depend on other files or on the clock are never stale, and a change to the derivation needs no migration.gallery._read_asset_factsis the only metadata I/O. - Reconciliation. The root is walked with
statonly, never descending into staging directories. A row is used only when its(mtime_ns, ctime_ns, size)equals the walked stat, so a stale or orphan row costs at most one re-read and can never show a file or wrong metadata. - When a read is persisted. Only when no reader failed, the file's mtime is at least 2 s before now (a file still being written is not cached) and a re-stat after the read returns the same key. Persisted reads are committed every 64 files, so an interrupted cold scan keeps its progress.
- Pruning. Rows for files the walk did not see are deleted only by a walk without errors that found at least one media file, because the output folder is created on demand and an unmounted drive looks like an empty folder. A missing root keeps its rows.
delete_gallery_assetscallsGalleryIndex.forget(), which removes rows at once. - Versioned table. The table is
gallery_assets_v{N},N = GALLERY_INDEX_VERSION. Bump the constant when the columns change or when_read_asset_factsreads something different. The migration drops lower-versioned tables only when the stored highest version is belowN, and never a higher one. - Never fails the gallery. An error while opening or loading falls back to reading every file uncached, with a warning, after
recover_if_corruptquarantined a corrupt file. A write error keeps the results already read. - Locking and moves. One
threading.Locksurrounds a reconciliation, so concurrent cold requests read each file once;forget()does not take it.GalleryIndex.rename(root_key, moves)re-keys rows of files moved between projects, so a moved video is not probed again. - Job completion writes no row. The next listing picks the output up, and
gallery_asset_for_output_pathreads uncached.
Asset records¶
web/asset_records.py is the assets component: one row per file in assets_entries(asset_id, starred), keyed by the output-root-relative path so records survive a moved or renamed output folder. The choices cannot be rebuilt from the files, so it is a data component and not a column of the gallery cache. A new choice is a column with a default, not a new table.
- Newer versions are accepted.
migratenever refuses a stored version above its own, because later versions only add columns, which this code ignores. - Moves and deletes.
AssetRecords.renameruns withGalleryIndex.renameingallery._rekey_movedand replaces any row at the new ID. Any delete callsAssetRecords.forget. - Listing never fails on it.
list_gallery_assetsreads the starred set once per listing through_starred_ids, which warns and lists every file unstarred on anAssetRecordsError. The star route turns the same error into a 500, because a star that did not save must not look saved. - Not pruned. A file deleted or renamed outside ziv keeps its row, and a new file that reuses the name shows the old star.
tests/web/test_web_asset_records.py covers the store and tests/web/test_web_gallery.py the listing, move and delete.
LoRA index¶
web/lora_index.py is the loras component, a cache of which model family each LoRA file was trained for, so the Workspace lists the LoRAs made for the selected model first. utils/lora_family.py holds the pure detector (metadata first, then a key and shape fingerprint); utils/safetensors_stream.read_header_and_metadata reads a header in one bounded read, so no tensor data is read.
- Table and key. Rows of
loras_entries_v{N}(N = LORA_FAMILY_VERSION) are keyed by(root, name)and holdfamily,size_variantand the single-captioncaptionof the header beside the stat key. A row is used only when the stat key equals the file's. - Lookup. One
os.statper name and one database read. A file with no row or a stale key is detected inline and stored, so each file is read once until it changes; a header that cannot be classified is stored as an unknown family. The same write deletes rows for LoRAs no longer listed, unless the list is empty, so an unreadable folder does not wipe the cache. - Cache only. A database error warns once and every LoRA is reported with an unknown family, so the picker falls back to a flat list.
- Versioning. Bump
LORA_FAMILY_VERSIONwhen the detection rules or the columns change; the next lookup creates the new table, drops the lower ones and re-detects. It never toucheslora_sources.
LoRA sources¶
web/lora_sources.py is the lora_sources component. It is the only record of where an installed LoRA file came from, so unlike the LoRA index it is not a cache.
- Rows. One row per LoRA name in
lora_sources_v1: the file'ssize_byteswhen written,curated_id,repo,revision,file,sha256, theauto_triggerwith itstrigger_source(curatedorcard), theuser_triggeranddownloaded_at. A row for a file the user only typed a trigger for has norepo. - Hugging Face downloads (not curated ones) also keep the repository's
author,downloadsandlicence, and its first preview image, shrunk to a 384-pixel WebP (utils/lora_preview.py) inlora_images_v1. The Installed tab shows these offline. Fetching the image is best effort and never fails the download;forgetremoves the image with the row. - Written by the lane item's finalize step after a download,
PUT /api/loras/{name}/triggerandDELETE /api/loras/{name}. See Model downloads. - Trusted only while the size matches.
web/lora_details.describe_installedignores a row whosesize_bytesdiffers from the file's current size, so a file replaced by hand is not described by an old row. - Order of trust for a trigger (
resolve_trigger): the trigger the user typed, the curated entry's (a file is a curated entry when its name and size match), the model card'sinstance_promptsaved at download, then a headerss_tag_frequencythat holds a single caption. The file name andss_output_nameare never a trigger.