silloncommon internals

The shared data layer and the client/server command protocol.

Database (ORM + queries)

silloncommon.database

SimulationTable

Bases: SQLModel

The core database table representing a single simulation run.

This table stores all lightweight execution data, including tracking metadata, JSON parameters, metrics, and git states. It has a one-to-many relationship with ArtifactTable for heavy file storage.

Attributes:

Name	Type	Description
id	Optional[int]	Primary key generated by the database.
uuid	str	Unique identifier for the run.
name	Optional[str]	Human-readable name given to the run.
date	Optional[str]	Timestamp of when the run occurred.
parameters	Dict[str, Any]	JSON dictionary of input parameters.
results	Dict[str, Any]	JSON dictionary of output metrics/values.
meta_data	Dict[str, Any]	JSON dictionary of environment metadata.
tag	Optional[List[Any]]	JSON list of user-defined tags.
note	Optional[List[Any]]	JSON list of user notes.
executiontime	Optional[str]	The duration of the run.
status	Optional[str]	Run status (e.g., "SUCCESS", "FAILURE", "KILLED").
platform	Optional[str]	Environment type (e.g., "python", "cpp").
hostname	Optional[str]	Network name of the machine executing the run.
isdirty	Optional[bool]	True if uncommitted changes existed during the run.
organisation	Optional[str]	Organization associated with the run.
author	Optional[str]	Author/user who triggered the run.
project	Optional[str]	Project name of the .sillon repository.
hashes	Optional[str]	Dictionarry of all the hashes of the simulation
sillonversion	Optional[str]	Version of Simtrack used to record the run.
artifacts	List[ArtifactTable]	Linked artifacts generated by this run.

ArtifactTable

Bases: SQLModel

Represents a physical file (blob) generated by a simulation run.

Attributes:

Name	Type	Description
id	Optional[int]	Primary key.
name	Optional[str]	Human-readable filename.
hash	str	The SHA hash used for content-addressable storage deduplication.
path	str	The physical path where the blob is stored.
type	Optional[str]	Categorization tag (e.g., "plot", "mesh", "log").
size	Optional[str]	File size representation.
run_id	Optional[int]	Foreign key linking to the parent SimulationTable.
simulation	Optional[SimulationTable]	Relationship to the parent run.

FigureTable

Bases: SQLModel

Represents a figure produced and tracked during a simulation run.

A figure is stored on disk like an artifact (content-addressable copy) but carries its own provenance metadata: which parameters and results were used to draw it, a caption, the file format... This is what lets a user answer "what data was used for this figure?" months later.

Attributes:

Name	Type	Description
id	Optional[int]	Primary key.
name	Optional[str]	Human-readable figure name.
hsh	str	The SHA hash of the figure file for integrity tracking.
path	str	The storage pointer of the figure file.
meta	Dict[str, Any]	Provenance metadata. Reserved keys: used (list of parameter/result names the figure was built from), caption, format.
date	Optional[str]	Timestamp of when the figure was logged.
run_id	Optional[int]	Foreign key linking to the parent run.
simulation	Optional[SimulationTable]	Relationship to the parent run.

AnalysisTable

Bases: SQLModel

Represents post-processed data attached to a run after the fact.

Unlike results, analyses are not produced by the simulation itself: they are computed later (e.g., evaluating a fitted function on a new grid) and linked back to the run for reuse. The heavy data lives in the run's HDF5 glob under the analysis group; this row holds the pointer and context.

Attributes:

Name	Type	Description
id	Optional[int]	Primary key.
name	Optional[str]	The analysis name (also the glob dataset name).
hsh	str	The SHA hash of the analysis data.
pointer	str	The dataset pointer inside the run's glob.
meta	Dict[str, Any]	Free-form context (inputs used, comment...).
date	Optional[str]	Timestamp of when the analysis was attached.
run_id	Optional[int]	Foreign key linking to the parent run.
simulation	Optional[SimulationTable]	Relationship to the parent run.

DbSelectResult dataclass

Standardized dataclass for holding queried key-value metrics/parameters.

Attributes:

Name	Type	Description
run_name	str	The name of the run.
date	str	Timestamp of the run.
key	str	The queried parameter/metric name.
value	str	The value of the parameter/metric.

DbSelectResultArtifact dataclass

Standardized dataclass for holding queried artifact pointers.

Attributes:

Name	Type	Description
run_name	str	The name of the run.
date	str	Timestamp of the run.
value	str	The path or hash pointer of the artifact.

make_engine(db_path)

Create a SQLite engine with sillon's pragmas attached.

The single place engines are built, so no code path can accidentally get an unconfigured (non-WAL) connection.

schema_version(engine)

The schema version recorded in a database file, or 0 if none is.

migrate_schema(engine)

Brings an existing database up to the current schema, idempotently.

Creates missing tables, adds any column the ORM declares and the file lacks, creates the declared indexes, backfills started_at, and records the schema version so a reader can tell what it is looking at.

Best-effort throughout: a database that cannot be migrated — read-only media, a pre-existing duplicate blocking an index — must still open.

Parameters:

Name	Type	Description	Default
engine	Engine	The SQLAlchemy engine to migrate.	required

Returns:

Name	Type	Description
Engine	Engine	The same engine (for chaining).

sqlite_url(db_path)

Build a SQLite connection URL from a filesystem path.

Uses forward slashes even on Windows: str(Path) there yields C:\Users\...\database.sql, and backslashes inside a URL are not portable. sqlite:///C:/Users/.../database.sql is the documented form.

get_engine(project_path)

Initializes and returns the SQLite database engine.

Parameters:

Name	Type	Description	Default
project_path	Path	The root pathlib.Path of the current sillon project.	required

Returns:

Name	Type	Description
Engine	Engine	A SQLAlchemy engine connected to .sillon/database.sql.

insert_multiple_simulations(simulations, session)

Saves a dictionary of multiple simulation objects to the database.

Iterates over a collection of runs, splits their standard metrics from heavy artifacts, and constructs the relational rows before bulk-committing them to SQLite.

Parameters:

Name	Type	Description	Default
simulations	dict	A dictionary mapping run IDs to run objects.	required
session	Session	The active SQLModel database session.	required

Returns:

Type	Description
List[SimulationTable]	List[SimulationTable]: A list of the successfully committed database rows.

insert_simulation(run, session)

Saves a single simulation object to the database.

Parameters:

Name	Type	Description	Default
run	Any	The simulation data object.	required
session	Session	The active SQLModel database session.	required

Returns:

Name	Type	Description
SimulationTable	SimulationTable	The successfully committed database row for the run.

insert_imported_run(record, session, name=None)

Inserts a run that was produced somewhere else, as it was.

The sibling of insert_simulation, which is duck-typed on a live Simulation and therefore cannot be used here: Simulation.__init__ stamps date = now() and mints a name, so a run inserted through it loses the identity it is supposed to be keeping.

Everything the row carried is preserved -- uuid, both timestamps, status, runtime, hostname, author and the sillon version that wrote it -- except the integer primary keys, which are machine-local. Child rows are attached through the relationship, so SQLAlchemy assigns each a fresh run_id on commit, exactly as it does for a freshly logged run.

Unknown keys in record are ignored rather than raising, so a bundle written by a newer sillon imports its fields that this one understands and says so, instead of refusing outright.

Parameters:

Name	Type	Description	Default
record	dict	{"run": {...}, "artifacts": [...], "figures": [...], "analyses": [...]}, as select_run_snapshot produces it.	required
session	Session	The active session. Not committed here -- the caller owns the transaction, so a whole import succeeds or fails together.	required
name	str	A replacement name, for when the original collides with a local run. Defaults to the recorded name.	None

Returns:

Name	Type	Description
SimulationTable	SimulationTable	The pending row, added to the session.

parse_db_result(statement_out)

Parses raw SQL tuples from JSON queries into Dataclass objects.

Parameters:

Name	Type	Description	Default
statement_out	List[tuple]	Raw tuple rows returned by a SQLModel execution.	required

Returns:

Type	Description
List[Union[DbSelectResult, DbSelectResultArtifact]]	List[Any]: A list of populated DbSelectResult or DbSelectResultArtifact objects.

select_param_user(engine, search_key=None, search_id=None)

Retrieves targeted parameters for the user CLI/Dashboard, parsed into objects.

Parameters:

Name	Type	Description	Default
engine	Engine	The SQLAlchemy engine.	required
search_key	Optional[List[str]]	Keys to filter for (e.g., ["learning_rate"]).	None
search_id	Optional[List[str]]	Run names to limit the query to.	None

Returns:

Type	Description
List[Union[DbSelectResult, DbSelectResultArtifact]]	List[DbSelectResult]: Clean, parsed dataclasses ready for UI presentation.

select_metadata_user(engine, search_key=None, search_id=None)

Retrieves targeted metadata for the user CLI/Dashboard, parsed into objects.

Parameters:

Name	Type	Description	Default
engine	Engine	The SQLAlchemy engine.	required
search_key	Optional[List[str]]	Keys to filter for (e.g., ["hostname"]).	None
search_id	Optional[List[str]]	Run names to limit the query to.	None

Returns:

Type	Description
List[Union[DbSelectResult, DbSelectResultArtifact]]	List[DbSelectResult]: Clean, parsed dataclasses ready for UI presentation.

select_result_user(engine, search_key=None, search_id=[])

Retrieves targeted results and artifacts for the CLI, parsed into objects.

Parameters:

Name	Type	Description	Default
engine	Engine	The SQLAlchemy engine.	required
search_key	Optional[List[str]]	Target keys to filter.	None
search_id	List[str]	Run names to limit the query to.	[]

Returns:

Name	Type	Description
tuple	Tuple[List[Union[DbSelectResult, DbSelectResultArtifact]], List[Union[DbSelectResult, DbSelectResultArtifact]]]	Parsed UI-ready results and parsed UI-ready artifacts.

select_all_user(engine)

Retrieves a top-level summary of all runs for the UI dashboard.

Parameters:

Name	Type	Description	Default
engine	Engine	The SQLAlchemy engine.	required

Returns:

Name	Type	Description
tuple	Tuple[Sequence[Any], Sequence[Any]]	Raw UI summary details and connected artifact IDs.

select_key_user(engine, search_id)

Retrieves deep data for specific runs to display in a CLI comparison.

Parameters:

Name	Type	Description	Default
engine	Engine	The SQLAlchemy engine.	required
search_id	List[str]	The target run names (e.g., ["run_A", "run_B"]).	required

Returns:

Name	Type	Description
tuple	Tuple[List[Any], List[Any]]	Run details and associated artifact pointers.

select_run_snapshot(engine, search_id)

Retrieves the full database row of a single run, by name or uuid.

Unlike the column-oriented selects above, this returns every field of the run (including tags and notes) plus its linked artifact, figure, and analysis rows, as plain dictionaries safe to use outside of a session.

Databases created before the figure/analysis tables existed are handled gracefully (the corresponding lists are empty).

Parameters:

Name	Type	Description	Default
engine	Engine	The SQLAlchemy engine.	required
search_id	str	The run name or uuid to look up.	required

Returns:

Name	Type	Description
tuple	Tuple[Optional[Dict[str, Any]], List[Dict[str, Any]], List[Dict[str, Any]], List[Dict[str, Any]]]	A tuple containing: - Optional[dict]: The run row as a dictionary, or None if not found. - List[dict]: The artifact rows linked to the run. - List[dict]: The figure rows linked to the run. - List[dict]: The analysis rows linked to the run.

db_insert_analysis(engine, run_db_id, name, pointer, hsh, meta=None)

Inserts a post-processed analysis row linked to an existing run.

The analysis table is created on the fly if the database predates it.

Parameters:

Name	Type	Description	Default
engine	Engine	The SQLAlchemy engine.	required
run_db_id	int	The database id (primary key) of the parent run.	required
name	str	The analysis name (also the glob dataset name).	required
pointer	str	The dataset pointer inside the run's glob.	required
hsh	str	The SHA hash of the analysis data.	required
meta	dict	Free-form context of the analysis.	None

Returns:

Name	Type	Description
dict	Dict[str, Any]	The inserted analysis row as a dictionary.

select_snapshot_name(engine, search_id)

Retrieves deep data for specific runs to display in a CLI comparison.

Parameters:

Name	Type	Description	Default
engine	Engine	The SQLAlchemy engine.	required
search_id	List[str]	The target run names (e.g., ["run_A", "run_B"]).	required

Returns:

Name	Type	Description
tuple	Tuple[List[Any], List[Any]]	Run details and associated artifact pointers.

select_param_all(engine)

Extracts the entire parameters column for all runs in the database.

select_result_all(engine)

Extracts the entire results column for all runs in the database.

select_metadata_all(engine)

Extracts the entire metadata column for all runs in the database.

select_uuids(engine)

Every run uuid in the database.

Also the set an import dedups against: a uuid already here means that run is already here, so importing it again is a no-op rather than a duplicate.

select_run_index(engine)

Builds a rich, glob-free searchable index of every run, in bulk.

Returns everything needed to filter runs cheaply (without touching any HDF5 glob): every database column plus the names of each run's artifacts, analyses, and figures. The number of SQL queries is constant, independent of the number of runs — this is the backbone of the two-phase query (cheap DB filtering first, glob reads only on the survivors).

Parameters:

Name	Type	Description	Default
engine	Engine	The SQLAlchemy engine.	required

Returns:

Type	Description
List[Dict[str, Any]]	List[dict]: One entry per run with its identity and columns (id, uuid, name, date, status, author, hostname, platform, runtime, sillonversion), its JSON columns (parameters, results, meta_data, tag, note, hashes), and the linked-item name lists (result_names, artifacts, analyses, figures).

select_run_identities(engine)

Returns the identity fields of every run (id, name, uuid, date).

Lightweight helper for storage operations (pruning, exporting) that need to map run names to their uuid storage folders and creation date.

Parameters:

Name	Type	Description	Default
engine	Engine	The SQLAlchemy engine.	required

Returns:

Type	Description
List[Dict[str, Any]]	List[dict]: One {id, name, uuid, date, started_at, when} per run, where when is the timezone-correct sort key.

db_delete_runs(engine, identifiers)

Deletes runs and their linked rows from the database.

Removes the matching SimulationTable rows along with their artifacts, figures, and analyses. Matches on run name or uuid.

Parameters:

Name	Type	Description	Default
engine	Engine	The SQLAlchemy engine.	required
identifiers	List[str]	Run names or uuids to delete.	required

Returns:

Type	Description
List[str]	List[str]: The names of the runs that were deleted.

resolve_available_name(name, existing)

The first free name in the series name, name_2, name_3, ...

The pure half of next_available_name, so a caller holding many names to place — an import — can fetch the taken set once instead of querying per run.

Parameters:

Name	Type	Description	Default
name	str	The desired base name.	required
existing	set[str]	Names already taken.	required

Returns:

Name	Type	Description
str	str	A name not in existing.

select_names(engine)

Every run name in the database.

next_available_name(engine, name, taken=())

Returns a unique run name, appending an increment on collision.

A name is considered taken if it already exists in the database or appears in taken (in-flight runs not yet committed).

Parameters:

Name	Type	Description	Default
engine	Engine	The SQLAlchemy engine.	required
name	str	The desired base run name.	required
taken	Sequence[str]	Extra names to treat as already used (e.g. runs currently registered on the server but not yet dumped).	()

Returns:

Name	Type	Description
str	str	A run name guaranteed not to collide with existing or in-flight runs.

check_hashes_exists(input_hash, engine)

Checks the database to see if a specific file blob hash already exists.

Crucial for content-addressable storage to prevent duplicating massive files.

Parameters:

Name	Type	Description	Default
input_hash	str	The SHA hash of the file.	required
engine	Engine	The SQLAlchemy engine.	required

Returns:

Type	Description
Optional[Tuple[str, str]]	Optional[tuple]: The path and hash if it exists, otherwise None.

select_by_hash(engine, hsh)

Finds every figure or artifact whose content hash matches hsh.

Answers "which run owns this file?" — figure and artifact hashes are stored and indexed, so a file hashed with get_hash can be traced back to its run.

Parameters:

Name	Type	Description	Default
engine	Engine	The SQLAlchemy engine.	required
hsh	str	The SHA-256 hash to look up.	required

Returns:

Type	Description
List[Dict[str, Any]]	List[dict]: One {run_name, run_uuid, kind, name} per match (kind is "figure" or "artifact").

db_rename_run(engine, identifier, new_name)

Renames a run (matched by name or uuid) to new_name.

Storage is uuid-based, so a rename is a pure metadata update.

Parameters:

Name	Type	Description	Default
engine	Engine	The SQLAlchemy engine.	required
identifier	str	The current run name or uuid.	required
new_name	str	The new name.	required

Returns:

Type	Description
Optional[Dict[str, str]]	dict | None: {"old": ..., "new": ...} on success, or None if no run matched identifier.

db_append_note_tag(engine, run_names, notes=None, tags=None)

Strictly handles the SQLite updating logic. Returns a list of the run names that were successfully updated.

db_append_metadata(engine, run_names, metadata)

Merges new metadata keys into the meta_data column of existing runs.

Parameters:

Name	Type	Description	Default
engine	Engine	The SQLAlchemy engine.	required
run_names	List[str]	The run names or uuids to update.	required
metadata	dict	The metadata key/value pairs to merge in.	required

Returns:

Type	Description
List[str]	List[str]: The names of the runs that were successfully updated.

db_backfill_code_versions(engine, updates)

Writes recovered code versions onto existing runs.

Parameters:

Name	Type	Description	Default
engine	Engine	The active SQLAlchemy database engine.	required
updates	dict	{run uuid: version dict} from silloncore.engine.backfill_code_versions.	required

Returns:

Name	Type	Description
int	int	How many rows were updated.

Command protocol

silloncommon.commands

JSON-RPC handler

silloncommon.rpcHandler

NumpyEncoder

Bases: JSONEncoder

JSON encoder that handles NumPy and Python scientific types. Raises TypeError for unknown custom objects rather than silently producing useless output.

RPCHandler

decode_request(request)

Decodes the outer JSON-RPC envelope. Does not decode params — that is done in _dispatch where the decode hook can be applied.

decode_params(params)

Decodes the params payload, reconstructing special types such as complex numbers encoded by NumpyEncoder.

decode_response(response)

Decodes a JSON-RPC response. Raises if the response contains an error.

encode_request(command, command_id, run_id)

Encodes a command into a JSON-RPC request string. Uses NumpyEncoder to handle scientific types in the params payload.

encode_response(response, id, error='')

Encodes a result or error into a JSON-RPC response string.