Other contracts¶
Interpreter overrides¶
A platform package or a study replaces a whole interpreter on a dataset subclass and reuses the original code; the layer flags on the specs of the override pick up the replacement. The registry rules, arguments, errors and an example are in the docstrings of PlatformDataset and PlatformDataset.register_interpreter (Datasets).
- One answer per original. On one class, an override beats the original. A spec that the override leaves out leaves the dataset, with the layer specs on it.
- One scenario. An override that changes only the logic can sit on the class of one scenario. An override that changes a spec applies to all scenarios: a collection of both raises
ValueError. Use the override class for every scenario, and set the behavior per scenario with a parameter default or a dataset attribute. - After the chain.
PlatformDataset._fetchis the hook to process each result after its interpreter chain: a subclass overrides it and changes the result ofsuper()._fetch(...). Nested fetches throughparent_dataset.fetchget the processed result too, and a flag derived from processed inputs passes the hook again. So the processing selects its flags, or gives the same result when it runs twice. It is not part of the version identity: to change results with a version, register an override.
Describe and availability¶
The describe methods and their differences are in the Datasets reference. They read the specs, the source inventory, and model flags where a flag family, a layer or a default attribute needs them. Facts that need time series are flags, not descriptions.
Availability. A flag is available on a dataset when the dataset can answer it. An interpreter implements _is_available(flag) with cheap reads of the source inventory: True, False, None for unknown (the default), or an Availability(available, reason).
- On a platform dataset, the answering interpreter decides;
Nonestays unknown until the first successful fetch. On any other dataset that is not a collection,Nonecounts as available. - A collection is available if one child is, else unknown if one child is unknown. A comparison needs both children.
availability(flag)gives theAvailabilitywith its reason;is_flag_available(flag)itsavailablefield.Availability.of_inputscombines the inputs of a derived flag.describe_flags(available_only=True)keeps available and unknown flags;flag_infogives the reason.
Cache¶
Dataset.fetch caches results in the Cache of the dataset (for example PickleCache) when use_cache is set. The cache stores entries; fetch checks them. A custom cache implements entry, set, delete and list_keys (Cache).
- Key. The key (
CacheKey) holds the dataset class and name, not the source: two datasets of one class and name with different sources share their entries. - Version identity. One
(class name, version)pair per link of the interpreter chain: the original interpreter with the version of its spec, then each override with itsregister_interpreter(version=); for a layer flag also the aggregation with its fields andversion; for a collection its children (class and name) and, for a concat, its layout. Bump a version by hand when the logic changes. - Check. An entry is up to date if its call did not raise, the version identity and the resolved parameters are the same, each recorded config field has the same value (same type and value:
1,1.0andTruediffer), each recorded file has the same size and content hash (the modification time for a file over 16 MiB), and each dependency is up to date, with its call rebuilt from its record. A stale entry is a miss, and the new result replaces it.fetchstores a result only if its record can be up to date. - Limits. A logic change without a version bump keeps the old entries. A file read without
record_file_readis not checked. Unused keys stay untildelete. Two processes that store one key at the same time: the lastsetwins.
Validation¶
A check of mesqual.validation does not return ERROR: it raises, and DatasetValidator reports the exception as ERROR (Validation).
KPIs¶
A KPI is one value per object and dataset, with the attributes of both (KPIs).
KPIDefinitionandCustomKPIDefinitionare the base classes for own KPIs. A platform package or a study subclasses them with its own name, unit and computation per dataset.- The unit of a
FlagAggKPIDefinitionis the unit of its aggregation if it has one, else the unit of the flag. - Two
Aggregations with the same name are equal. Dataset.add_kpisskips a KPI whose name is already in the collection and logs a warning;add_kpis_from_definitionsadds through it.