Flags and layers¶
A flag spec declares one variable. The fetch path, the flag index and the describe methods read it. Classes, fields and signatures are in the API reference: Flag System, Layer Flags, Units.
The figure shows a line flow spec with one variant dimension, direction. The code below and the synthetic platform use two, flow_{side}_{direction}.
1. Specs¶
Base specs have their own logic, one class per item type: ModelFlagSpec (the model table of one object type, Node.Model), TimeSeriesFlagSpec (usually one column per object of a model flag, Node.Results.price) and OtherFlagSpec (any other table). Layer specs name a base spec and an aggregation: AreaFlagSpec ({area_layer}.Results.price) and BorderFlagSpec ({border_layer}.Results.price_spread).
LINE_FLOW_SPEC = TimeSeriesFlagSpec(
name='Line.Results.flow_{side}_{direction}',
description='Power flow on the line.',
unit=Units.MW,
model_flag='Line.Model',
variants=(SIDE, DIRECTION),
fill_value=0,
)
class LineInterpreter(MyInterpreterBase):
flag_specs = (LINE_MODEL_SPEC, LINE_FLOW_SPEC)
def _fetch(self, flag, effective_config, **params):
...
- Flag names. The flag grammar belongs to the platform. mesqual only fills in the
{placeholders}of a name template, one per variant dimension. Everything mesqual needs comes from spec fields (model_flag,topology_type, ...), never from the name. - Checks.
__post_init__checks the spec itself:quantity_typeonly for an ambiguous unit, placeholders match the variant dimensions, dimension names are unique,edge_endsonly on anEdgemodel spec. Instance creation checks the references:model_flagnames aModelFlagSpecor a layer model of the index that answers the fetch. A platform dataset and a link collection check this on their full index at construction; another dataset checks its own index in its outermost fetch. - No raw names. A spec names no raw tables or columns, and it does not list the flags it fetches: availability comes from the interpreter (other-contracts.md), dependencies are recorded at runtime (core-contracts.md).
- Same spec. Two specs are the same when their class, name, dimensions,
versionand class fields are equal; for layer specs also the base and the aggregation (same_as). Two specs of one name that are not the same raise in a collection and in a flag index. A name that two answering interpreters declare raisesAmbiguousOverrideError. - Flag index.
FlagIndexmaps a flag name to its spec.FlagIndexhas no implicit rules.
2. Dimensions: variants and parameters¶
One class, Dimension, for both roles. A spec lists it in variants= or params=; a dataset class lists its own in fetch_params (dataset parameters).
| Variant | Parameter | |
|---|---|---|
| In the flag name | Yes, as a placeholder | No |
| Each value is | its own flag | a setting of the same flag |
| Values | finite, each with a description | finite with descriptions, or open with validate |
| Set by | the flag name | fetch argument, then param_defaults, then the dimension default |
- Variants.
spec.expand(dataset)gives the flag names,spec.variant_of(flag)the values with attribute access (v.direction). Each flag stays a normal flag name:fetch, the cache and comparisons do not know about variants. - Flag families. A variant whose
valuesis a function of the dataset gives a family, for exampleNode.Results.generation_{fuel}with the fuels ofGenerator.Model. The function may fetch model flags, never time series flags.flag_is_acceptedmatches the template without calling it;fetchand the describe methods call it. A function that needs an unavailable flag gives no values, so the family has no flags on that dataset; any other error raises. - Parameters. A missing
REQUIREDvalue or a value outsidevaluesraisesValueError. Aboolmatches only aboolvalue. The resolved parameters go to_fetchas keyword arguments and into the cache key. - One owner. Every keyword argument of
fetchexceptgranularityis a parameter of the spec of the flag or of the dataset class, for examplecomparison_typeofDatasetComparison. A keyword argument that neither declares raisesValueError, and a name with two owners in one dataset raises at construction. A class has thefetch_paramsof its base classes. - Collections resolve only their own dataset parameters. They pass the flag parameters of the call on, and pass each child the dataset parameters that the child accepts and that the call or the
param_defaultsof the collection gave, never a dimension default. So a child gives the same result inside the collection as alone. - Typed
fetch. A subclass may overridefetchonly to add typed keyword-only arguments for its dataset parameters, defaultUNSET, and passes on only the arguments that are notUNSET.
3. New spec, variant or parameter¶
Ask in this order; the first "yes" decides:
| Question | Yes | Example |
|---|---|---|
| Do the logic, the unit or the quantity type differ? | New spec | A price and a volume of one object. |
| Is it another quantity that a user shows next to the original, and does one implementation cover all values? | Variant | Flow up, down and net; supply per fuel type (res, coal, gas); area layers. |
| Does it select how the same quantity is computed, usually one setting per study? | Parameter | Which solution of a run to read (first, final); how losses count in a trade balance. |
| Is it one more categorical axis of one result that users want in one frame? | dimension level in the schema |
exp, imp, net_exp of a trade balance. |
- Every value explicitly. Interpreters and aggregations handle each variant and parameter value with one branch or lookup entry.
fetchchecks the values before_fetch, so a lookup is enough. An aggregation raisesValueErrorfor a value that it does not know or cannot aggregate (a border sum ofmax(up, down)is not the sum of the line maxima). So a new value fails loudly until its code exists. A platform package fetches every flag of every spec with every parameter value in one test, as the tests ofmesqual.testingdo. - Derived variants. One spec per name pattern. The interpreter computes derived values (
net = up - down) in the same spec and fetches the source values throughparent_dataset.fetch, so an override of a source value reaches them.
4. Values, units, schema, topology, memberships¶
- Fill value.
fill_valueis the value for an object or row that one side of a sum or comparison lacks;Noneor NaN means no fill. A comparison may override it with its dataset parameterfill_value. - Quantity type. The unit registry gives each unit
INTENSIVEorEXTENSIVE(UNKNOWNforNaU,MissingUnitand units outside the registry). A spec setsquantity_typeonly for a unit that a platform markedAMBIGUOUSwithUnits.register_quantity_type. No unit inUnitsisAMBIGUOUS. The quantity type decides how the granularity conversion treats a flag. The type of each unit is in the docstring ofUnits.get_quantity_type_enum(Units). - Schema.
FrameSchemadeclares the index and column levels, each with a name (orNone: unnamed) and a role:time,object,counterpart,object_from,object_to,dimension,dataset(added by collections). A time series spec has the default schema(time: time)x(<index of its model>: object); a model spec(<index>: object)x(None: dimension); a layer spec the base schema with the object level renamed to theindexof the layer. AnOtherFlagSpecand a time series spec withmodel_flag=Nonedeclare their schema.schemamay be a function of the variant values (and of the layer, on a layer spec). - Topology types. A model spec declares its topology type; a time series spec takes it from its model spec.
NodeandNodeConnectedElementaggregate toArea,EdgetoBorder; anAreaflag can give aBorderflag (a price spread), aBorderflag anAreaflag (a trade balance). The two areas of a border need not be neighbours. The visualization type follows from the topology type unless the spec sets it. - Memberships. A model spec declares which columns refer to other model flags (
memberships={'country': 'Country.Model'}) and, for anEdgemodel, its two ends (edge_ends=('node_from', 'node_to'), in line direction). Memberships are the only place where these column names appear.
5. Layers¶
An area layer groups the objects of one type into areas; a border layer gives the borders between the areas of an area layer, with the edges that cross them. The platform registers both and declares every layer spec.
MyPlatformDataset.register_area_layer(AreaLayer(
name='Country', index='country', base_model='Node.Model', layer_model='Country.Model',
description='Countries of the nodes.',
))
MyPlatformDataset.register_border_layer(BorderLayer(
name='CountryBorder', index='country_border', area_layer='Country', edge_model='Line.Model',
description='Borders between two countries, crossed by lines.',
))
AREA_PRICE_SPEC = AreaFlagSpec(
name='{area_layer}.Results.price',
base=NODE_PRICE_SPEC,
aggregation=WeightedMean(weight='Node.Results.demand'),
description='Demand-weighted mean of the node prices of the area.',
)
BORDER_FLOW_SPEC = BorderFlagSpec(
name='{border_layer}.Results.flow_{side}_{direction}',
base=LINE_FLOW_SPEC,
aggregation=BorderFlowSum(),
description='Sum of the line flows across the border, in border direction.',
)
@MyPlatformDataset.register_interpreter
class MyAreaFlags(AreaFlagInterpreter):
flag_specs = (AREA_PRICE_SPEC,)
Layers.
- Several area layers can group the same objects in different ways, for example
Countryand a customMacroregion. A base object with an empty value in the membership column belongs to no area. - Layers are registered per class and resolved along the MRO, like interpreters; a study subclass can add more. The platform declares the layer model flags too (
AreaModel,BorderModelor source tables).
Layer specs.
- A layer spec takes item type, unit, quantity type and
fill_valuefrom its base, its topology type from its class. Its first variant dimension is the layer, named bylayer_placeholder(defaultarea_layerorborder_layer) and resolved per dataset. The spec has the variant dimensions of the base, without the consumed ones and without the layer dimension of a layer base. Its layer dimension takes all matching layers.consumesnames base dimensions that the aggregation resolves itself; the availability check then reads the base flag of their first values only, so the aggregation lists the other base flags it fetches ininput_flags. - A layer spec on a layer spec of its own class raises
TypeError. - The base parameters stay;
paramsadds parameters of the aggregation, which go toaggregateonly. A layer flag passes each nested fetch only the parameters that the spec of that flag declares. - A time series spec with a
model_flagtemplate ('{area_layer}.Model') is for a platform that reads area values from its source.
Interpreters and aggregations.
AreaFlagInterpreterandBorderFlagInterpreterfetch the base flag and the mapping (base model or border model) throughparent_dataset.fetch, so overrides of the base reach the layer flags, and callaggregate. A spec that consumes base dimensions getsvalues=None: the interpreter does not fetch the base flag, and the aggregation fetches what it needs. Only a model spec on a node base always gets the base model. An aggregation on a node or node-connected base (area spec) or an edge base (border spec) maps one object level; an area spec on a border base gets the border model and may build several column levels (example-trade-balance.md).- A layer flag is available when the flags its fetch reads are available: the base flag, the base or border model, and the
input_flagsof the aggregation. The reason names the deciding inputs. - An aggregation is a frozen dataclass with a
version(part of the version identity). It lists the flags it fetches besides the base ininput_flags, and checks the spec at declaration incheck_spec. A platform or study subclassesAreaAggregationorBorderAggregationfor special cases. aggregate(values, mapping, dataset, layer, variant, effective_config, **params)gets the platform dataset and theeffective_configof the layer flag fetch; aBorderAggregationgets the border model in place ofmapping. It passeseffective_configto each fetch, as an interpreter does:dataset.fetch(flag, effective_config), orself.fetch(dataset, flag, effective_config, params), which also leaves out the parameters that the spec offlagdoes not declare.
describe_class_flags() and describe_flags() list a layer spec once, with its layers and their model flags. describe_class_area_layers() and describe_class_border_layers() list each layer once. objects_without_area(dataset, layer) and edges_on_no_border(dataset, layer) count what a layer leaves out.