Synthetic test data¶
The examples in the API reference run on this platform, with the shared names of StudyManager References.
testing
¶
Synthetic scenarios for tests and examples, without an external platform package.
The platform is also the reference implementation of a platform package: flag specs with variants, parameters and memberships, a flag family, derived flags, area and border layers, availability from a source inventory, and default attributes.
What DummyDataset gives for the default DummyScenario:
- Model flags
Country.Model,Node.Model,Line.Model,Generator.ModelandDemand.Model, with points for the nodes and square polygons for the countries, so map code runs without geodata. - Time series: node price, demand per product (
total,hourly,quarter_hourly), both with the parametersolution, generation, line flowLine.Results.flow_{side}_{direction}and line capacity. The source has thetotaldemand, thesentnetflow and theupcapacity. The other variants derive from them throughparent_dataset.fetch, so an override of a source variant reaches them. - Derived flags that list their inputs in
INPUT_FLAGS: line loading, node demand, and generation per node and fuel (a flag family). - The area layer
Countryand the border layerCountryBorder, with area price, demand and generation, and border flows, capacities and price spread. - Availability: a spec reads one raw table (
SPEC_TABLE) or derives from input flags (INPUT_FLAGS).missing_tablesof aDummyScenarioremoves raw tables, so the flags that need them are unavailable.
make_dummy_study() builds a ready study with comparisons;
examples/quickstart.py runs on it.
Examples:
>>> dataset = DummyDataset(DummyScenario(name='base', days=2))
>>> dataset.fetch('Country.Results.price').columns.tolist()
['C1', 'C2']
DummyConfig
dataclass
¶
Bases: DatasetConfig
Config of DummyDataset.
Attributes:
| Name | Type | Description |
|---|---|---|
price_offset |
float
|
Value in EUR/MWh added to every node price. Default: 0.0. |
Source code in src/mesqual/testing/dataset.py
45 46 47 48 49 50 51 52 53 | |
DummyDataset
¶
Bases: PlatformDataset[DummyInterpreter, DummyConfig, str, FlagIndex]
Platform dataset of one synthetic scenario.
Flags: the model tables 'Country.Model', 'Node.Model', 'Line.Model',
'Generator.Model' and 'Demand.Model'; time series of node prices,
demand, generation, line flows and line capacities; the derived line
loading, node demand and generation per node and fuel; the area layer
'Country' on 'Node.Model' and the border layer 'CountryBorder' on
'Line.Model'. describe_class_flags() lists each spec with its unit,
variants, parameters and description.
A flag whose raw table the scenario lists in missing_tables is
unavailable, and so is each flag that reads it. fetch raises
FlagUnavailableError, a ValueError, for it. Without the table
'countries', the geometry columns of the border model are NaN.
Default attributes: 'first_date', 'last_date' and 'days' from the scenario. Explicit attributes win.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
scenario
|
DummyScenario
|
The scenario spec. |
required |
name
|
str | None
|
Dataset name. Default: the scenario name. |
None
|
attributes
|
dict | None
|
Extra attributes. |
None
|
cache
|
Cache | None
|
Optional cache of fetch results. |
None
|
config
|
DummyConfig | None
|
Optional instance config. |
None
|
Examples:
>>> from mesqual.testing import DummyScenario
>>> dataset = DummyDataset(DummyScenario(name='base', days=3))
>>> prices = dataset.fetch('Node.Results.price')
Source code in src/mesqual/testing/dataset.py
138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 | |
build_source_inventory
¶
build_source_inventory() -> DummyInventory
Return the inventory of the synthetic source: its fingerprint and raw tables.
Source code in src/mesqual/testing/dataset.py
190 191 192 | |
get_config_type
classmethod
¶
get_config_type() -> type[DummyConfig]
Return DummyConfig.
Source code in src/mesqual/testing/dataset.py
201 202 203 204 | |
get_child_dataset_type
classmethod
¶
get_child_dataset_type() -> type[DummyInterpreter]
Return DummyInterpreter, the base of all interpreters of this dataset.
Source code in src/mesqual/testing/dataset.py
206 207 208 209 | |
DummyPeriodDataset
¶
Bases: DummyDataset
A DummyDataset with a time level of (run, period) tuples instead of a DatetimeIndex.
It inherits the interpreters of DummyDataset. See DummyPeriodSource for
the run and period numbers.
Source code in src/mesqual/testing/dataset.py
599 600 601 602 603 604 605 606 | |
DummyNetwork
¶
Model tables of a synthetic network.
Countries are squares side by side from west to east. Nodes are placed round robin in the countries, and each node lies inside the square of its country. Node k connects by line to node k-1 and to node k-2, in this order. Generator k and demand k sit at node k // 2, two generators and two demands per node. More nodes, lines, generators or demands only append rows, so existing rows stay the same.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
scenario
|
DummyScenario
|
The scenario spec. |
required |
Source code in src/mesqual/testing/network.py
11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 | |
countries
property
¶
countries: DataFrame
Country table, index 'Country', columns name and geometry (square polygons).
nodes
property
¶
nodes: DataFrame
Node table, index 'Node', columns country, voltage, base_price and location (points).
lines
property
¶
lines: DataFrame
Line table, index 'Line', columns node_from, node_to, capacity and voltage.
generators
property
¶
generators: DataFrame
Generator table, index 'Generator', columns node, fuel and capacity.
demands
property
¶
demands: DataFrame
Demand table, index 'Demand', columns node, sector and base_demand.
DummyScenario
dataclass
¶
Spec of one synthetic scenario.
The generator derives the network, the model tables and the time series from
these fields. Equal specs give equal data. The values of an object on a day
depend only on the seed, the date, the object id and its model properties. So
a scenario split into parts gives the same values as the unsplit scenario,
and more objects (for example a higher n_nodes) or removed objects leave
the other objects unchanged.
Attributes:
| Name | Type | Description |
|---|---|---|
name |
str
|
Scenario name. It becomes the dataset name. |
start |
str
|
First day, for example '2024-03-30'. |
days |
int
|
Number of days. |
tz |
str | None
|
Time zone of the time index, or None for naive timestamps. |
resolution |
str
|
Time step at the start, for example '60min'. |
resolution_change |
tuple[str, str] | None
|
Optional (date, resolution): from that date on the series use the new time step, for example ('2024-03-31', '15min'). |
seed |
int
|
Seed of the random noise. |
profile |
str
|
'daily' for realistic daily, weekly and seasonal profiles with
noise (see |
demand_factor |
float
|
Factor on the demand of every demand object. |
n_countries |
int
|
Number of countries, at least 1. Nodes are placed round
robin in the countries, so a change of |
n_nodes |
int
|
Number of nodes, placed round robin in the countries. |
n_lines |
int
|
Number of lines. Line k connects the nodes in a fixed sequence: each new node connects to the previous node and to the node before that. |
n_generators |
int
|
Number of generators. Generators k and k+1 (k even) sit at
node k/2+1, so at most two generators per node. The fuels repeat in
the order of |
n_demands |
int
|
Number of demands, placed like the generators: demands k and k+1 (k even) sit at node k/2+1. The default gives every node two demands. |
node_without_country |
bool
|
If True, adds the node 'N0' without country, lines and demands. |
removed_objects |
tuple[str, ...]
|
Object ids to remove. A removed node also removes its
lines, generators and demands. The nodes of a removed country keep
their place with no country ( |
text_changes |
tuple[tuple[str, str, str, str], ...]
|
(model table, object, column, value) entries that change text properties, for example ('generators', 'G1', 'fuel', 'gas'). |
missing_tables |
tuple[str, ...]
|
Raw tables the source does not provide, for example
('demand_load',). See |
nan_gaps |
tuple[tuple[str, str], ...]
|
(start, end) windows where all time series values are NaN.
The end is exclusive. A timestamp without offset is local time in
|
missing_rows |
tuple[tuple[str, str], ...]
|
(start, end) windows where the time series have no rows. |
duplicate_rows |
int
|
Number of rows at the start of each time series that appear twice. The copies are appended at the end, so the raw index is not monotonic. |
Source code in src/mesqual/testing/scenario.py
11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 | |
day_list
property
¶
day_list: list[Timestamp]
All days of the scenario as naive timestamps at midnight.
resolution_on
¶
resolution_on(day: Timestamp) -> str
Time step of the series on the given day.
Source code in src/mesqual/testing/scenario.py
114 115 116 117 118 119 120 | |
timestamp
¶
timestamp(value: str) -> Timestamp
Parse a timestamp string in the time zone of the scenario.
A string without offset is local time in tz. A string with offset is
converted to tz.
Source code in src/mesqual/testing/scenario.py
122 123 124 125 126 127 128 129 130 131 132 133 | |
replace
¶
replace(**changes) -> DummyScenario
Return a copy with the given fields changed.
Source code in src/mesqual/testing/scenario.py
135 136 137 | |
split
¶
split(at: str) -> tuple[DummyScenario, DummyScenario]
Split the scenario into two parts with consecutive date ranges.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
at
|
str
|
First day of the second part. It must lie after the first day and not after the last day. |
required |
Returns:
| Type | Description |
|---|---|
tuple[DummyScenario, DummyScenario]
|
The parts ' |
Source code in src/mesqual/testing/scenario.py
139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 | |
DummyPeriodSource
¶
Bases: DummySource
A DummySource with a time level of (run, period) tuples instead of timestamps.
The run is the calendar day as an integer YYYYMMDD, for example 20240330, so the parts of a split scenario keep their run numbers. The period counts the time steps within the day, starting at 1.
Source code in src/mesqual/testing/source.py
126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 | |
DummySource
¶
Raw tables of one synthetic scenario, as a simulation platform would write them.
Model tables come from DummyNetwork. Time series tables have a time index
named 'time' and one column per object. The values of an object on a day
depend only on the seed, the date, the object id and its model properties.
Adding or removing other objects leaves them unchanged.
With profile='daily' the series look like those of a real power system:
solar output is 0 at night and peaks around noon, longer and higher in
summer; wind output varies slowly and is higher in winter; demand has a
morning and an evening peak and is lower at night, on weekends and in
summer; nuclear runs flat, coal and gas follow the net demand (demand minus
solar and wind output); prices follow the net demand. All shapes are taken
at the middle of each time step, and the noise is constant within an hour,
so values at 15 minutes match those at 60 minutes. With profile='constant'
every series is constant.
Time series tables:
- 'node_price': EUR/MWh per node.
- 'generator_generation': MWh per period and generator.
- 'demand_load': MW per demand, times
demand_factor. - 'line_flow': MW per line, net flow where it enters the line, positive from node_from to node_to.
- 'line_capacity': MW per line in line direction. The capacity against the
line direction is
DOWN_CAPACITY_FACTORtimes this value.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
scenario
|
DummyScenario
|
The scenario spec. |
required |
Source code in src/mesqual/testing/source.py
31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 | |
has_table
¶
has_table(table: str) -> bool
True if the source provides the raw table.
Source code in src/mesqual/testing/source.py
85 86 87 | |
read
¶
read(table: str) -> DataFrame
Return a copy of a raw table.
Raises:
| Type | Description |
|---|---|
KeyError
|
If the source does not provide the table. |
Source code in src/mesqual/testing/source.py
89 90 91 92 93 94 95 96 97 98 99 | |
make_dummy_dataset
¶
make_dummy_dataset(scenario: DummyScenario | tuple[DummyScenario, ...], name: str | None = None) -> Dataset
Build the dataset of one scenario or of a scenario split into parts.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
scenario
|
DummyScenario | tuple[DummyScenario, ...]
|
One scenario, or the parts of |
required |
name
|
str | None
|
Dataset name. Default: the scenario name, or for parts their name without the '.N' suffix ('base' for 'base.1' and 'base.2'). |
None
|
Returns:
| Type | Description |
|---|---|
Dataset
|
A |
Raises:
| Type | Description |
|---|---|
ValueError
|
If no part is given, or if the parts have different names
without the suffix and no |
Source code in src/mesqual/testing/dataset.py
609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 | |
make_dummy_study
¶
make_dummy_study(start: str = '2024-03-28', days: int = 5, tz: str | None = 'Europe/Berlin', seed: int = 0, study_folder: str | Path | None = None) -> StudyManager
Build a study of four synthetic scenarios with comparisons against 'base'.
Scenarios:
- 'base': split into two consecutive periods at the middle day, one
DummyDatasetper period, merged into one dataset. The series switch from 60 to 15 minutes at the same day (resolution_change). - 'high_demand': 'base' with 20% more demand, one dataset.
- 'without_N4': 'base' without node N4 and its lines, generators and demands.
- 'short': 'base' one day shorter.
Comparisons: each other scenario against 'base', named 'DatasetComparison.
Every scenario has the attributes 'first_date', 'last_date', 'days', 'demand_factor' and 'periods' (number of merged period datasets).
The default horizon from Thursday 2024-03-28 has a weekend and the DST change on 2024-03-31 in Europe/Berlin.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
start
|
str
|
First day. |
'2024-03-28'
|
days
|
int
|
Number of days, at least 2. |
5
|
tz
|
str | None
|
Time zone of the time index, or None for naive timestamps. |
'Europe/Berlin'
|
seed
|
int
|
Seed of the random noise. |
0
|
study_folder
|
str | Path | None
|
Optional folder of the study, passed to |
None
|
Returns:
| Type | Description |
|---|---|
StudyManager
|
The study, with the scenarios in |
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
Examples:
>>> study = make_dummy_study()
>>> deltas = study.comp.fetch('Node.Results.price')
Source code in src/mesqual/testing/study.py
14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 | |
Interpreters and inventory¶
The interpreters of DummyDataset and the inventory of its source. A study subclasses DummyInterpreter for its own flags, or an interpreter to override it.
Synthetic platform dataset with its interpreters.
DummyInterpreter
¶
Bases: Dataset[DummyConfig, str, FlagIndex]
Base of the interpreters of DummyDataset.
A subclass declares its flags in flag_specs. A spec that reads a raw
table names it in SPEC_TABLE. A derived spec lists the flags it
fetches through the parent dataset in INPUT_FLAGS. An input flag may
use the variant placeholders of the spec, for example
'Demand.Results.demand_{product}'. _is_available reads one of the
two, so every spec must be in exactly one. A subclass that breaks this,
or whose input flag has a placeholder that is no variant of its spec,
raises ValueError at class creation.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
source
|
DummySource
|
The raw tables of the scenario. |
required |
parent_dataset
|
Dataset | None
|
The platform dataset that holds this interpreter. |
None
|
Source code in src/mesqual/testing/dataset.py
56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 | |
get_config_type
classmethod
¶
get_config_type() -> type[DummyConfig]
Return DummyConfig.
Source code in src/mesqual/testing/dataset.py
99 100 101 102 | |
DummyModelInterpreter
¶
Bases: DummyInterpreter
Model tables of countries, nodes, lines, generators and demands.
Source code in src/mesqual/testing/dataset.py
212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 | |
DummyTimeSeriesInterpreter
¶
Bases: DummyInterpreter
Node prices, generation, demand, line flows and line capacities, each from one raw table.
Source code in src/mesqual/testing/dataset.py
287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 | |
SOLUTION_PRICE_STEP
class-attribute
¶
SOLUTION_PRICE_STEP: dict[str, float] = {'first': -1.0, 'final': 0.0}
EUR/MWh added to the final node price per solution, times the number in the node id (N3: 3).
SOLUTION_DEMAND_FACTOR
class-attribute
¶
SOLUTION_DEMAND_FACTOR: dict[str, float] = {'first': 0.98, 'final': 1.0}
Factor on the final total demand, per solution.
DummyDerivedInterpreter
¶
Bases: DummyInterpreter
Line loading, node demand and generation per node and fuel, from their INPUT_FLAGS.
Nodes without demands or generators have no column.
Source code in src/mesqual/testing/dataset.py
447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 | |
DummyAreaFlags
¶
Bases: AreaFlagInterpreter
Flags on the area layer 'Country'.
The layer model is the declared 'Country.Model', so there is no area model spec.
Source code in src/mesqual/testing/dataset.py
540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 | |
DummyBorderFlags
¶
Bases: BorderFlagInterpreter
Flags on the border layer 'CountryBorder'.
Source code in src/mesqual/testing/dataset.py
568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 | |
DummyInventory
dataclass
¶
Bases: SourceInventory
What a synthetic source holds.
Attributes:
| Name | Type | Description |
|---|---|---|
fingerprint |
str
|
Hash of the scenario spec. Equal specs give equal fingerprints. |
variables |
tuple[str, ...]
|
Names of the raw tables the source provides. |
Source code in src/mesqual/testing/source.py
19 20 21 22 23 24 25 26 27 28 | |