Skip to content

Dataset References

Describe a dataset

The describe methods show what a dataset offers. They load no time series data.

Describe methods work on an instance. A method with class_ in its name works on the class.

  • describe_class_flags(): the flag specs that a dataset, interpreter or platform dataset class declares.
  • describe_flags(): the flags that a dataset can deliver, with resolved values, parameter defaults and availability.
  • flag_info(flag): all details of one flag, as a FlagInfo.
  • describe_class_area_layers(): the area layers registered on a dataset class.
  • describe_class_border_layers(): the border layers registered on a dataset class.
  • describe_datasets(max_depth=None): the tree of a dataset and its children, down to max_depth levels.
  • describe_config(): the config fields of a dataset, with their effective values.
  • StudyManager.describe_study(): the scenarios and comparisons of a study, and the flags of each.

Use describe_class_flags to see what a platform offers before you build a dataset. Use describe_flags to see what you can fetch from a dataset you have.

Both give one row per spec, with the same index. The columns of describe_class_flags are the first columns of describe_flags, in the same order. describe_flags only adds columns.

A layer spec on area or border layers (AreaFlagSpec, BorderFlagSpec) has one row. layers names its layers. object names the model flags of these layers, in the same order, for example Country.Model.

describe_class_flags() describe_flags()
Question Which flags does this kind of dataset declare? Which flags can this dataset deliver, and with which settings?
Called on a dataset, interpreter or platform dataset class. A collection class raises TypeError. an instance
Reads spec declarations only model flags and the source inventory; never time series
Flag families template, for example Node.Results.generation_{fuel}, with the declared values or resolved per dataset resolved from the dataset, for example one entry per fuel
Layers the layers registered on the class the registered layers of the dataset
Parameter defaults spec default effective default and its source: spec, dataset, config, required or per dataset
Availability none available, unknown, partial or unavailable, and the number of flags
Collections not supported: a collection has no flags of its own union of the children, plus which children have each flag

All describe methods except flag_info and describe_config take as_records=True and then return plain dicts, ready for JSON.

datasets

Datasets, collections, comparisons, platform datasets and their configs.

INHERIT module-attribute

INHERIT = _Inherit()

FetchRecord.call_config of a call that got the effective config of the enclosing fetch as its config.

UNSET module-attribute

UNSET = Unset()

Default of the typed dataset parameters of a fetch override: the call does not give the parameter.

The parameter then resolves from param_defaults of the effective config, else from the default of its dimension (Dataset.fetch_params).

Dataset

Bases: Generic[DatasetConfigType, FlagType, FlagIndexType], ABC

Base class of all datasets: one fetch(flag) interface for all data.

A scenario, a scenario merged from several runs or sources, a collection of scenarios and a scenario comparison are all datasets.

A dataset declares its flags with flag specs in flag_specs and implements _fetch(flag, effective_config, **params). accepted_flags comes from the specs. A dataset can have attributes (scenario metadata), a KPI collection and a cache of fetch results.

Attributes:

Name Type Description
name str

Name of the dataset.

kpi_collection KPICollection

The KPIs of the dataset.

Examples:

A dataset class:

class NodeResults(Dataset):
    flag_specs = (NODE_MODEL_SPEC, NODE_PRICE_SPEC)

    def _fetch(self, flag, effective_config, **params):
        ...

A fetch:

>>> prices = dataset.fetch('Node.Results.price')
Source code in src/mesqual/datasets/dataset.py
 136
 137
 138
 139
 140
 141
 142
 143
 144
 145
 146
 147
 148
 149
 150
 151
 152
 153
 154
 155
 156
 157
 158
 159
 160
 161
 162
 163
 164
 165
 166
 167
 168
 169
 170
 171
 172
 173
 174
 175
 176
 177
 178
 179
 180
 181
 182
 183
 184
 185
 186
 187
 188
 189
 190
 191
 192
 193
 194
 195
 196
 197
 198
 199
 200
 201
 202
 203
 204
 205
 206
 207
 208
 209
 210
 211
 212
 213
 214
 215
 216
 217
 218
 219
 220
 221
 222
 223
 224
 225
 226
 227
 228
 229
 230
 231
 232
 233
 234
 235
 236
 237
 238
 239
 240
 241
 242
 243
 244
 245
 246
 247
 248
 249
 250
 251
 252
 253
 254
 255
 256
 257
 258
 259
 260
 261
 262
 263
 264
 265
 266
 267
 268
 269
 270
 271
 272
 273
 274
 275
 276
 277
 278
 279
 280
 281
 282
 283
 284
 285
 286
 287
 288
 289
 290
 291
 292
 293
 294
 295
 296
 297
 298
 299
 300
 301
 302
 303
 304
 305
 306
 307
 308
 309
 310
 311
 312
 313
 314
 315
 316
 317
 318
 319
 320
 321
 322
 323
 324
 325
 326
 327
 328
 329
 330
 331
 332
 333
 334
 335
 336
 337
 338
 339
 340
 341
 342
 343
 344
 345
 346
 347
 348
 349
 350
 351
 352
 353
 354
 355
 356
 357
 358
 359
 360
 361
 362
 363
 364
 365
 366
 367
 368
 369
 370
 371
 372
 373
 374
 375
 376
 377
 378
 379
 380
 381
 382
 383
 384
 385
 386
 387
 388
 389
 390
 391
 392
 393
 394
 395
 396
 397
 398
 399
 400
 401
 402
 403
 404
 405
 406
 407
 408
 409
 410
 411
 412
 413
 414
 415
 416
 417
 418
 419
 420
 421
 422
 423
 424
 425
 426
 427
 428
 429
 430
 431
 432
 433
 434
 435
 436
 437
 438
 439
 440
 441
 442
 443
 444
 445
 446
 447
 448
 449
 450
 451
 452
 453
 454
 455
 456
 457
 458
 459
 460
 461
 462
 463
 464
 465
 466
 467
 468
 469
 470
 471
 472
 473
 474
 475
 476
 477
 478
 479
 480
 481
 482
 483
 484
 485
 486
 487
 488
 489
 490
 491
 492
 493
 494
 495
 496
 497
 498
 499
 500
 501
 502
 503
 504
 505
 506
 507
 508
 509
 510
 511
 512
 513
 514
 515
 516
 517
 518
 519
 520
 521
 522
 523
 524
 525
 526
 527
 528
 529
 530
 531
 532
 533
 534
 535
 536
 537
 538
 539
 540
 541
 542
 543
 544
 545
 546
 547
 548
 549
 550
 551
 552
 553
 554
 555
 556
 557
 558
 559
 560
 561
 562
 563
 564
 565
 566
 567
 568
 569
 570
 571
 572
 573
 574
 575
 576
 577
 578
 579
 580
 581
 582
 583
 584
 585
 586
 587
 588
 589
 590
 591
 592
 593
 594
 595
 596
 597
 598
 599
 600
 601
 602
 603
 604
 605
 606
 607
 608
 609
 610
 611
 612
 613
 614
 615
 616
 617
 618
 619
 620
 621
 622
 623
 624
 625
 626
 627
 628
 629
 630
 631
 632
 633
 634
 635
 636
 637
 638
 639
 640
 641
 642
 643
 644
 645
 646
 647
 648
 649
 650
 651
 652
 653
 654
 655
 656
 657
 658
 659
 660
 661
 662
 663
 664
 665
 666
 667
 668
 669
 670
 671
 672
 673
 674
 675
 676
 677
 678
 679
 680
 681
 682
 683
 684
 685
 686
 687
 688
 689
 690
 691
 692
 693
 694
 695
 696
 697
 698
 699
 700
 701
 702
 703
 704
 705
 706
 707
 708
 709
 710
 711
 712
 713
 714
 715
 716
 717
 718
 719
 720
 721
 722
 723
 724
 725
 726
 727
 728
 729
 730
 731
 732
 733
 734
 735
 736
 737
 738
 739
 740
 741
 742
 743
 744
 745
 746
 747
 748
 749
 750
 751
 752
 753
 754
 755
 756
 757
 758
 759
 760
 761
 762
 763
 764
 765
 766
 767
 768
 769
 770
 771
 772
 773
 774
 775
 776
 777
 778
 779
 780
 781
 782
 783
 784
 785
 786
 787
 788
 789
 790
 791
 792
 793
 794
 795
 796
 797
 798
 799
 800
 801
 802
 803
 804
 805
 806
 807
 808
 809
 810
 811
 812
 813
 814
 815
 816
 817
 818
 819
 820
 821
 822
 823
 824
 825
 826
 827
 828
 829
 830
 831
 832
 833
 834
 835
 836
 837
 838
 839
 840
 841
 842
 843
 844
 845
 846
 847
 848
 849
 850
 851
 852
 853
 854
 855
 856
 857
 858
 859
 860
 861
 862
 863
 864
 865
 866
 867
 868
 869
 870
 871
 872
 873
 874
 875
 876
 877
 878
 879
 880
 881
 882
 883
 884
 885
 886
 887
 888
 889
 890
 891
 892
 893
 894
 895
 896
 897
 898
 899
 900
 901
 902
 903
 904
 905
 906
 907
 908
 909
 910
 911
 912
 913
 914
 915
 916
 917
 918
 919
 920
 921
 922
 923
 924
 925
 926
 927
 928
 929
 930
 931
 932
 933
 934
 935
 936
 937
 938
 939
 940
 941
 942
 943
 944
 945
 946
 947
 948
 949
 950
 951
 952
 953
 954
 955
 956
 957
 958
 959
 960
 961
 962
 963
 964
 965
 966
 967
 968
 969
 970
 971
 972
 973
 974
 975
 976
 977
 978
 979
 980
 981
 982
 983
 984
 985
 986
 987
 988
 989
 990
 991
 992
 993
 994
 995
 996
 997
 998
 999
1000
1001
1002
1003
1004
1005
1006
1007
1008
1009
1010
1011
1012
1013
1014
1015
1016
1017
1018
1019
1020
1021
1022
1023
1024
1025
1026
1027
1028
1029
1030
1031
1032
1033
1034
1035
1036
1037
1038
1039
1040
1041
1042
1043
1044
1045
1046
1047
1048
1049
1050
1051
1052
1053
1054
1055
1056
1057
1058
1059
1060
1061
1062
1063
1064
1065
1066
1067
1068
1069
1070
1071
1072
1073
1074
1075
1076
1077
1078
1079
1080
1081
1082
1083
1084
1085
1086
1087
1088
1089
1090
1091
1092
1093
1094
1095
1096
1097
1098
1099
1100
1101
1102
1103
1104
1105
1106
1107
1108
1109
1110
1111
1112
1113
1114
1115
1116
1117
1118
1119
1120
1121
1122
1123
1124
1125
1126
1127
1128
1129
1130
1131
1132
1133
1134
1135
1136
1137
1138
1139
1140
1141
1142
1143
1144
1145
1146
1147
1148
1149
1150
1151
1152
1153
1154
1155
1156
1157
1158
1159
1160
1161
1162
1163
1164
1165
1166
1167
1168
1169
1170
1171
1172
1173
1174
1175
1176
1177
1178
1179
1180
1181
1182
1183
1184
1185
1186
1187
1188
1189
1190
1191
1192
1193
1194
1195
1196
1197
1198
1199
1200
1201
1202
1203
1204
1205
1206
1207
1208
1209
1210
1211
1212
1213
1214
1215
1216
1217
1218
1219
1220
1221
1222
1223
1224
1225
1226
1227
1228
1229
1230
1231
1232
1233
1234
1235
1236
1237
1238
1239
1240
1241
1242
1243
1244
1245
1246
1247
1248
1249
1250
1251
1252
1253
1254
1255
1256
1257
1258
1259
1260
1261
1262
1263
1264
1265
1266
1267
1268
1269
1270
1271
1272
1273
1274
1275
1276
1277
1278
1279
1280
1281
1282
1283
1284
1285
1286
1287
1288
1289
1290
1291
1292
1293
1294
1295
1296
1297
1298
1299
1300
1301
1302
1303
1304
1305
1306
1307
1308
1309
1310
1311
1312
1313
1314
1315
1316
1317
1318
1319
1320
1321
1322
1323
1324
1325
1326
1327
1328
1329
1330
1331
1332
1333
1334
1335
1336
1337
1338
1339
1340
1341
1342
1343
1344
1345
1346
1347
1348
1349
1350
1351
1352
1353
1354
1355
1356
1357
1358
1359
1360
1361
1362
1363
1364
1365
1366
1367
1368
1369
1370
1371
1372
1373
1374
1375
1376
1377
1378
1379
1380
1381
1382
1383
1384
1385
1386
1387
1388
1389
1390
1391
1392
1393
1394
1395
1396
1397
1398
1399
1400
1401
1402
1403
1404
1405
1406
1407
1408
1409
1410
1411
1412
1413
1414
1415
1416
1417
1418
1419
1420
1421
1422
1423
1424
1425
1426
1427
1428
1429
1430
1431
1432
1433
1434
1435
1436
1437
1438
1439
1440
1441
1442
1443
1444
1445
1446
1447
1448
1449
1450
1451
1452
1453
1454
1455
1456
1457
1458
1459
1460
1461
1462
1463
1464
1465
1466
1467
1468
1469
1470
1471
1472
1473
1474
1475
1476
1477
1478
1479
1480
1481
1482
1483
1484
1485
1486
1487
1488
1489
1490
1491
1492
1493
1494
1495
1496
1497
1498
1499
1500
1501
1502
1503
1504
1505
1506
1507
1508
1509
1510
1511
1512
1513
1514
1515
1516
1517
1518
1519
1520
1521
1522
1523
1524
1525
1526
1527
1528
1529
1530
1531
1532
1533
1534
1535
1536
1537
1538
1539
1540
1541
1542
1543
1544
1545
1546
1547
1548
1549
1550
1551
1552
1553
1554
1555
1556
1557
1558
1559
1560
1561
1562
1563
1564
1565
1566
1567
1568
1569
1570
1571
1572
1573
1574
1575
1576
1577
1578
1579
1580
1581
1582
1583
1584
1585
1586
1587
1588
1589
1590
1591
1592
1593
1594
1595
1596
1597
1598
1599
1600
1601
1602
1603
1604
1605
1606
1607
1608
1609
1610
1611
1612
1613
1614
1615
1616
1617
1618
1619
1620
1621
1622
1623
1624
1625
1626
1627
1628
1629
1630
1631
1632
1633
1634
1635
1636
1637
1638
1639
1640
1641
1642
1643
1644
1645
1646
1647
1648
1649
1650
1651
1652
1653
1654
1655
1656
1657
1658
1659
1660
1661
1662
1663
1664
1665
1666
1667
1668
1669
1670
1671
1672
1673
1674
1675
1676
1677
1678
1679
1680
1681
1682
1683
1684
1685
1686
1687
1688
1689
1690
1691
1692
1693
1694
1695
1696
1697
1698
1699
1700
1701
1702
1703
1704
1705
1706
1707
1708
1709
1710
1711
1712
1713
1714
1715
1716
1717
1718
1719
1720
1721
1722
1723
1724
1725
1726
1727
1728
1729
1730
1731
1732
1733
1734
1735
1736
1737
1738
1739
1740
1741
1742
1743
1744
1745
1746
1747
1748
1749
1750
1751
1752
1753
1754
1755
1756
1757
1758
1759
1760
1761
1762
1763
1764
1765
1766
1767
1768
1769
1770
1771
1772
1773
1774
1775
1776
1777
1778
1779
1780
1781
1782
1783
1784
1785
1786
1787
1788
1789
1790
1791
1792
1793
1794
1795
1796
1797
1798
1799
1800
1801
1802
1803
1804
1805
1806
1807
1808
1809
1810
1811
1812
1813
1814
1815
1816
1817
1818
1819
1820
1821
1822
1823
1824
1825
1826
1827
1828
1829
1830
1831
1832
1833
1834
1835
1836
1837
1838
1839
1840
1841
1842
1843
1844
1845
1846
1847
1848
1849
1850
1851
1852
1853
1854
1855
1856
1857
1858
1859
1860
1861
1862
1863
1864
1865
1866
1867
1868
1869
1870
1871
1872
1873
1874
1875
1876
1877
1878
1879
1880
1881
1882
1883
1884
1885
1886
1887
1888
1889
1890
1891
1892
1893
1894
1895
1896
1897
1898
1899
1900
1901
1902
1903
1904
1905
1906
1907
1908
1909
1910
1911
1912
1913
1914
1915
1916
1917
1918
1919
1920
1921
1922
1923
1924
1925
1926
1927
1928
1929
1930
1931
1932
1933
1934
1935
1936
1937
1938
1939
1940
1941
1942
1943
1944
1945
1946
1947
1948
1949
1950
1951
1952
1953
1954
1955
1956
1957
1958
1959
1960
1961
1962
1963
1964
1965
1966
1967
1968
1969
1970
1971
1972
1973
1974
1975
1976
1977
1978
1979
1980
1981
1982
1983
1984
1985
1986
1987
1988
1989
1990
1991
1992
1993
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
2025
2026
2027
2028
2029
2030
2031
2032
2033
2034
2035
2036
2037
2038
2039
2040
2041
2042
2043
2044
2045
2046
2047
2048
2049
2050
2051
class Dataset(Generic[DatasetConfigType, FlagType, FlagIndexType], ABC):
    """Base class of all datasets: one `fetch(flag)` interface for all data.

    A scenario, a scenario merged from several runs or sources, a collection of
    scenarios and a scenario comparison are all datasets.

    A dataset declares its flags with flag specs in `flag_specs` and
    implements `_fetch(flag, effective_config, **params)`. `accepted_flags`
    comes from the specs. A dataset can have attributes (scenario metadata), a
    KPI collection and a cache of fetch results.

    Attributes:
        name (str): Name of the dataset.
        kpi_collection (KPICollection): The KPIs of the dataset.

    Examples:
        A dataset class:

        ```python
        class NodeResults(Dataset):
            flag_specs = (NODE_MODEL_SPEC, NODE_PRICE_SPEC)

            def _fetch(self, flag, effective_config, **params):
                ...
        ```

        A fetch:

        >>> prices = dataset.fetch('Node.Results.price')
    """

    flag_specs: ClassVar[tuple[FlagSpec, ...]] = ()
    """Flag specs of this dataset. `accepted_flags` and the flag index come from them."""

    description: str | None = None
    """Optional free text about the dataset. A platform class or a study sets it, on the class or the instance.
    `describe_datasets()` shows it."""

    _AFTER_CACHE_FIELDS: ClassVar[tuple[str, ...]] = ('time_index_filter', 'granularity')
    """Config fields that the outermost fetch applies after the cache. `_fetch` gets them as `None`."""

    fetch_params: ClassVar[tuple[Dimension, ...]] = ()
    """Dataset parameters: keyword arguments of `fetch` that the dataset class uses itself.

    A class has the dataset parameters of its base classes and adds its own:
    `fetch_params` of a subclass lists only the new ones, and the class
    gets the merged tuple.

    A flag spec declares the parameters of one flag (`FlagSpec.params`).
    `fetch_params` declares the parameters of every fetch of this dataset
    class, for example `comparison_type` of `DatasetComparison`. Both kinds
    resolve the same way: the keyword argument, then `param_defaults` of the
    effective config, then the `default` of the dimension. The resolved value
    goes to `_fetch` as a keyword argument.

    A name may not be both a flag parameter and a dataset parameter of one
    dataset. A collection also accepts the dataset parameters of its
    children. It passes each child the flag parameters of the call and the
    dataset parameters that the child accepts and that the call or the
    `param_defaults` of the collection gave, its own ones included (for
    example `comparison_type` to the comparisons of a comparison of
    comparisons). A value in the call and the same value in `param_defaults`
    reach the child alike. The collection never passes the default of a
    dimension.

    Examples:
        ```python
        class NodePrices(Dataset):
            flag_specs = (NODE_PRICE_SPEC,)
            fetch_params = (Dimension('currency', 'Currency of the prices.', {
                'EUR': 'euro', 'USD': 'US dollar',
            }, default='EUR'),)

            def _fetch(self, flag, effective_config, currency, **params):
                ...

        NodePrices().fetch('Node.Results.price', currency='USD')
        ```

    A subclass may override `fetch` to show its dataset parameters in the
    IDE: one typed keyword-only argument per dataset parameter, with the
    default `UNSET`. The override passes on to `super().fetch` only the
    arguments that are not `UNSET` (`_given`). So a parameter that the call
    leaves out still resolves from `param_defaults`, and `call_kwargs` holds
    only what the caller gave.
    """

    _fetch_param_names_of_all_classes: ClassVar[set[str]] = set()
    """Names of the dataset parameters that any `Dataset` subclass declares.

    A `param_defaults` entry of a class config (`set_class_config`) with one
    of these names never raises as unknown, so a global config can set
    `comparison_type` for all datasets.
    """

    _reads_source: ClassVar[bool] = True
    """`True`: `fetch` raises `ValueError` up front for an unavailable flag. Datasets that read the data
    and platform datasets (with the answer of the interpreter chain). `False` for other collections,
    which fetch only from children on which the flag is not unavailable."""

    _resolves_flag_params: ClassVar[bool] = True
    """`False` for collections: they check flag parameter names and pass the
    explicit flag parameters to their children, which resolve them with their
    own config. A collection always resolves its own `fetch_params`."""

    _warns_about_unknown_config_fields: bool = True
    """Log a warning for config keys that the config type does not define.

    The warning is logged once per config class and key in the process, not
    once per dataset instance. So one undeclared key, fetched across many
    scenarios and interpreters, gives one warning. A key that the config
    class of the dataset or of a parent dataset declares never warns, so a
    key of a platform config does not warn on any interpreter. `False` for
    collections, which pass such keys on to their children."""

    def __init_subclass__(cls, **kwargs):
        super().__init_subclass__(**kwargs)
        if 'fetch_params' in cls.__dict__:
            merged = {d.name: d for base in reversed(cls.__bases__) for d in getattr(base, 'fetch_params', ())}
            merged.update({d.name: d for d in cls.__dict__['fetch_params']})
            cls.fetch_params = tuple(merged.values())
        Dataset._fetch_param_names_of_all_classes.update(d.name for d in cls.fetch_params)

    def __init__(
        self,
        name: str = None,
        parent_dataset: Dataset = None,
        attributes: dict = None,
        cache: Cache = None,
        config: DatasetConfigType = None,
    ):
        """
        Initialize a new Dataset instance.

        Args:
            name: Human-readable identifier. If None, auto-generates from class name
            parent_dataset: Optional parent dataset for hierarchical relationships
            attributes: Dictionary of metadata attributes for the dataset. Keys must be
                of type str, values of type bool, int, float or str.
            cache: Optional cache of fetch results
            config: Configuration object controlling dataset behavior

        Raises:
            TypeError: If an attribute key or value has an invalid type.
            ValueError: A time series spec names a model flag without model spec.
            AmbiguousOverrideError: Two specs give the same flag name.
            ValueError: A name is a flag parameter and a dataset parameter
                (`fetch_params`).
        """
        self.name = name or f'{self.__class__.__name__}_{str(id(self))}'
        self._flag_index: FlagIndex | None = None
        self._parent_dataset = parent_dataset
        self._attributes: dict = dict()
        self._add_attributes(attributes or dict())
        self._cache = cache
        self._config = config
        self._fetch_records: dict[tuple, FetchRecord] = {}
        self._fetched_flags: set[str] = set()
        self._flag_index = self._build_flag_index()
        self._check_param_name_clash()

        from mesqual.kpis.collection import KPICollection

        self.kpi_collection: KPICollection = KPICollection()

    def fetch(
        self,
        flag: FlagType,
        config: dict | DatasetConfigType = None,
        *,
        granularity: str | pd.Timedelta | None | Unset = UNSET,
        **kwargs: Any,
    ) -> pd.Series | pd.DataFrame:
        """Fetch the data of one flag.

        `fetch` takes two kinds of input:

        - `config`: overrides of config fields for this call.
        - `**kwargs`: parameters. Every keyword argument is a parameter. A
          flag spec declares flag parameters (`FlagSpec.params`), for example
          `solution`. The dataset class declares dataset parameters
          (`fetch_params`), for example `comparison_type` of a comparison.

        Each parameter resolves the same way, first match wins:

        1. The keyword argument of this call.
        2. `param_defaults` of the effective config.
        3. The `default` of the dimension.

        `_fetch` gets the resolved parameters as keyword arguments. A
        collection resolves only its own dataset parameters. It passes the
        flag parameters of the call to its children. It gives each child
        the dataset parameters that the child accepts and that the call or
        the `param_defaults` of the collection gave, its own ones included,
        never the default of a dimension (`_kwargs_for_child`).

        Config resolution, later wins: config class defaults, the class
        config of each class in the MRO (base first), the instance config,
        and the `config` of this call. Only the explicit fields of a layer
        override lower layers (the keys of a dict, or the explicit fields of
        a config object), also with a `None` value. `DatasetConfig.CLEAR`
        resets a field to its default.

        Steps:

        1. Raise `UnknownFlagError` if no flag spec declares the flag,
           and `ValueError` if this dataset does not accept it or if
           a dataset that reads the data itself (not a collection)
           has the flag unavailable (`_is_available` is `False`).
           Collections fetch only from children on which the flag is
           not unavailable.
        2. Resolve the effective config.
        3. Resolve the parameters.
        4. Open a frame in the fetch context. Nested fetches and the
           config fields that `_fetch` reads are recorded in it
           (`get_fetch_record`).
        5. Call `_fetch` with the resolved parameters, or read the
           cache, and post-process.
        6. In the outermost call, apply the time index filter to a time
           series flag, then convert it to the `granularity` of the
           effective config.

        Settings applied after the cache (`time_index_filter`,
        `granularity`): they apply only in the outermost `fetch` call, from
        its effective config, and only to time series flags, first the
        filter, then the conversion. `_fetch` gets them as `None`, so a
        cached value is never filtered or converted. Nested fetches
        (collection children, interpreters, layer flags) return unfiltered
        native data: such a setting in the config of a child dataset has no
        effect when the child is fetched through a collection. The fetch
        depth lives in a context variable: run code in a thread started
        inside a fetch with `contextvars.copy_context().run(...)`, else its
        fetches count as outermost.

        The config field `granularity` converts a time series flag to
        another time resolution with `TimeSeriesGranularityConverter` and
        the quantity type of the flag: intensive values are averaged or
        repeated, extensive values summed or split. `None` means the native
        resolution. On a row `MultiIndex` it converts the datetime level per
        combination of the other levels. Model and other flags are returned
        unchanged. The argument `granularity` is a short form of
        `config={'granularity': ...}` in the outermost call. In a nested
        fetch it has no effect, so the call keeps its config as given.

        Args:
            flag: The flag. It must be in `accepted_flags` or match a flag
                family of the dataset.
            config: Config overrides for this call: a dict with config field
                names as keys, or a config object of the config type of the
                dataset. A platform config may add its own fields.
            granularity: Target time resolution of a time series flag, a
                pandas frequency such as `'15min'` or `'1h'`. `None`: the
                native resolution. Left out: the `granularity` of the
                effective config. It overrides the `config` of the call.
                No effect in a nested fetch.
            **kwargs: Parameters: flag parameters of the spec and dataset
                parameters of the dataset class (for a collection, also of
                its children).

        Returns:
            DataFrame or Series with the data of the flag.

        Raises:
            UnknownFlagError: No flag spec declares the flag.
            FlagUnavailableError: The flag is not available
                (`is_flag_available` is `False`). A subclass of `ValueError`.
            ValueError: The dataset does not accept the flag; a keyword
                argument is no declared parameter; a parameter is missing or
                has a value that its dimension does not allow;
                `param_defaults` names an unknown parameter (outermost call
                only). A name in the instance config or in `config` must be
                a flag parameter of the flag index or a dataset parameter of
                this dataset (for a collection, also of its children). A
                name in a class config (`set_class_config`) may also be a
                dataset parameter of any `Dataset` subclass. The
                `granularity` of the effective config is not a frequency
                such as `'15min'`.
            TypeError: A `granularity` conversion of a result whose row
                index has no datetime level.

        Examples:
            A time series in another resolution. The base scenario has hourly
            rows, then quarter-hourly rows. The result has one row per hour:

            >>> hourly = dataset.fetch('Node.Results.price', granularity='1h')
            >>> len(dataset.fetch('Node.Results.price')), len(hourly)
            (332, 119)
        """
        is_outermost = FetchContext.is_outermost()
        enclosing = FetchContext.current()
        with FetchContext.internal():
            if granularity is not UNSET and is_outermost:
                config = self._with_granularity(config, granularity)
            spec = self._spec_of_accepted_flag(flag)
            if self._reads_source:
                self._raise_if_unavailable(flag)
            if is_outermost:
                self.flag_index.check_references()
            if is_outermost:
                self._check_param_default_names(config)
            record, effective_config, params = self._start_call(
                spec,
                flag,
                config,
                self._call_config(config, enclosing),
                kwargs,
            )
            if is_outermost:
                self._check_granularity(effective_config.granularity)
            self._warn_about_unknown_config_fields(effective_config)
        shapes_output = is_outermost and spec.item_type is ItemTypeEnum.TimeSeries
        target_granularity = effective_config.granularity if shapes_output else None
        try:
            with FetchContext.frame(record, self) as frame:
                self._check_dataset_values(spec, flag, params)
                data = self._fetch_from_cache_or_source(flag, effective_config, frame, **params)
            self._fetched_flags.add(str(flag))
        finally:
            self._fetch_records[record.key] = frame.to_record()
        if shapes_output:
            data = self._apply_time_index_filter(data, effective_config)
            data = self._convert_granularity(data, flag, target_granularity)
        return data

    def get_fetch_record(
        self,
        flag: FlagType,
        config: dict | DatasetConfigType = None,
        *,
        granularity: str | pd.Timedelta | None | Unset = UNSET,
        **kwargs: Any,
    ) -> FetchRecord | None:
        """Return the record of the last fetch with the same flag, keyword arguments, config and resolved parameters.

        A fetch that raised also leaves a record, with `raised=True`.

        The record lists the fetches that the call made directly, each with
        its own record, and the config fields that `_fetch` read. The lookup
        uses the record key (`FetchRecord.key`): the keyword arguments and
        the call config as given, and the parameters, resolved as in `fetch`.

        Args:
            flag: The flag.
            config: Fetch-time config, as in `fetch`. `INHERIT` finds a call
                that got the effective config of its enclosing fetch; the
                parameters then resolve with the config of this dataset. So
                a child record whose parameters came from the `param_defaults`
                of the parent config is not found here. Reach it through the
                `dependencies` of the parent record.
            granularity: Applies as in an outermost fetch. To look up a
                nested call, leave it out. No effect with `config=INHERIT`.
            **kwargs: Parameters, as in `fetch`.

        Returns:
            The record, or `None` if the flag was not fetched with these arguments.

        Examples:
            >>> high = study.scen.get_dataset('high_demand')
            >>> prices = high.fetch('Node.Results.price', solution='first')
            >>> high.get_fetch_record('Node.Results.price', solution='first').call_kwargs
            {'solution': 'first'}
            >>> high.get_fetch_record('Node.Results.price') is None
            True
        """
        with FetchContext.internal():
            if config is INHERIT:
                layer, call_config = None, INHERIT
            else:
                layer = config if granularity is UNSET else self._with_granularity(config, granularity)
                call_config = self._call_config(layer, None)
            spec = self.flag_index.get_spec(flag)
            record, *_ = self._start_call(spec, flag, layer, call_config, kwargs)
        return self._fetch_records.get(record.key)

    def _start_call(
        self,
        spec: FlagSpec,
        flag: FlagType,
        config: dict | DatasetConfigType | None,
        call_config: Any,
        call_kwargs: dict[str, Any],
    ) -> tuple[FetchRecord, DatasetConfigType, dict[str, Any]]:
        """Resolve the config and the parameters of a call and build its record, before its frame opens.

        `fetch`, `get_fetch_record` and `_rebuild_call` use it, so a stored
        call and its rebuild resolve the same way.

        Args:
            spec: The spec of the flag.
            flag: The flag.
            config: The fetch-time config layer of the call.
            call_config: `FetchRecord.call_config` of the record.
            call_kwargs: The keyword arguments of the call, as given.

        Returns:
            The record, the effective config and the resolved parameters.
        """
        effective_config = self._prepare_config(config)
        params, param_default_reads = self._resolve_params(spec, flag, call_kwargs, effective_config)
        record = self._new_record(flag, call_config, call_kwargs, params, param_default_reads)
        return record, effective_config, params

    def version_identity(self, flag: FlagType) -> tuple:
        """Return the version identity of a flag: one entry per link.

        A dataset gives `((type(self).__qualname__, spec version),)`. A
        platform dataset gives one pair per link of the interpreter chain of
        the flag. The answering interpreter of a platform dataset gives the
        identity of its platform dataset, so a version change of an override
        changes the records of the override and of everything that calls it.
        A collection gives one `(class path, name)` pair per child that it can
        read the flag from; its children record their own identities. A concat collection
        adds its layout first (`DatasetConcatCollection.version_identity`).
        """
        parent = self._parent_dataset
        if parent is not None and parent._answering_instance(flag) is self:
            return parent.version_identity(flag)
        return ((type(self).__qualname__, self.flag_index.get_spec(flag).version),)

    def _class_path(self) -> str:
        return f'{type(self).__module__}.{type(self).__qualname__}'

    @staticmethod
    def _call_config(config: dict | DatasetConfigType | None, enclosing: FetchFrame | None) -> Any:
        if config is None:
            return {}
        if enclosing is not None and config is enclosing.watched_config:
            return INHERIT
        from mesqual.datasets.dataset_config import DatasetConfig

        return dict(DatasetConfig._explicit_items(config))

    def _new_record(
        self,
        flag: FlagType,
        call_config: Any,
        call_kwargs: dict[str, Any],
        params: dict[str, Any],
        param_default_reads: dict[str, Any] | None = None,
    ) -> FetchRecord:
        return FetchRecord(
            dataset_class=self._class_path(),
            dataset_name=self.name,
            flag=str(flag),
            call_kwargs=snapshot(dict(call_kwargs)),
            call_config=call_config if call_config is INHERIT else snapshot(call_config),
            params=snapshot(params),
            version=self.version_identity(flag),
            param_default_reads=snapshot(param_default_reads or {}),
        )

    @property
    def _flag_param_names(self) -> set[str]:
        """Names of the flag parameters of the own specs: of the own flag index, else of `flag_specs`.

        A dataset with a parent has no index of its own, so the check reads
        its `flag_specs`, not the index of the parent.
        """
        if self._flag_index is not None:
            return self._flag_index.parameter_names
        return {d.name for spec in self.flag_specs for d in spec.params}

    @property
    def _accepted_dataset_params(self) -> dict[str, Dimension]:
        """The dataset parameters that `fetch` accepts, by name: the `fetch_params` of this class."""
        return {d.name: d for d in self.fetch_params}

    def _check_param_name_clash(self) -> None:
        """Raise `ValueError` if a name is a flag parameter and a dataset parameter of this dataset."""
        self._raise_on_param_name_clash(self._flag_param_names, set(self._accepted_dataset_params))

    def _raise_on_param_name_clash(self, flag_names: set[str], dataset_names: set[str]) -> None:
        """Raise `ValueError` if a name is in `flag_names` and in `dataset_names`."""
        clash = flag_names & dataset_names
        if clash:
            raise ValueError(
                f'Dataset {self.name}: {sorted(clash)} are flag parameters (FlagSpec.params) and dataset '
                f'parameters (fetch_params). A parameter name must have one owner. Rename one of them.'
            )

    @staticmethod
    def _given(**kwargs) -> dict[str, Any]:
        """`kwargs` without the `UNSET` values: the arguments that a typed `fetch` override got from the caller."""
        return {name: value for name, value in kwargs.items() if value is not UNSET}

    def _kwargs_for(self, flag: FlagType, kwargs: dict[str, Any]) -> dict[str, Any]:
        """The parameters of `kwargs` that the spec of `flag` or this dataset declares."""
        declared = {d.name for d in self.flag_index.get_spec(flag).params} | set(self._accepted_dataset_params)
        return {k: v for k, v in kwargs.items() if k in declared}

    def _spec_of_accepted_flag(self, flag: FlagType) -> FlagSpec:
        spec = self.flag_index.get_spec(flag)
        if not self.flag_is_accepted(flag):
            raise ValueError(f'Flag {flag} not accepted by Dataset "{self.name}" of type {type(self)}.')
        return spec

    def _answering_instance(self, flag: FlagType) -> Dataset | None:
        """The child that answers `flag`, or `None` for a dataset without children that answer flags."""
        return None

    def _check_param_default_names(self, config: dict | DatasetConfigType | None) -> None:
        """Raise `ValueError` for a `param_defaults` name that no parameter of this dataset declares.

        Names of the instance config and of the call `config` must be flag
        parameters of the flag index or dataset parameters of this dataset
        (for a collection, also of its children). Names of a class config
        may also be dataset parameters of any `Dataset` subclass, so one
        class config can serve all datasets.
        """
        from mesqual.datasets.dataset_config import DatasetConfigManager

        own = self.flag_index.parameter_names | set(self._accepted_dataset_params)
        class_layers = [layer for _, layer in DatasetConfigManager.get_layers(type(self))[1:]]
        strict_names = set().union(*map(self._param_default_names, (self._config, config)))
        loose_names = set().union(*map(self._param_default_names, class_layers))
        unknown = (strict_names - own) | (loose_names - own - self._fetch_param_names_of_all_classes)
        if unknown:
            raise ValueError(
                f'Dataset {self.name}: param_defaults names parameters that this dataset does not declare: '
                f'{sorted(unknown)}. Flag parameters: {sorted(self.flag_index.parameter_names)}. '
                f'Dataset parameters: {sorted(self._accepted_dataset_params)}.'
            )

    @staticmethod
    def _param_default_names(config: dict | DatasetConfigType | None) -> set[str]:
        """Names in the explicit `param_defaults` of one config layer."""
        from mesqual.datasets.dataset_config import DatasetConfig

        param_defaults = dict(DatasetConfig._explicit_items(config)).get('param_defaults')
        return set(param_defaults) if isinstance(param_defaults, dict) else set()

    def _params_to_resolve(self, spec: FlagSpec) -> tuple[Dimension, ...]:
        """The parameters that this dataset resolves: the flag parameters (unless a collection) and `fetch_params`.

        Raises:
            ValueError: A flag parameter of `spec` has the name of a dataset
                parameter, for example of a spec added to the flag index
                after construction.
        """
        clash = {d.name for d in spec.params} & {d.name for d in self.fetch_params}
        if clash:
            raise ValueError(
                f'Spec {spec.name!r}: {sorted(clash)} are flag parameters of the spec and dataset parameters '
                f'(fetch_params) of {type(self).__name__}. A parameter name must have one owner. Rename one of them.'
            )
        flag_params = spec.params if self._resolves_flag_params else ()
        return (*flag_params, *self.fetch_params)

    def _resolve_params(
        self,
        spec: FlagSpec,
        flag: FlagType,
        call_kwargs: dict[str, Any],
        effective_config: DatasetConfigType,
    ) -> tuple[dict[str, Any], dict[str, Any]]:
        """Resolve the parameters of a call.

        Each parameter that this dataset resolves gets the keyword argument,
        else the `param_defaults` entry, else the default of its dimension
        (`Dimension.resolve`). The values of dimensions that are resolved per
        dataset are checked later, in the frame of the fetch.
        The other keyword arguments pass through unchanged.

        Returns:
            The resolved parameters, and the `param_defaults` entries that
                the resolution used.
        """
        accepted = {d.name for d in spec.params} | set(self._accepted_dataset_params)
        unknown = set(call_kwargs) - accepted
        if unknown:
            raise ValueError(
                f'Flag {flag}: no flag spec or dataset class declares the parameters {sorted(unknown)}. '
                f'Declared: {sorted(accepted)}.'
            )
        defaults = effective_config.param_defaults or {}
        dimensions = self._params_to_resolve(spec)
        reads = {d.name: defaults[d.name] for d in dimensions if d.name not in call_kwargs and d.name in defaults}
        given = {**reads, **call_kwargs}
        params = {}
        for dimension in dimensions:
            if not dimension.has_default and dimension.name not in given:
                raise ValueError(
                    f'Flag {flag}: parameter {dimension.name!r} is required. Pass it to fetch or set param_defaults.'
                )
            try:
                params[dimension.name] = dimension.resolve(given)
            except ValueError as error:
                raise ValueError(f'Flag {flag}: {error}') from error
        params.update({name: value for name, value in call_kwargs.items() if name not in params})
        return params, reads

    def _check_dataset_values(self, spec: FlagSpec, flag: FlagType, params: dict[str, Any]) -> None:
        """Dataset-specific parameter checks: the variant and parameter values whose allowed values come from the dataset.

        A subclass adds its own checks, for example of its dataset
        parameters. It runs inside the frame of the fetch, so the fetches of the values
        functions are recorded as dependencies. Each dataset in a chain of
        routed calls checks the values again with the root dataset.
        """
        root = self._root_dataset()
        try:
            if spec.is_family and self._resolves_flag_params:
                spec.check_variant_values(flag, root)
            for dimension in self._params_to_resolve(spec):
                if dimension.is_resolved_per_dataset:
                    dimension.check_value(params[dimension.name], root)
        except FlagUnavailableError as error:
            raise FlagUnavailableError(str(flag), self.name, error.reason) from error
        except ValueError as error:
            raise ValueError(f'Flag {flag}: {error}') from error

    def _root_dataset(self) -> Dataset:
        dataset = self
        while dataset._parent_dataset is not None:
            dataset = dataset._parent_dataset
        return dataset

    def _fetch_from_cache_or_source(
        self, flag: FlagType, effective_config: DatasetConfigType, frame: FetchFrame, **kwargs
    ) -> pd.Series | pd.DataFrame:
        """Return the cached value if the entry of the cache key is up to date, else call `_fetch` and store the result.

        `_fetch` gets `_cache_config(effective_config)`, so the settings that
        apply after the cache never reach it. An entry is stored only if its
        record is up to date right after the fetch, so the cache never
        holds an entry that no later fetch can use. If the entry is not
        stored, or `Cache.set` raises, the fetch still returns the value
        and logs the reason at INFO, once per dataset class and flag. A stale
        entry logs its reason at DEBUG.
        """
        use_cache = self._cache is not None and effective_config.use_cache
        frame.stores = use_cache
        cache_config = self._cache_config(effective_config)
        if use_cache:
            key = CacheKey.of(frame.record)
            check = DependencyCheck(self, frame.record, cache_config)
            entry = self._up_to_date_entry(flag, key, check)
            if entry is not None:
                stored, value = entry
                frame.add_stored(stored)
                return value.copy()

        watched_config = FetchContext.watch(cache_config, frame)
        raw_data = self._fetch(flag, watched_config, **kwargs)
        processed_data = self._post_process_data(raw_data, flag, watched_config)

        if use_cache:
            record = frame.to_record()
            reason = check.stale_reason(record)
            if reason is None and not self._can_snapshot(record):
                reason = 'a recorded value cannot be deep-copied'
            if reason is None:
                try:
                    self._cache.set(key, processed_data, record)
                except Exception as error:
                    reason = f'the cache cannot store it ({type(error).__name__}: {error})'
            if reason is not None:
                _NotStoredLogs.log(self, flag, reason)

        return processed_data.copy()

    def _up_to_date_entry(self, flag: FlagType, key: CacheKey, check: DependencyCheck) -> Entry | None:
        """The entry of `key` if `check` finds it up to date, else `None`.

        A stale entry logs its reason at DEBUG. An exception of
        `Cache.entry` logs a warning and counts as a miss.
        """
        try:
            entry = self._cache.entry(key)
        except Exception as error:
            logger.warning(f'Flag {flag}: the cache cannot read the entry ({type(error).__name__}: {error}).')
            return None
        if entry is None:
            return None
        reason = check.stale_reason(entry[0])
        if reason is not None:
            logger.debug(f'Flag {flag}: cache entry is stale: {reason}')
            return None
        return entry

    @classmethod
    def _cache_config(cls, effective_config: DatasetConfigType) -> DatasetConfigType:
        """The effective config without the settings that apply after the cache. `_fetch` gets this config.

        The fields in `_AFTER_CACHE_FIELDS` (`time_index_filter` and
        `granularity`) are `None`. A read of the whole config records them
        as `None`, so neither setting makes an entry stale.
        """
        cache_config = copy.copy(effective_config)
        for name in cls._AFTER_CACHE_FIELDS:
            object.__setattr__(cache_config, name, None)
        return cache_config

    @staticmethod
    def _can_snapshot(record: FetchRecord) -> bool:
        """`True` if the record can be deep-copied. A value that `snapshot` kept as is cannot be checked later."""
        try:
            copy.deepcopy(record)
            return True
        except Exception:
            return False

    def _rebuild_call(
        self, stored: FetchRecord, parent_config: DatasetConfigType
    ) -> tuple[FetchRecord, DatasetConfigType]:
        """Rebuild the call of a stored dependency record with the current configs, for `DependencyCheck`.

        The config of the call is the current config of this dataset, then
        the recorded `call_config`, or `parent_config` for `INHERIT`. The
        parameters resolve from the recorded `call_kwargs` with that config.

        Returns:
            The record of the call as it would start now, and the config
                that its `_fetch` would get.
        """
        with FetchContext.internal():
            config = parent_config if stored.call_config is INHERIT else stored.call_config
            spec = self.flag_index.get_spec(stored.flag)
            record, effective_config, *_ = self._start_call(
                spec,
                stored.flag,
                config,
                stored.call_config,
                dict(stored.call_kwargs),
            )
            return record, self._cache_config(effective_config)

    def _child_datasets(self) -> tuple[Dataset, ...]:
        """All datasets that this dataset holds and fetches from: for a collection its children, also the interpreters of a platform dataset.

        Use it to walk the whole dataset tree, as `DependencyCheck` does.
        `_describe_children` differs: it hides the interpreters of a
        platform dataset, because the describe methods show a platform
        dataset as one dataset.
        """
        return ()

    @staticmethod
    def _with_granularity(config: dict | DatasetConfigType | None, granularity: Any) -> dict:
        """The fetch-time config `config` with `granularity` on top, for the `granularity` argument of `fetch`."""
        from mesqual.datasets.dataset_config import DatasetConfig

        return {**dict(DatasetConfig._explicit_items(config)), 'granularity': granularity}

    @staticmethod
    def _check_granularity(granularity: Any) -> None:
        """Raise `ValueError` if `granularity` is not `None` and not a positive fixed frequency (`parse_frequency`)."""
        if granularity is None:
            return
        from mesqual.energy_data_handling.granularity_converter import TimeSeriesGranularityConverter

        try:
            TimeSeriesGranularityConverter.parse_frequency(granularity)
        except ValueError as error:
            raise ValueError(f'granularity: {error}') from error

    def _convert_granularity(
        self,
        data: pd.Series | pd.DataFrame,
        flag: FlagType,
        granularity: str | pd.Timedelta | None,
    ) -> pd.Series | pd.DataFrame:
        """Convert `data` to `granularity` with the quantity type of the flag. A row `MultiIndex` converts on its time level."""
        if granularity is None:
            return data
        from mesqual.energy_data_handling.granularity_converter import TimeSeriesGranularityConverter

        time_level = self._time_level_of(data) if isinstance(data.index, pd.MultiIndex) else None
        quantity_type = self.flag_index.get_quantity_type_enum(flag)
        return TimeSeriesGranularityConverter().convert(data, granularity, quantity_type, time_level=time_level)

    def _apply_time_index_filter(
        self, data: pd.Series | pd.DataFrame, effective_config: DatasetConfigType
    ) -> pd.Series | pd.DataFrame:
        filters = effective_config.time_index_filter
        if filters is None:
            return data
        if isinstance(filters, TimeIndexFilter):
            filters = (filters,)
        time_level = self._time_level_of(data)
        for time_index_filter in filters:
            data = time_index_filter.apply(data, time_level)
        return data

    @staticmethod
    def _time_level_of(data: pd.Series | pd.DataFrame) -> int | None:
        """Position of the time level of a result, used by a time index filter without `level`.

        The time level is the first `DatetimeIndex` level of the row index.
        A result without such a level has no time level.
        """
        index = data.index
        if isinstance(index, pd.MultiIndex):
            for position, level in enumerate(index.levels):
                if isinstance(level, pd.DatetimeIndex):
                    return position
            return None
        return 0 if isinstance(index, pd.DatetimeIndex) else None

    def _warn_about_unknown_config_fields(self, effective_config: DatasetConfigType) -> None:
        if self._warns_about_unknown_config_fields:
            _UnknownConfigFieldWarnings.warn(self.name, effective_config, self._config_fields_of_parents())

    def _config_fields_of_parents(self) -> frozenset[str]:
        parent = self._parent_dataset
        if parent is None:
            return frozenset()
        return parent.get_config_type()._field_names() | parent._config_fields_of_parents()

    @property
    def accepted_flags(self) -> set[FlagType]:
        """
        Set of the flags of the specs of this dataset.

        It holds concrete flag names, computed from the declarations in
        `flag_specs` without opening source data. Flags of a flag family are
        not listed, because their values come from the data of the dataset.
        `flag_is_accepted` accepts them by their name template. Whether a
        source has the data of a flag is a separate question (availability).

        A subclass may narrow the spec flags using declarations of siblings
        or attributes, never source data.

        Returns:
            Set of flags that can be used with the fetch() method

        Examples:
            >>> 'Line.Results.flow_sent_net' in dataset.accepted_flags
            True
        """
        return {flag for spec in self.flag_specs if not spec.is_family for flag in spec.expand()}

    def get_accepted_flags_containing_x(self, x: str, match_case: bool = False) -> set[FlagType]:
        """
        Find all accepted flags containing a specific substring.

        Useful for discovering related data flags or filtering flags by category.

        Args:
            x: Substring to search for in flag names
            match_case: If True, performs case-sensitive search. Default is False.

        Returns:
            Set of accepted flags containing the substring

        Examples:
            >>> sorted(dataset.get_accepted_flags_containing_x('line.results.flow_sent'))
            ['Line.Results.flow_sent_down', 'Line.Results.flow_sent_net', 'Line.Results.flow_sent_up']
            >>> dataset.get_accepted_flags_containing_x('line.results', match_case=True)
            set()
        """
        return {f for f in self.accepted_flags if FlagDescriber.name_contains(f, x, match_case)}

    def flag_is_accepted(self, flag: FlagType) -> bool:
        """
        Boolean check whether a flag is accepted by the Dataset.

        `True` for a flag in `accepted_flags` and for a flag of a flag family
        spec of this dataset. Uses only declarations, never source data.
        """
        if flag in self.accepted_flags:
            return True
        return any(spec.is_family and spec.covers(flag) for spec in self.flag_specs)

    @property
    def flag_index(self) -> FlagIndex:
        """
        The flag index: lookup from a flag to its spec.

        Built from `flag_specs` at construction. A dataset constructed with a
        parent dataset, for example an interpreter, uses the index of its
        parent. Collections and platform datasets document their own rule.
        """
        if self._flag_index is None and self._parent_dataset is not None:
            return self._parent_dataset.flag_index
        return self._flag_index

    def _build_flag_index(self) -> FlagIndex | None:
        """Return an index of `flag_specs`, or `None` for a dataset with a parent: it uses the index of the parent."""
        if self._parent_dataset is not None:
            return None
        return FlagIndex(self.flag_specs)

    def _result_schema(self, flag: FlagType) -> FrameSchema:
        """The schema of the result of a fetch of `flag`. A collection transforms the schema of its children."""
        return self.flag_index.get_schema(flag)

    @property
    def cache(self) -> Cache | None:
        """The cache of fetch results of this dataset, or `None`.

        `fetch` uses it when `use_cache` is set (see
        [Cache](../architecture/other-contracts.md#cache)).
        """
        return self._cache

    def add_kpis_from_definitions(self, kpi_definitions: KPIDefinition | list[KPIDefinition]):
        """
        Generate and add KPIs from one or more KPI definitions.

        KPI definitions are templates that generate concrete KPI instances
        based on the dataset's structure. This method processes definitions
        and adds the resulting KPIs to the dataset's KPI collection.

        Args:
            kpi_definitions: Single KPIDefinition or list of definitions.
                Each definition's generate_kpis() method is called with
                this dataset to produce KPI instances.

        Examples:
            >>> from mesqual.kpis import Aggregations, FlagAggKPIBuilder
            >>> definitions = FlagAggKPIBuilder().for_flag('Country.Results.price').with_aggregation(Aggregations.Mean).build()
            >>> dataset.add_kpis_from_definitions(definitions)
            >>> len(dataset.kpi_collection)
            2

        See Also:
            - [`add_kpi()`][mesqual.datasets.Dataset.add_kpi]: add one KPI.
            - [`add_kpis()`][mesqual.datasets.Dataset.add_kpis]: add several KPIs.
        """
        from mesqual.kpis.definitions.base import KPIDefinition

        if isinstance(kpi_definitions, KPIDefinition):
            kpis = kpi_definitions.generate_kpis(self)
            self.add_kpis(kpis)
        else:
            for kpi_def in kpi_definitions:
                kpis = kpi_def.generate_kpis(self)
                self.add_kpis(kpis)

    def add_kpis(self, kpis: Iterable[KPI]):
        """
        Add multiple KPIs to this dataset's KPI collection.

        Args:
            kpis: Iterable of KPI instances, factories, or classes to add
        """
        duplicates = []
        for kpi in kpis:
            if kpi in self.kpi_collection:
                duplicates.append(kpi)
            else:
                self.add_kpi(kpi)
        if duplicates:
            _num_duplicates = len(duplicates)
            logger.warning(
                f'{_num_duplicates} duplicates found and not added again or overwritten in {self.name}. ({duplicates[:3]}...)'
            )

    def add_kpi(self, kpi: KPI):
        """
        Add a single KPI to this dataset's KPI collection.

        Args:
            kpi: KPI instance, factory, or class to add
        """
        self.kpi_collection.add(kpi)

    def clear_kpi_collection(self):
        """Clear the KPI collection."""
        from mesqual.kpis.collection import KPICollection

        self.kpi_collection = KPICollection()

    @property
    def attributes(self) -> dict:
        """
        Access the metadata attributes dictionary for this dataset.

        Attributes store scenario-level metadata such as configuration parameters,
        simulation settings, or descriptive labels. These are useful for filtering,
        grouping, and annotating datasets in collections.

        Returns:
            dict: Dictionary of attribute key-value pairs. Keys are strings,
                values are primitive types (bool, int, float, str).

        Examples:
            >>> dataset.attributes
            {'first_date': '2024-03-28', 'last_date': '2024-04-01', 'days': 5, 'demand_factor': 1.0, 'periods': 2}

        See Also:
            - [`set_attributes()`][mesqual.datasets.Dataset.set_attributes] - Set attribute values
            - [`get_attributes_series()`][mesqual.datasets.Dataset.get_attributes_series] - Convert to pandas Series
        """
        return self._attributes

    def get_attributes_series(self) -> pd.Series:
        """
        Convert dataset attributes to a pandas Series.

        Creates a Series with attribute names as the index and attribute
        values as data. The Series name is set to the dataset name, making
        it suitable for concatenation with other datasets' attribute series.

        Returns:
            pd.Series: Series containing attribute values, indexed by attribute
                names, with the dataset name as the Series name

        Examples:
            >>> series = dataset.get_attributes_series()
            >>> series.name, series['days']
            ('base', 5)

        See Also:
            - [`attributes`][mesqual.datasets.Dataset.attributes] - Access raw attributes dictionary
            - [`set_attributes()`][mesqual.datasets.Dataset.set_attributes] - Set attribute values
        """
        att_series = pd.Series(self.attributes, name=self.name)
        return att_series

    def set_attributes(self, **kwargs: bool | int | float | str) -> None:
        """
        Set one or more metadata attributes on this dataset.

        Attributes are key-value pairs that store scenario metadata. They must
        use string keys and primitive values (bool, int, float, str) to ensure
        serializability and consistent comparison behavior.

        Args:
            **kwargs: Attribute key-value pairs to set. Keys must be strings,
                values must be bool, int, float, or str.

        Raises:
            TypeError: If any key is not a string
            TypeError: If any value is not bool, int, float, or str

        Examples:
            >>> dataset.set_attributes(year=2030, includes_nuclear=True)
            >>> dataset.attributes['year']
            2030

        See Also:
            - [`attributes`][mesqual.datasets.Dataset.attributes] - Access attributes dictionary
            - [`get_attributes_series()`][mesqual.datasets.Dataset.get_attributes_series] - Convert to pandas Series
        """
        self._add_attributes(kwargs)

    def _add_attributes(self, attributes: dict) -> None:
        for key, value in attributes.items():
            if not isinstance(key, str):
                raise TypeError(f'Attribute keys must be of type str. Your key {key} is of type {type(key)}.')
            if not isinstance(value, (bool, int, float, str)):
                raise TypeError(
                    f'Attribute values must be of type (bool, int, float, str). '
                    f'Your value for {key} ({value}) is of type {type(value)}.'
                )
            self._attributes[key] = value

    @property
    def parent_dataset(self) -> 'DatasetLinkCollection':
        """The parent dataset of an interpreter: the platform dataset that holds it and its sibling interpreters.

        An interpreter answers a part of the flags of a platform dataset. It
        fetches the flags of its siblings through the parent, so it needs no
        reference to them. A fetch through the parent reaches an override of the
        sibling interpreter. The parent is a `DatasetLinkCollection`, usually a
        `PlatformDataset`.

        Returns:
            DatasetLinkCollection: The parent collection.

        Raises:
            RuntimeError: The parent is not set, for example for an interpreter
                used without a parent dataset.

        Examples:
            An interpreter that sums the generation per node from two flags of its siblings:

            ```python
            class NodeGeneration(MyInterpreter):
                flag_specs = (NODE_GENERATION_SPEC,)

                def _fetch(self, flag, effective_config, **params):
                    generation = self.parent_dataset.fetch('Generator.Results.generation', effective_config)
                    node_of_generator = self.parent_dataset.fetch('Generator.Model', effective_config)['node']
                    return generation.T.groupby(node_of_generator).sum().T
            ```

        See Also:
            - [`DatasetLinkCollection.get_dataset_by_type()`][mesqual.datasets.DatasetLinkCollection.get_dataset_by_type]:
              a sibling interpreter by its class.
        """
        if self._parent_dataset is None:
            raise RuntimeError(f'Parent dataset called without / before assignment.')
        return self._parent_dataset

    @parent_dataset.setter
    def parent_dataset(self, parent_dataset: 'DatasetLinkCollection'):
        """
        Set the parent collection for this dataset.

        Args:
            parent_dataset: The DatasetLinkCollection that will contain this dataset

        Raises:
            TypeError: If parent_dataset is not a DatasetLinkCollection instance
        """
        from mesqual.datasets.dataset_collection import DatasetLinkCollection

        if not isinstance(parent_dataset, DatasetLinkCollection):
            raise TypeError(f'Parent parent_dataset must be of type {DatasetLinkCollection.__name__}')
        self._parent_dataset = parent_dataset

    def _post_process_data(
        self, data: pd.Series | pd.DataFrame, flag: FlagType, config: DatasetConfigType
    ) -> pd.Series | pd.DataFrame:
        """Apply the standard post-processing to the result of `_fetch`.

        Handles duplicate index entries and sorts the row index if
        `auto_sort_index` is set. A model flag with duplicate ids raises,
        independent of the config. For other flags, `keep_duplicate_indices`
        decides which rows stay.

        Args:
            data: Raw data from `_fetch`.
            flag: The fetched flag.
            config: Effective config of the fetch.

        Returns:
            The post-processed data.

        Raises:
            ValueError: A model flag has duplicate ids, or
                `keep_duplicate_indices` is not `'first'`, `'last'` or `'all'`.

        Note:
            `fetch` calls this method. A subclass can override it to add
            post-processing and call `super()`.
        """
        if self.flag_index.get_item_type(flag) is ItemTypeEnum.Model:
            self._raise_on_duplicate_ids(data, flag)
        else:
            data = self._drop_duplicate_indices(data, flag, config.keep_duplicate_indices)
        if config.auto_sort_index:
            data = self._sort_index(data)
        return data

    def _sort_index(self, data: pd.Series | pd.DataFrame) -> pd.Series | pd.DataFrame:
        """Sort the row index of a result. `auto_sort_index` turns it on."""
        return data.sort_index()

    def _raise_on_duplicate_ids(self, data: pd.Series | pd.DataFrame, flag: FlagType) -> None:
        duplicated = data.index[data.index.duplicated()].unique()
        if len(duplicated):
            raise ValueError(
                f'Dataset {self.name}: model flag {flag} has duplicate ids {list(duplicated[:5])}. '
                f'A model flag has one row per object.'
            )

    def _drop_duplicate_indices(
        self,
        data: pd.Series | pd.DataFrame,
        flag: FlagType,
        keep: str,
    ) -> pd.Series | pd.DataFrame:
        if keep == 'all':
            return data
        if keep not in ('first', 'last'):
            raise ValueError(f"keep_duplicate_indices must be 'first', 'last' or 'all', got {keep!r}.")
        duplicated = data.index.duplicated(keep=keep)
        if duplicated.any():
            _DuplicateIndexWarnings.warn(self, flag, int(duplicated.sum()), keep)
            data = data.loc[~duplicated]
        return data

    def _prepare_config(self, config: dict | DatasetConfigType = None) -> DatasetConfigType:
        """
        Prepare the effective configuration for a fetch operation.

        Resolves the final configuration by merging the provided config override
        with the dataset's instance config (which already includes class-level
        and base defaults via DatasetConfigManager).

        Args:
            config: Optional override configuration. Can be:
                - None: Use instance config as-is
                - dict: Merge its keys
                - DatasetConfig: Merge directly with instance config

        Returns:
            The fully resolved configuration for the fetch operation.

        Raises:
            TypeError: If config is neither None, dict, nor DatasetConfig.
        """
        if config is None:
            return self.instance_config
        from mesqual.datasets.dataset_config import DatasetConfig

        if isinstance(config, (dict, DatasetConfig)):
            return self.instance_config.merge(config)
        raise TypeError(f'Config must be dict or {DatasetConfig.__name__}, got {type(config)}')

    @abstractmethod
    def _fetch(self, flag: FlagType, effective_config: DatasetConfigType, **kwargs) -> pd.Series | pd.DataFrame:
        """
        Abstract method implementing the actual data retrieval logic.

        Subclasses must implement this method to define how data is retrieved
        for each flag. This is the core data access method that fetch() calls
        after configuration resolution and before post-processing.

        Args:
            flag: The validated flag to fetch data for
            effective_config: The fully resolved configuration for this operation
            **kwargs: Additional implementation-specific arguments

        Returns:
            DataFrame or Series containing the requested data. The returned
                data will be post-processed by _post_process_data() before being
                returned to the caller.

        Note:
            - This method should not perform flag validation (handled by fetch())
            - This method should not apply post-processing (handled separately)
            - This method should not handle caching (handled by fetch())
        """
        return pd.DataFrame()

    def fetch_multiple_flags_and_concat(
        self,
        flags: Iterable[FlagType],
        flag_concat_axis: int = 1,
        concat_level_name: str = 'variable',
        concat_level_at_top: bool = True,
        config: dict | DatasetConfigType = None,
        *,
        granularity: str | pd.Timedelta | None | Unset = UNSET,
        **kwargs: Any,
    ) -> Union[pd.Series, pd.DataFrame]:
        """
        Fetch multiple flags and concatenate results into a single DataFrame.

        Convenience method for retrieving data from multiple flags and combining
        them into a single DataFrame with a MultiIndex. Useful for comparative
        analysis of multiple variables or creating wide-format data structures.

        All results must have the same shape of axes. Before the concat, each
        result is checked against the first one: both are a `Series` or both
        are a `DataFrame`, and each axis has the same number of levels and the
        same index type per level (for example `DatetimeIndex`). A `Series`
        has only its index to check.

        Args:
            flags: Iterable of flags to fetch and concatenate
            flag_concat_axis: Axis along which to concatenate (0=rows,
                1=columns). Default is 1 (columns). It is not the
                `concat_axis` of a concat collection, which is fixed when
                the collection is built.
            concat_level_name: Name for the new MultiIndex level identifying
                the source flag. Default is 'variable'.
            concat_level_at_top: If True, the flag level is the outermost level
                in the MultiIndex. If False, it's moved to the innermost level.
                Default is True.
            config: Optional configuration override (see fetch() for details)
            granularity: Target time resolution, as in `fetch`.
            **kwargs: Parameters. Each fetch gets the flag parameters that
                its spec declares, and the dataset parameters of this dataset.

        Returns:
            DataFrame with concatenated data and a MultiIndex identifying the
                source flag for each section

        Raises:
            ValueError: If two results differ in kind, number of levels or index
                type on an axis. The message names both flags.
            ValueError: A keyword argument is no flag parameter of any of the
                flags and no dataset parameter of this dataset.

        Examples:
            >>> demand_and_gas = dataset.fetch_multiple_flags_and_concat(
            ...     ['Node.Results.demand_total', 'Node.Results.generation_gas'], concat_level_name='variable',
            ... )
            >>> list(demand_and_gas.columns.names)
            ['variable', 'Node']

        See Also:
            - [`fetch()`][mesqual.datasets.Dataset.fetch] - Single flag data retrieval
        """
        flags = list(flags)
        self._check_declared_by_any(flags, kwargs)
        dfs = {
            str(flag): self.fetch(flag, config, granularity=granularity, **self._kwargs_for(flag, kwargs))
            for flag in flags
        }
        self._check_same_axes(dfs)
        df = pd.concat(
            dfs,
            axis=flag_concat_axis,
            names=[concat_level_name],
        )
        if not concat_level_at_top:
            df = move_top_level_last(df, flag_concat_axis)
        return df

    def _check_declared_by_any(self, flags: list[FlagType], kwargs: dict[str, Any]) -> None:
        declared = set(self._accepted_dataset_params).union(
            *({d.name for d in self.flag_index.get_spec(flag).params} for flag in flags)
        )
        unknown = set(kwargs) - declared
        if unknown:
            raise ValueError(
                f'No flag spec of {[str(f) for f in flags]} and no dataset class declares the parameters '
                f'{sorted(unknown)}. Declared: {sorted(declared)}.'
            )

    @classmethod
    def _check_same_axes(cls, dfs: dict[str, pd.Series | pd.DataFrame]) -> None:
        if not dfs:
            return
        (first_flag, first), *others = dfs.items()
        for flag, other in others:
            difference = cls._axes_difference(first, other)
            if difference:
                raise ValueError(f'Cannot concat the flags {first_flag!r} and {flag!r}: {difference}')

    @classmethod
    def _axes_difference(cls, first: pd.Series | pd.DataFrame, other: pd.Series | pd.DataFrame) -> str | None:
        if type(first) is not type(other):
            return f'one is a {type(first).__name__}, the other a {type(other).__name__}.'
        for name, ax_first, ax_other in zip(('index', 'columns'), first.axes, other.axes):
            if ax_first.nlevels != ax_other.nlevels:
                return f'the {name} has {ax_first.nlevels} and {ax_other.nlevels} levels.'
            types_first, types_other = cls._level_types(ax_first), cls._level_types(ax_other)
            if types_first != types_other:
                return f'the {name} has the index types {types_first} and {types_other}.'
        return None

    @staticmethod
    def _level_types(ax: pd.Index) -> list[str]:
        levels = ax.levels if isinstance(ax, pd.MultiIndex) else [ax]
        return ['Index' if isinstance(level, pd.RangeIndex) else type(level).__name__ for level in levels]

    @classmethod
    def get_flag_type(cls) -> Type[FlagType]:
        """
        Get the flag type class for this dataset type.

        Returns the type used for data flags in this dataset class. Subclasses
        can override to specify a custom flag type for type checking and
        validation.

        Returns:
            Type[FlagType]: The flag type class (default: FlagTypeProtocol)

        Note:
            Override in subclasses that use custom flag types.
        """
        from mesqual.flag.flag import FlagTypeProtocol

        return FlagTypeProtocol

    @classmethod
    def get_config_type(cls) -> Type[DatasetConfigType]:
        """
        Get the configuration type class for this dataset type.

        Returns the DatasetConfig subclass used by this dataset. Platform
        interfaces typically override this to return their extended config
        class with platform-specific options.

        Returns:
            Type[DatasetConfigType]: The config type class (default: DatasetConfig)

        Examples:
            >>> from mesqual.testing import DummyDataset
            >>> DummyDataset.get_config_type()
            <class 'mesqual.testing.dataset.DummyConfig'>

        Note:
            Override in platform dataset subclasses to return platform-specific
            config types with additional options.
        """
        from mesqual.datasets.dataset_config import DatasetConfig

        return DatasetConfig

    @property
    def instance_config(self) -> DatasetConfigType:
        """
        Get the effective configuration for this dataset instance.

        Computes the merged configuration by combining:

        1. Base config defaults
        2. Class-level config (set via set_class_config)
        3. Instance-level config (passed to __init__ or set via set_instance_config)

        Later settings override earlier ones. This is the configuration used
        by fetch() unless overridden by a fetch-time config parameter.

        Returns:
            DatasetConfigType: The fully resolved configuration for this instance

        Examples:
            >>> dataset.instance_config.use_cache
            True

        See Also:
            - [`set_instance_config()`][mesqual.datasets.Dataset.set_instance_config] - Replace instance configuration
            - [`set_class_config()`][mesqual.datasets.Dataset.set_class_config] - Set class-level defaults
            - [`DatasetConfigManager`][mesqual.datasets.DatasetConfigManager] - Configuration management system
        """
        from mesqual.datasets.dataset_config import DatasetConfigManager

        return DatasetConfigManager.get_effective_config(self.__class__, self._config)

    def describe_config(self) -> pd.DataFrame:
        """Describe the config fields of this dataset, one row per field.

        Returns:
            DataFrame indexed by field name with the columns `type`,
                `default`, `value` (effective value without a fetch-time config),
                `layer` (the layer that set the value: `'default'`,
                `'class <Name>'` or `'instance'`) and `description`.

        Examples:
            >>> dataset.describe_config().loc['use_cache', ['value', 'layer']].tolist()
            [True, 'default']
        """
        from mesqual.datasets.dataset_config import DatasetConfigManager

        layers = DatasetConfigManager.get_layers(type(self), self._config)
        effective = self.instance_config
        rows = {}
        for field in dataclasses.fields(effective):
            setting_layer = 'default'
            for label, config in layers[1:]:
                if field.name in config.explicit_fields:
                    setting_layer = label
            rows[field.name] = {
                'type': field.type if isinstance(field.type, str) else getattr(field.type, '__name__', str(field.type)),
                'default': effective._field_default(field.name),
                'value': getattr(effective, field.name),
                'layer': setting_layer,
                'description': field.metadata.get('description', ''),
            }
        return pd.DataFrame.from_dict(rows, orient='index')

    def set_instance_config(self, config: DatasetConfigType) -> None:
        """
        Replace the instance-level configuration for this dataset.

        Sets the configuration that will be merged with class-level defaults
        to produce the effective configuration used by fetch().

        Args:
            config: New configuration object to use for this instance

        Examples:
            >>> from mesqual.datasets import DatasetConfig
            >>> dataset.set_instance_config(DatasetConfig(use_cache=False, auto_sort_index=False))

        See Also:
            - [`instance_config`][mesqual.datasets.Dataset.instance_config] - Get the effective configuration
            - [`set_instance_config_kwargs()`][mesqual.datasets.Dataset.set_instance_config_kwargs] - Update settings
            - [`set_class_config()`][mesqual.datasets.Dataset.set_class_config] - Set class-level defaults
        """
        self._config = config

    def merge_instance_config(self, config: DatasetConfigType) -> None:
        """Merge `config` on top of the instance config of this dataset.

        Uses `DatasetConfig.merge` on copies: a field that `config` sets
        explicitly wins, and the other explicit fields of the instance
        config stay. Without an instance config, a copy of `config` becomes
        the instance config. Later changes to `config` do not reach the
        dataset.

        Args:
            config: Upper layer for the instance config.

        Examples:
            >>> from mesqual.datasets import DatasetConfig
            >>> dataset.set_instance_config(DatasetConfig(use_cache=False))
            >>> dataset.merge_instance_config(DatasetConfig(granularity='1h'))
            >>> dataset.instance_config.use_cache, dataset.instance_config.granularity
            (False, '1h')

        See Also:
            - [`set_instance_config()`][mesqual.datasets.Dataset.set_instance_config] - Replace the config
        """
        upper = copy.deepcopy(config)
        self._config = upper if self._config is None else copy.deepcopy(self._config).merge(upper)

    def set_instance_config_kwargs(self, **kwargs: Any) -> None:
        """
        Update individual configuration settings on this instance.

        Modifies specific attributes of the existing instance configuration
        without replacing the entire config object. Useful for tweaking
        individual settings.

        Args:
            **kwargs: Configuration attribute names and values to set

        Examples:
            >>> dataset.set_instance_config_kwargs(use_cache=False)
            >>> dataset.instance_config.use_cache
            False

        Creates an instance config of `get_config_type()` first if the
        dataset has none.

        See Also:
            - [`set_instance_config()`][mesqual.datasets.Dataset.set_instance_config] - Replace entire configuration
            - [`instance_config`][mesqual.datasets.Dataset.instance_config] - Get the effective configuration
        """
        if self._config is None:
            self._config = self.get_config_type()()
        for key, value in kwargs.items():
            setattr(self._config, key, value)

    @classmethod
    def set_class_config(cls, config: DatasetConfigType) -> None:
        """
        Set the class-level configuration for all instances of this dataset type.

        Class-level configuration serves as the default for all instances of
        this class. Instance-level configuration (set via set_instance_config)
        can override these defaults.

        Args:
            config: Configuration object to use as class-level defaults

        Examples:
            >>> from mesqual.datasets import DatasetConfig
            >>> from mesqual.testing import DummyDataset
            >>> DummyDataset.set_class_config(DatasetConfig(use_cache=False))
            >>> study.scen.get_dataset('high_demand').instance_config.use_cache
            False

        Note:
            This affects all instances of the class, including existing ones
            that haven't overridden the setting at instance level.

        See Also:
            - [`set_instance_config()`][mesqual.datasets.Dataset.set_instance_config] - Override for specific instances
            - [`DatasetConfigManager`][mesqual.datasets.DatasetConfigManager] - Configuration management system
        """
        from mesqual.datasets.dataset_config import DatasetConfigManager

        DatasetConfigManager.set_class_config(cls, config)

    def is_flag_available(self, flag: FlagType) -> bool | None:
        """Return whether this dataset can answer a flag, without reading time series data.

        A flag that the dataset does not accept, or a flag of a family with
        a variant value that is not a value on the dataset, is not available
        (`False`). Else:

        - Platform dataset: the `_is_available` of the answering
          interpreter decides. If it returns `None`, the flag is unknown
          (`None`) until its first successful fetch from the platform
          dataset.
        - Collection: the union of the children (available in one child,
          else unknown in one child, else `False`). Comparison: the
          intersection of both children (`False` if one child is `False`,
          else unknown if one child is unknown, else available).
        - Any other dataset: its own `_is_available`. `None` counts as
          available.

        The interpreter of a layer flag implements `_is_available` with
        the availability of the flags that its fetch reads: the base flag,
        the base model or border model, and the `input_flags` of the
        aggregation. `availability` and `flag_info` give the reason.

        Returns:
            `True`, `False`, or `None` for unknown.
        """
        return self.availability(flag).available

    def availability(self, flag: FlagType) -> Availability:
        """Return whether this dataset can answer a flag, with the reason. Reads no time series data.

        The same rules as `is_flag_available`, which returns the
        `available` field of the result. An interpreter uses it on its
        parent dataset to combine the availability of the flags that it
        fetches (`Availability.of_inputs`).

        Examples:
            >>> from mesqual.testing import DummyDataset, DummyScenario
            >>> without_prices = DummyDataset(DummyScenario(missing_tables=('node_price',)))
            >>> without_prices.availability('Country.Results.price').reason
            "DummyAreaFlags._is_available: unavailable input Node.Results.price (DummyTimeSeriesInterpreter._is_available: table 'node_price' not in the source)"
        """
        return self._availability(flag)

    def _is_available(self, flag: FlagType) -> bool | Availability | None:
        """Hook: `True` or `False` when this dataset knows cheaply whether it can answer `flag`, `None` for unknown.

        An interpreter implements it with cheap checks, for example on the
        source inventory of its parent dataset
        (`self.parent_dataset.source_inventory`). It reads the inventory and
        does not fetch. Default: `None`.

        It may return an `Availability` instead of a bool, so the reason
        names the missing table or input. An `Availability(None, reason)`
        is unknown with that reason. Unlike `None`, it decides the
        availability on a platform dataset, so an earlier interpreter of
        the chain is not asked.

        Examples:
            ```python
            class PriceInterpreter(MyInterpreter):
                def _is_available(self, flag):
                    return 'prices.csv' in self.parent_dataset.source_inventory.files

            class VolumeInterpreter(MyInterpreter):
                def _is_available(self, flag):
                    found = 'volumes.csv' in self.parent_dataset.source_inventory.files
                    return Availability(found, 'volumes.csv in the source' if found else 'no volumes.csv')
            ```
        """
        return None

    def _availability(self, flag: FlagType) -> Availability:
        if not self.flag_is_accepted(flag):
            return Availability(False, f'{self.name} does not accept the flag.')
        return self._family_value_problem(flag) or self._checked_availability(flag)

    def _family_value_problem(self, flag: FlagType) -> Availability | None:
        """`Availability(False, reason)` if a family variant value of `flag` is not a value on this dataset.

        Also if the values of a family variant need an unavailable flag.
        """
        spec = self.flag_index.get_spec(flag)
        if not spec.is_family:
            return None
        try:
            spec.check_variant_values(flag, self._root_dataset())
        except FlagUnavailableError as error:
            return Availability(False, error.reason)
        except ValueError as error:
            return Availability(False, str(error))
        return None

    def _checked_availability(self, flag: FlagType) -> Availability:
        """Availability of a flag that this dataset accepts."""
        checked = self._availability_from_hook(self._is_available(flag), f'{type(self).__name__}._is_available')
        return checked or Availability(True, f'{type(self).__name__} has no availability check')

    @staticmethod
    def _availability_from_hook(checked: bool | Availability | None, source: str) -> Availability | None:
        """Turn the result of an `_is_available` hook into an `Availability`.

        `None` stays `None`. A bool gets `source` as reason. An `Availability`
        keeps its state and gets `source` before its reason.
        """
        if checked is None:
            return None
        if isinstance(checked, Availability):
            return Availability(checked.available, f'{source}: {checked.reason}', checked.datasets)
        return Availability(checked, source)

    def _raise_if_unavailable(self, flag: FlagType) -> None:
        """Raise `FlagUnavailableError` if the availability check says unavailable.

        A family flag whose values need an unavailable flag raises in
        `_check_dataset_values`, which resolves the values anyway.
        """
        availability = self._checked_availability(flag)
        if availability.available is False:
            raise FlagUnavailableError(str(flag), self.name, availability.reason)

    def _flag_availability_by_spec(self) -> dict[str, dict[str, Availability]]:
        """Per spec name: the availability of each flag of the spec. Flag families resolve with the dataset."""
        result = {}
        for spec in self.flag_index.specs:
            flags = spec.expand(self._root_dataset()) if spec.is_family else spec.expand()
            result[spec.name] = {flag: self._availability(flag) for flag in flags}
        return result

    def _values_dataset(self, flag: FlagType) -> Dataset:
        """The dataset that resolves the values functions of the dimensions of `flag`."""
        return self._root_dataset()

    def _flag_chain_names(self, flag: FlagType) -> tuple[str, ...]:
        """The interpreter chain of `flag`, for `flag_info`. Empty for a dataset that does not route flags."""
        return ()

    def _describe_children(self) -> list[Dataset]:
        """The children shown in `describe_datasets` and in the union of `describe_flags`."""
        return []

    @classmethod
    def _declared_flag_specs(cls) -> tuple[FlagSpec, ...]:
        """The specs that the class declares, for `describe_class_flags`."""
        return tuple(cls.flag_specs)

    @classmethod
    def _class_layers(cls) -> LayerSet:
        """The area and border layers registered on the class. A plain dataset class has none."""
        return LayerSet()

    @classmethod
    def _class_flag_index(cls) -> FlagIndex:
        """The flag index of the declared specs and registered layers of the class, without reference checks.

        It answers class-level questions, for example the layers of a layer
        spec in `describe_class_flags`.
        """
        return FlagIndex(cls._declared_flag_specs(), cls._class_layers())

    @classmethod
    def _describes_class_flags(cls) -> bool:
        """`False` for a collection class other than a leaf: it has no flags of its own."""
        return True

    @classmethod
    def describe_class_flags(
        cls,
        contains: str | None = None,
        layer: str | None = None,
        item_type: ItemTypeEnum | str | None = None,
        as_records: bool = False,
    ) -> pd.DataFrame | list[dict]:
        """Describe the flag specs that this class declares, one row per spec. Needs no source.

        On a platform dataset class: the specs of its answering
        interpreters. On a dataset or interpreter class: its specs.
        Flag families show their template, and dimensions with values per
        dataset show `'resolved per dataset'`.

        Args:
            contains: Keep specs whose name or one of whose flags contains
                this substring, case-insensitive.
            layer: Keep specs on this area or border layer.
            item_type: Keep specs of this item type.
            as_records: Return a list of plain dicts, ready for JSON.

        Returns:
            One row per spec, indexed by name, with the columns `item_type`
                (`Model`, `TimeSeries` or `Other`), `object` (the model flag; for
                a layer spec without `model_flag`, the model flags of its
                layers, in the order of `layers`), `unit`, `quantity_type`, `params` (flag and dataset parameters
                with their defaults), `description`, `variants` and `layers`.

        Raises:
            TypeError: Called on a collection class other than a platform
                dataset class, or through an instance of one. A collection
                has no flags of its own: use `describe_flags()` on the
                instance.

        See Also:
            - [`describe_flags`][mesqual.datasets.Dataset.describe_flags]: the flags of one dataset
              instance, with resolved flag families and layers, effective parameter defaults and
              availability. Same index, and the same columns plus some more.

        Examples:
            >>> from mesqual.testing import DummyDataset
            >>> DummyDataset.describe_class_flags(contains='loading')[['object', 'unit']]
                                      object      unit
            name
            Line.Results.loading  Line.Model  per_unit
        """
        rows = FlagDescriber(cls).class_spec_rows()
        rows = FlagDescriber.filter_rows(rows, contains, layer, item_type)
        return FlagDescriber.output(rows, as_records, hidden=('flags_list',))

    def describe_flags(
        self,
        contains: str | None = None,
        layer: str | None = None,
        item_type: ItemTypeEnum | str | None = None,
        available_only: bool = True,
        as_records: bool = False,
    ) -> pd.DataFrame | list[dict]:
        """Describe the flags of this dataset, one row per spec. Loads no time series data.

        As `describe_class_flags`, with the values of flag families and layers
        resolved for this dataset (from model flags), the effective parameter
        defaults with their source (`spec`, `dataset`, `config`, `required`,
        or `per dataset` on a collection whose children resolve different
        defaults), and the availability. A collection describes the union of
        its children.

        Args:
            contains: Keep specs whose name or one of whose flags contains
                this substring, case-insensitive.
            layer: Keep specs on this area or border layer.
            item_type: Keep specs of this item type.
            available_only: Drop the specs whose flags are all unavailable.
                Specs with unknown availability stay.
            as_records: Return a list of plain dicts, ready for JSON.

        Returns:
            One row per spec, indexed by name, with the columns of
                `describe_class_flags` in the same order, and `flags` (number of flags that are
                available or unknown), `availability` (`available`, `unknown`,
                `partial` if some flags are unavailable, or `unavailable`) and
                `datasets` (for a collection: the children that have the flag,
                if not all of them).

        See Also:
            - [`describe_class_flags`][mesqual.datasets.Dataset.describe_class_flags]: the specs as
              declared on the class, without a source; no resolved values and no availability.

        Examples:
            >>> dataset.describe_flags(contains='loading')[['object', 'availability']]
                                      object availability
            name
            Line.Results.loading  Line.Model    available
        """
        rows = FlagDescriber.filter_rows(FlagDescriber(self).flag_rows(), contains, layer, item_type)
        if available_only:
            rows = [row for row in rows if row['availability'] != 'unavailable']
        hidden = ('flags_list',) if self._describe_children() else ('flags_list', 'datasets')
        return FlagDescriber.output(rows, as_records, hidden=hidden)

    def flag_info(self, flag: FlagType) -> FlagInfo:
        """Return everything mesqual knows about one flag of this dataset, without its data.

        Returns:
            `FlagInfo` with the spec and its base, the aggregation, the
                variant values, the combined description, unit and quantity type
                with their source, the schema, the parameters with their
                effective defaults and source, `fill_value`, the model flag, the
                availability with its reason, the interpreter chain and the
                recorded dependencies of the last fetch without arguments.

        Raises:
            UnknownFlagError: No spec declares the flag.

        Examples:
            >>> dataset.flag_info('Line.Results.flow_received_net').description
            'Power flow on the line. Side: where the flow leaves the line, after losses. Direction: up minus down.'
        """
        return FlagDescriber(self).flag_info(flag)

    @classmethod
    def describe_class_area_layers(cls, as_records: bool = False) -> pd.DataFrame | list[dict]:
        """Describe the area layers registered on this class, one row per layer.

        Args:
            as_records: Return a list of plain dicts, ready for JSON.

        Returns:
            One row per area layer, indexed by name, with the columns
                `index`, `refers_to` (base model and layer model),
                `mapping_columns` (the membership column of the base model),
                `flags` (number of declared flag specs on the layer) and
                `description`. Empty for a class without area layers.

        Raises:
            TypeError: Called on a collection class other than a platform
                dataset class, or through an instance of one.

        See Also:
            - [`describe_class_border_layers`][mesqual.datasets.Dataset.describe_class_border_layers]: the border layers.
        """
        return FlagDescriber.output(
            FlagDescriber(cls).area_layer_rows(), as_records, columns=FlagDescriber.LAYER_COLUMNS
        )

    @classmethod
    def describe_class_border_layers(cls, as_records: bool = False) -> pd.DataFrame | list[dict]:
        """Describe the border layers registered on this class, one row per layer.

        Args:
            as_records: Return a list of plain dicts, ready for JSON.

        Returns:
            One row per border layer, indexed by name, with the columns
                `index`, `refers_to` (area layer and edge model),
                `mapping_columns` (the edge ends of the edge model), `flags`
                (number of declared flag specs on the layer) and `description`.
                Empty for a class without border layers.

        Raises:
            TypeError: Called on a collection class other than a platform
                dataset class, or through an instance of one.

        See Also:
            - [`describe_class_area_layers`][mesqual.datasets.Dataset.describe_class_area_layers]: the area layers.
        """
        return FlagDescriber.output(
            FlagDescriber(cls).border_layer_rows(), as_records, columns=FlagDescriber.LAYER_COLUMNS
        )

    def describe_datasets(
        self,
        max_depth: int | None = None,
        as_records: bool = False,
    ) -> pd.DataFrame | list[dict]:
        """Describe the tree of this dataset and its children, one row per dataset.

        A platform dataset is a leaf: its interpreters are not listed.

        Args:
            max_depth: Number of levels below this dataset. `None`: the full
                tree. `1`: this dataset and its direct children. `0`: this
                dataset only.
            as_records: Return a list of plain dicts, ready for JSON.

        Returns:
            One row per dataset, depth first, indexed by `path` (the names
                from this dataset down, joined with `/`), with the columns
                `name`, `class`, `description`, `attributes`, `config` (the
                explicit fields of the instance config) and `children` (all
                children, also below `max_depth`).

        Raises:
            ValueError: `max_depth` is negative.

        Examples:
            >>> study.scen.describe_datasets(max_depth=1)['class'].tolist()
            ['DatasetConcatCollection', 'DatasetMergeCollection', 'DummyDataset', 'DummyDataset', 'DummyDataset']
        """
        if max_depth is not None and max_depth < 0:
            raise ValueError(f'max_depth must be None or at least 0, got {max_depth}.')
        return FlagDescriber.output(DatasetTreeDescriber(self).rows(max_depth), as_records, index='path')

    def __str__(self) -> str:
        return self.name

flag_specs class-attribute

flag_specs: tuple[FlagSpec, ...] = ()

Flag specs of this dataset. accepted_flags and the flag index come from them.

description class-attribute instance-attribute

description: str | None = None

Optional free text about the dataset. A platform class or a study sets it, on the class or the instance. describe_datasets() shows it.

fetch_params class-attribute

fetch_params: tuple[Dimension, ...] = ()

Dataset parameters: keyword arguments of fetch that the dataset class uses itself.

A class has the dataset parameters of its base classes and adds its own: fetch_params of a subclass lists only the new ones, and the class gets the merged tuple.

A flag spec declares the parameters of one flag (FlagSpec.params). fetch_params declares the parameters of every fetch of this dataset class, for example comparison_type of DatasetComparison. Both kinds resolve the same way: the keyword argument, then param_defaults of the effective config, then the default of the dimension. The resolved value goes to _fetch as a keyword argument.

A name may not be both a flag parameter and a dataset parameter of one dataset. A collection also accepts the dataset parameters of its children. It passes each child the flag parameters of the call and the dataset parameters that the child accepts and that the call or the param_defaults of the collection gave, its own ones included (for example comparison_type to the comparisons of a comparison of comparisons). A value in the call and the same value in param_defaults reach the child alike. The collection never passes the default of a dimension.

Examples:

class NodePrices(Dataset):
    flag_specs = (NODE_PRICE_SPEC,)
    fetch_params = (Dimension('currency', 'Currency of the prices.', {
        'EUR': 'euro', 'USD': 'US dollar',
    }, default='EUR'),)

    def _fetch(self, flag, effective_config, currency, **params):
        ...

NodePrices().fetch('Node.Results.price', currency='USD')

A subclass may override fetch to show its dataset parameters in the IDE: one typed keyword-only argument per dataset parameter, with the default UNSET. The override passes on to super().fetch only the arguments that are not UNSET (_given). So a parameter that the call leaves out still resolves from param_defaults, and call_kwargs holds only what the caller gave.

accepted_flags property

accepted_flags: set[FlagType]

Set of the flags of the specs of this dataset.

It holds concrete flag names, computed from the declarations in flag_specs without opening source data. Flags of a flag family are not listed, because their values come from the data of the dataset. flag_is_accepted accepts them by their name template. Whether a source has the data of a flag is a separate question (availability).

A subclass may narrow the spec flags using declarations of siblings or attributes, never source data.

Returns:

Type Description
set[FlagType]

Set of flags that can be used with the fetch() method

Examples:

>>> 'Line.Results.flow_sent_net' in dataset.accepted_flags
True

flag_index property

flag_index: FlagIndex

The flag index: lookup from a flag to its spec.

Built from flag_specs at construction. A dataset constructed with a parent dataset, for example an interpreter, uses the index of its parent. Collections and platform datasets document their own rule.

cache property

cache: Cache | None

The cache of fetch results of this dataset, or None.

fetch uses it when use_cache is set (see Cache).

attributes property

attributes: dict

Access the metadata attributes dictionary for this dataset.

Attributes store scenario-level metadata such as configuration parameters, simulation settings, or descriptive labels. These are useful for filtering, grouping, and annotating datasets in collections.

Returns:

Name Type Description
dict dict

Dictionary of attribute key-value pairs. Keys are strings, values are primitive types (bool, int, float, str).

Examples:

>>> dataset.attributes
{'first_date': '2024-03-28', 'last_date': '2024-04-01', 'days': 5, 'demand_factor': 1.0, 'periods': 2}
See Also

parent_dataset property writable

parent_dataset: 'DatasetLinkCollection'

The parent dataset of an interpreter: the platform dataset that holds it and its sibling interpreters.

An interpreter answers a part of the flags of a platform dataset. It fetches the flags of its siblings through the parent, so it needs no reference to them. A fetch through the parent reaches an override of the sibling interpreter. The parent is a DatasetLinkCollection, usually a PlatformDataset.

Returns:

Name Type Description
DatasetLinkCollection 'DatasetLinkCollection'

The parent collection.

Raises:

Type Description
RuntimeError

The parent is not set, for example for an interpreter used without a parent dataset.

Examples:

An interpreter that sums the generation per node from two flags of its siblings:

class NodeGeneration(MyInterpreter):
    flag_specs = (NODE_GENERATION_SPEC,)

    def _fetch(self, flag, effective_config, **params):
        generation = self.parent_dataset.fetch('Generator.Results.generation', effective_config)
        node_of_generator = self.parent_dataset.fetch('Generator.Model', effective_config)['node']
        return generation.T.groupby(node_of_generator).sum().T
See Also

instance_config property

instance_config: DatasetConfigType

Get the effective configuration for this dataset instance.

Computes the merged configuration by combining:

  1. Base config defaults
  2. Class-level config (set via set_class_config)
  3. Instance-level config (passed to init or set via set_instance_config)

Later settings override earlier ones. This is the configuration used by fetch() unless overridden by a fetch-time config parameter.

Returns:

Name Type Description
DatasetConfigType DatasetConfigType

The fully resolved configuration for this instance

Examples:

>>> dataset.instance_config.use_cache
True
See Also

__init__

__init__(name: str = None, parent_dataset: Dataset = None, attributes: dict = None, cache: Cache = None, config: DatasetConfigType = None)

Initialize a new Dataset instance.

Parameters:

Name Type Description Default
name str

Human-readable identifier. If None, auto-generates from class name

None
parent_dataset Dataset

Optional parent dataset for hierarchical relationships

None
attributes dict

Dictionary of metadata attributes for the dataset. Keys must be of type str, values of type bool, int, float or str.

None
cache Cache

Optional cache of fetch results

None
config DatasetConfigType

Configuration object controlling dataset behavior

None

Raises:

Type Description
TypeError

If an attribute key or value has an invalid type.

ValueError

A time series spec names a model flag without model spec.

AmbiguousOverrideError

Two specs give the same flag name.

ValueError

A name is a flag parameter and a dataset parameter (fetch_params).

Source code in src/mesqual/datasets/dataset.py
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
def __init__(
    self,
    name: str = None,
    parent_dataset: Dataset = None,
    attributes: dict = None,
    cache: Cache = None,
    config: DatasetConfigType = None,
):
    """
    Initialize a new Dataset instance.

    Args:
        name: Human-readable identifier. If None, auto-generates from class name
        parent_dataset: Optional parent dataset for hierarchical relationships
        attributes: Dictionary of metadata attributes for the dataset. Keys must be
            of type str, values of type bool, int, float or str.
        cache: Optional cache of fetch results
        config: Configuration object controlling dataset behavior

    Raises:
        TypeError: If an attribute key or value has an invalid type.
        ValueError: A time series spec names a model flag without model spec.
        AmbiguousOverrideError: Two specs give the same flag name.
        ValueError: A name is a flag parameter and a dataset parameter
            (`fetch_params`).
    """
    self.name = name or f'{self.__class__.__name__}_{str(id(self))}'
    self._flag_index: FlagIndex | None = None
    self._parent_dataset = parent_dataset
    self._attributes: dict = dict()
    self._add_attributes(attributes or dict())
    self._cache = cache
    self._config = config
    self._fetch_records: dict[tuple, FetchRecord] = {}
    self._fetched_flags: set[str] = set()
    self._flag_index = self._build_flag_index()
    self._check_param_name_clash()

    from mesqual.kpis.collection import KPICollection

    self.kpi_collection: KPICollection = KPICollection()

fetch

fetch(flag: FlagType, config: dict | DatasetConfigType = None, *, granularity: str | Timedelta | None | Unset = UNSET, **kwargs: Any) -> Series | DataFrame

Fetch the data of one flag.

fetch takes two kinds of input:

  • config: overrides of config fields for this call.
  • **kwargs: parameters. Every keyword argument is a parameter. A flag spec declares flag parameters (FlagSpec.params), for example solution. The dataset class declares dataset parameters (fetch_params), for example comparison_type of a comparison.

Each parameter resolves the same way, first match wins:

  1. The keyword argument of this call.
  2. param_defaults of the effective config.
  3. The default of the dimension.

_fetch gets the resolved parameters as keyword arguments. A collection resolves only its own dataset parameters. It passes the flag parameters of the call to its children. It gives each child the dataset parameters that the child accepts and that the call or the param_defaults of the collection gave, its own ones included, never the default of a dimension (_kwargs_for_child).

Config resolution, later wins: config class defaults, the class config of each class in the MRO (base first), the instance config, and the config of this call. Only the explicit fields of a layer override lower layers (the keys of a dict, or the explicit fields of a config object), also with a None value. DatasetConfig.CLEAR resets a field to its default.

Steps:

  1. Raise UnknownFlagError if no flag spec declares the flag, and ValueError if this dataset does not accept it or if a dataset that reads the data itself (not a collection) has the flag unavailable (_is_available is False). Collections fetch only from children on which the flag is not unavailable.
  2. Resolve the effective config.
  3. Resolve the parameters.
  4. Open a frame in the fetch context. Nested fetches and the config fields that _fetch reads are recorded in it (get_fetch_record).
  5. Call _fetch with the resolved parameters, or read the cache, and post-process.
  6. In the outermost call, apply the time index filter to a time series flag, then convert it to the granularity of the effective config.

Settings applied after the cache (time_index_filter, granularity): they apply only in the outermost fetch call, from its effective config, and only to time series flags, first the filter, then the conversion. _fetch gets them as None, so a cached value is never filtered or converted. Nested fetches (collection children, interpreters, layer flags) return unfiltered native data: such a setting in the config of a child dataset has no effect when the child is fetched through a collection. The fetch depth lives in a context variable: run code in a thread started inside a fetch with contextvars.copy_context().run(...), else its fetches count as outermost.

The config field granularity converts a time series flag to another time resolution with TimeSeriesGranularityConverter and the quantity type of the flag: intensive values are averaged or repeated, extensive values summed or split. None means the native resolution. On a row MultiIndex it converts the datetime level per combination of the other levels. Model and other flags are returned unchanged. The argument granularity is a short form of config={'granularity': ...} in the outermost call. In a nested fetch it has no effect, so the call keeps its config as given.

Parameters:

Name Type Description Default
flag FlagType

The flag. It must be in accepted_flags or match a flag family of the dataset.

required
config dict | DatasetConfigType

Config overrides for this call: a dict with config field names as keys, or a config object of the config type of the dataset. A platform config may add its own fields.

None
granularity str | Timedelta | None | Unset

Target time resolution of a time series flag, a pandas frequency such as '15min' or '1h'. None: the native resolution. Left out: the granularity of the effective config. It overrides the config of the call. No effect in a nested fetch.

UNSET
**kwargs Any

Parameters: flag parameters of the spec and dataset parameters of the dataset class (for a collection, also of its children).

{}

Returns:

Type Description
Series | DataFrame

DataFrame or Series with the data of the flag.

Raises:

Type Description
UnknownFlagError

No flag spec declares the flag.

FlagUnavailableError

The flag is not available (is_flag_available is False). A subclass of ValueError.

ValueError

The dataset does not accept the flag; a keyword argument is no declared parameter; a parameter is missing or has a value that its dimension does not allow; param_defaults names an unknown parameter (outermost call only). A name in the instance config or in config must be a flag parameter of the flag index or a dataset parameter of this dataset (for a collection, also of its children). A name in a class config (set_class_config) may also be a dataset parameter of any Dataset subclass. The granularity of the effective config is not a frequency such as '15min'.

TypeError

A granularity conversion of a result whose row index has no datetime level.

Examples:

A time series in another resolution. The base scenario has hourly rows, then quarter-hourly rows. The result has one row per hour:

>>> hourly = dataset.fetch('Node.Results.price', granularity='1h')
>>> len(dataset.fetch('Node.Results.price')), len(hourly)
(332, 119)
Source code in src/mesqual/datasets/dataset.py
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
def fetch(
    self,
    flag: FlagType,
    config: dict | DatasetConfigType = None,
    *,
    granularity: str | pd.Timedelta | None | Unset = UNSET,
    **kwargs: Any,
) -> pd.Series | pd.DataFrame:
    """Fetch the data of one flag.

    `fetch` takes two kinds of input:

    - `config`: overrides of config fields for this call.
    - `**kwargs`: parameters. Every keyword argument is a parameter. A
      flag spec declares flag parameters (`FlagSpec.params`), for example
      `solution`. The dataset class declares dataset parameters
      (`fetch_params`), for example `comparison_type` of a comparison.

    Each parameter resolves the same way, first match wins:

    1. The keyword argument of this call.
    2. `param_defaults` of the effective config.
    3. The `default` of the dimension.

    `_fetch` gets the resolved parameters as keyword arguments. A
    collection resolves only its own dataset parameters. It passes the
    flag parameters of the call to its children. It gives each child
    the dataset parameters that the child accepts and that the call or
    the `param_defaults` of the collection gave, its own ones included,
    never the default of a dimension (`_kwargs_for_child`).

    Config resolution, later wins: config class defaults, the class
    config of each class in the MRO (base first), the instance config,
    and the `config` of this call. Only the explicit fields of a layer
    override lower layers (the keys of a dict, or the explicit fields of
    a config object), also with a `None` value. `DatasetConfig.CLEAR`
    resets a field to its default.

    Steps:

    1. Raise `UnknownFlagError` if no flag spec declares the flag,
       and `ValueError` if this dataset does not accept it or if
       a dataset that reads the data itself (not a collection)
       has the flag unavailable (`_is_available` is `False`).
       Collections fetch only from children on which the flag is
       not unavailable.
    2. Resolve the effective config.
    3. Resolve the parameters.
    4. Open a frame in the fetch context. Nested fetches and the
       config fields that `_fetch` reads are recorded in it
       (`get_fetch_record`).
    5. Call `_fetch` with the resolved parameters, or read the
       cache, and post-process.
    6. In the outermost call, apply the time index filter to a time
       series flag, then convert it to the `granularity` of the
       effective config.

    Settings applied after the cache (`time_index_filter`,
    `granularity`): they apply only in the outermost `fetch` call, from
    its effective config, and only to time series flags, first the
    filter, then the conversion. `_fetch` gets them as `None`, so a
    cached value is never filtered or converted. Nested fetches
    (collection children, interpreters, layer flags) return unfiltered
    native data: such a setting in the config of a child dataset has no
    effect when the child is fetched through a collection. The fetch
    depth lives in a context variable: run code in a thread started
    inside a fetch with `contextvars.copy_context().run(...)`, else its
    fetches count as outermost.

    The config field `granularity` converts a time series flag to
    another time resolution with `TimeSeriesGranularityConverter` and
    the quantity type of the flag: intensive values are averaged or
    repeated, extensive values summed or split. `None` means the native
    resolution. On a row `MultiIndex` it converts the datetime level per
    combination of the other levels. Model and other flags are returned
    unchanged. The argument `granularity` is a short form of
    `config={'granularity': ...}` in the outermost call. In a nested
    fetch it has no effect, so the call keeps its config as given.

    Args:
        flag: The flag. It must be in `accepted_flags` or match a flag
            family of the dataset.
        config: Config overrides for this call: a dict with config field
            names as keys, or a config object of the config type of the
            dataset. A platform config may add its own fields.
        granularity: Target time resolution of a time series flag, a
            pandas frequency such as `'15min'` or `'1h'`. `None`: the
            native resolution. Left out: the `granularity` of the
            effective config. It overrides the `config` of the call.
            No effect in a nested fetch.
        **kwargs: Parameters: flag parameters of the spec and dataset
            parameters of the dataset class (for a collection, also of
            its children).

    Returns:
        DataFrame or Series with the data of the flag.

    Raises:
        UnknownFlagError: No flag spec declares the flag.
        FlagUnavailableError: The flag is not available
            (`is_flag_available` is `False`). A subclass of `ValueError`.
        ValueError: The dataset does not accept the flag; a keyword
            argument is no declared parameter; a parameter is missing or
            has a value that its dimension does not allow;
            `param_defaults` names an unknown parameter (outermost call
            only). A name in the instance config or in `config` must be
            a flag parameter of the flag index or a dataset parameter of
            this dataset (for a collection, also of its children). A
            name in a class config (`set_class_config`) may also be a
            dataset parameter of any `Dataset` subclass. The
            `granularity` of the effective config is not a frequency
            such as `'15min'`.
        TypeError: A `granularity` conversion of a result whose row
            index has no datetime level.

    Examples:
        A time series in another resolution. The base scenario has hourly
        rows, then quarter-hourly rows. The result has one row per hour:

        >>> hourly = dataset.fetch('Node.Results.price', granularity='1h')
        >>> len(dataset.fetch('Node.Results.price')), len(hourly)
        (332, 119)
    """
    is_outermost = FetchContext.is_outermost()
    enclosing = FetchContext.current()
    with FetchContext.internal():
        if granularity is not UNSET and is_outermost:
            config = self._with_granularity(config, granularity)
        spec = self._spec_of_accepted_flag(flag)
        if self._reads_source:
            self._raise_if_unavailable(flag)
        if is_outermost:
            self.flag_index.check_references()
        if is_outermost:
            self._check_param_default_names(config)
        record, effective_config, params = self._start_call(
            spec,
            flag,
            config,
            self._call_config(config, enclosing),
            kwargs,
        )
        if is_outermost:
            self._check_granularity(effective_config.granularity)
        self._warn_about_unknown_config_fields(effective_config)
    shapes_output = is_outermost and spec.item_type is ItemTypeEnum.TimeSeries
    target_granularity = effective_config.granularity if shapes_output else None
    try:
        with FetchContext.frame(record, self) as frame:
            self._check_dataset_values(spec, flag, params)
            data = self._fetch_from_cache_or_source(flag, effective_config, frame, **params)
        self._fetched_flags.add(str(flag))
    finally:
        self._fetch_records[record.key] = frame.to_record()
    if shapes_output:
        data = self._apply_time_index_filter(data, effective_config)
        data = self._convert_granularity(data, flag, target_granularity)
    return data

get_fetch_record

get_fetch_record(flag: FlagType, config: dict | DatasetConfigType = None, *, granularity: str | Timedelta | None | Unset = UNSET, **kwargs: Any) -> FetchRecord | None

Return the record of the last fetch with the same flag, keyword arguments, config and resolved parameters.

A fetch that raised also leaves a record, with raised=True.

The record lists the fetches that the call made directly, each with its own record, and the config fields that _fetch read. The lookup uses the record key (FetchRecord.key): the keyword arguments and the call config as given, and the parameters, resolved as in fetch.

Parameters:

Name Type Description Default
flag FlagType

The flag.

required
config dict | DatasetConfigType

Fetch-time config, as in fetch. INHERIT finds a call that got the effective config of its enclosing fetch; the parameters then resolve with the config of this dataset. So a child record whose parameters came from the param_defaults of the parent config is not found here. Reach it through the dependencies of the parent record.

None
granularity str | Timedelta | None | Unset

Applies as in an outermost fetch. To look up a nested call, leave it out. No effect with config=INHERIT.

UNSET
**kwargs Any

Parameters, as in fetch.

{}

Returns:

Type Description
FetchRecord | None

The record, or None if the flag was not fetched with these arguments.

Examples:

>>> high = study.scen.get_dataset('high_demand')
>>> prices = high.fetch('Node.Results.price', solution='first')
>>> high.get_fetch_record('Node.Results.price', solution='first').call_kwargs
{'solution': 'first'}
>>> high.get_fetch_record('Node.Results.price') is None
True
Source code in src/mesqual/datasets/dataset.py
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
def get_fetch_record(
    self,
    flag: FlagType,
    config: dict | DatasetConfigType = None,
    *,
    granularity: str | pd.Timedelta | None | Unset = UNSET,
    **kwargs: Any,
) -> FetchRecord | None:
    """Return the record of the last fetch with the same flag, keyword arguments, config and resolved parameters.

    A fetch that raised also leaves a record, with `raised=True`.

    The record lists the fetches that the call made directly, each with
    its own record, and the config fields that `_fetch` read. The lookup
    uses the record key (`FetchRecord.key`): the keyword arguments and
    the call config as given, and the parameters, resolved as in `fetch`.

    Args:
        flag: The flag.
        config: Fetch-time config, as in `fetch`. `INHERIT` finds a call
            that got the effective config of its enclosing fetch; the
            parameters then resolve with the config of this dataset. So
            a child record whose parameters came from the `param_defaults`
            of the parent config is not found here. Reach it through the
            `dependencies` of the parent record.
        granularity: Applies as in an outermost fetch. To look up a
            nested call, leave it out. No effect with `config=INHERIT`.
        **kwargs: Parameters, as in `fetch`.

    Returns:
        The record, or `None` if the flag was not fetched with these arguments.

    Examples:
        >>> high = study.scen.get_dataset('high_demand')
        >>> prices = high.fetch('Node.Results.price', solution='first')
        >>> high.get_fetch_record('Node.Results.price', solution='first').call_kwargs
        {'solution': 'first'}
        >>> high.get_fetch_record('Node.Results.price') is None
        True
    """
    with FetchContext.internal():
        if config is INHERIT:
            layer, call_config = None, INHERIT
        else:
            layer = config if granularity is UNSET else self._with_granularity(config, granularity)
            call_config = self._call_config(layer, None)
        spec = self.flag_index.get_spec(flag)
        record, *_ = self._start_call(spec, flag, layer, call_config, kwargs)
    return self._fetch_records.get(record.key)

version_identity

version_identity(flag: FlagType) -> tuple

Return the version identity of a flag: one entry per link.

A dataset gives ((type(self).__qualname__, spec version),). A platform dataset gives one pair per link of the interpreter chain of the flag. The answering interpreter of a platform dataset gives the identity of its platform dataset, so a version change of an override changes the records of the override and of everything that calls it. A collection gives one (class path, name) pair per child that it can read the flag from; its children record their own identities. A concat collection adds its layout first (DatasetConcatCollection.version_identity).

Source code in src/mesqual/datasets/dataset.py
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
def version_identity(self, flag: FlagType) -> tuple:
    """Return the version identity of a flag: one entry per link.

    A dataset gives `((type(self).__qualname__, spec version),)`. A
    platform dataset gives one pair per link of the interpreter chain of
    the flag. The answering interpreter of a platform dataset gives the
    identity of its platform dataset, so a version change of an override
    changes the records of the override and of everything that calls it.
    A collection gives one `(class path, name)` pair per child that it can
    read the flag from; its children record their own identities. A concat collection
    adds its layout first (`DatasetConcatCollection.version_identity`).
    """
    parent = self._parent_dataset
    if parent is not None and parent._answering_instance(flag) is self:
        return parent.version_identity(flag)
    return ((type(self).__qualname__, self.flag_index.get_spec(flag).version),)

get_accepted_flags_containing_x

get_accepted_flags_containing_x(x: str, match_case: bool = False) -> set[FlagType]

Find all accepted flags containing a specific substring.

Useful for discovering related data flags or filtering flags by category.

Parameters:

Name Type Description Default
x str

Substring to search for in flag names

required
match_case bool

If True, performs case-sensitive search. Default is False.

False

Returns:

Type Description
set[FlagType]

Set of accepted flags containing the substring

Examples:

>>> sorted(dataset.get_accepted_flags_containing_x('line.results.flow_sent'))
['Line.Results.flow_sent_down', 'Line.Results.flow_sent_net', 'Line.Results.flow_sent_up']
>>> dataset.get_accepted_flags_containing_x('line.results', match_case=True)
set()
Source code in src/mesqual/datasets/dataset.py
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
def get_accepted_flags_containing_x(self, x: str, match_case: bool = False) -> set[FlagType]:
    """
    Find all accepted flags containing a specific substring.

    Useful for discovering related data flags or filtering flags by category.

    Args:
        x: Substring to search for in flag names
        match_case: If True, performs case-sensitive search. Default is False.

    Returns:
        Set of accepted flags containing the substring

    Examples:
        >>> sorted(dataset.get_accepted_flags_containing_x('line.results.flow_sent'))
        ['Line.Results.flow_sent_down', 'Line.Results.flow_sent_net', 'Line.Results.flow_sent_up']
        >>> dataset.get_accepted_flags_containing_x('line.results', match_case=True)
        set()
    """
    return {f for f in self.accepted_flags if FlagDescriber.name_contains(f, x, match_case)}

flag_is_accepted

flag_is_accepted(flag: FlagType) -> bool

Boolean check whether a flag is accepted by the Dataset.

True for a flag in accepted_flags and for a flag of a flag family spec of this dataset. Uses only declarations, never source data.

Source code in src/mesqual/datasets/dataset.py
989
990
991
992
993
994
995
996
997
998
def flag_is_accepted(self, flag: FlagType) -> bool:
    """
    Boolean check whether a flag is accepted by the Dataset.

    `True` for a flag in `accepted_flags` and for a flag of a flag family
    spec of this dataset. Uses only declarations, never source data.
    """
    if flag in self.accepted_flags:
        return True
    return any(spec.is_family and spec.covers(flag) for spec in self.flag_specs)

add_kpis_from_definitions

add_kpis_from_definitions(kpi_definitions: KPIDefinition | list[KPIDefinition])

Generate and add KPIs from one or more KPI definitions.

KPI definitions are templates that generate concrete KPI instances based on the dataset's structure. This method processes definitions and adds the resulting KPIs to the dataset's KPI collection.

Parameters:

Name Type Description Default
kpi_definitions KPIDefinition | list[KPIDefinition]

Single KPIDefinition or list of definitions. Each definition's generate_kpis() method is called with this dataset to produce KPI instances.

required

Examples:

>>> from mesqual.kpis import Aggregations, FlagAggKPIBuilder
>>> definitions = FlagAggKPIBuilder().for_flag('Country.Results.price').with_aggregation(Aggregations.Mean).build()
>>> dataset.add_kpis_from_definitions(definitions)
>>> len(dataset.kpi_collection)
2
See Also
Source code in src/mesqual/datasets/dataset.py
1032
1033
1034
1035
1036
1037
1038
1039
1040
1041
1042
1043
1044
1045
1046
1047
1048
1049
1050
1051
1052
1053
1054
1055
1056
1057
1058
1059
1060
1061
1062
1063
1064
def add_kpis_from_definitions(self, kpi_definitions: KPIDefinition | list[KPIDefinition]):
    """
    Generate and add KPIs from one or more KPI definitions.

    KPI definitions are templates that generate concrete KPI instances
    based on the dataset's structure. This method processes definitions
    and adds the resulting KPIs to the dataset's KPI collection.

    Args:
        kpi_definitions: Single KPIDefinition or list of definitions.
            Each definition's generate_kpis() method is called with
            this dataset to produce KPI instances.

    Examples:
        >>> from mesqual.kpis import Aggregations, FlagAggKPIBuilder
        >>> definitions = FlagAggKPIBuilder().for_flag('Country.Results.price').with_aggregation(Aggregations.Mean).build()
        >>> dataset.add_kpis_from_definitions(definitions)
        >>> len(dataset.kpi_collection)
        2

    See Also:
        - [`add_kpi()`][mesqual.datasets.Dataset.add_kpi]: add one KPI.
        - [`add_kpis()`][mesqual.datasets.Dataset.add_kpis]: add several KPIs.
    """
    from mesqual.kpis.definitions.base import KPIDefinition

    if isinstance(kpi_definitions, KPIDefinition):
        kpis = kpi_definitions.generate_kpis(self)
        self.add_kpis(kpis)
    else:
        for kpi_def in kpi_definitions:
            kpis = kpi_def.generate_kpis(self)
            self.add_kpis(kpis)

add_kpis

add_kpis(kpis: Iterable[KPI])

Add multiple KPIs to this dataset's KPI collection.

Parameters:

Name Type Description Default
kpis Iterable[KPI]

Iterable of KPI instances, factories, or classes to add

required
Source code in src/mesqual/datasets/dataset.py
1066
1067
1068
1069
1070
1071
1072
1073
1074
1075
1076
1077
1078
1079
1080
1081
1082
1083
def add_kpis(self, kpis: Iterable[KPI]):
    """
    Add multiple KPIs to this dataset's KPI collection.

    Args:
        kpis: Iterable of KPI instances, factories, or classes to add
    """
    duplicates = []
    for kpi in kpis:
        if kpi in self.kpi_collection:
            duplicates.append(kpi)
        else:
            self.add_kpi(kpi)
    if duplicates:
        _num_duplicates = len(duplicates)
        logger.warning(
            f'{_num_duplicates} duplicates found and not added again or overwritten in {self.name}. ({duplicates[:3]}...)'
        )

add_kpi

add_kpi(kpi: KPI)

Add a single KPI to this dataset's KPI collection.

Parameters:

Name Type Description Default
kpi KPI

KPI instance, factory, or class to add

required
Source code in src/mesqual/datasets/dataset.py
1085
1086
1087
1088
1089
1090
1091
1092
def add_kpi(self, kpi: KPI):
    """
    Add a single KPI to this dataset's KPI collection.

    Args:
        kpi: KPI instance, factory, or class to add
    """
    self.kpi_collection.add(kpi)

clear_kpi_collection

clear_kpi_collection()

Clear the KPI collection.

Source code in src/mesqual/datasets/dataset.py
1094
1095
1096
1097
1098
def clear_kpi_collection(self):
    """Clear the KPI collection."""
    from mesqual.kpis.collection import KPICollection

    self.kpi_collection = KPICollection()

get_attributes_series

get_attributes_series() -> Series

Convert dataset attributes to a pandas Series.

Creates a Series with attribute names as the index and attribute values as data. The Series name is set to the dataset name, making it suitable for concatenation with other datasets' attribute series.

Returns:

Type Description
Series

pd.Series: Series containing attribute values, indexed by attribute names, with the dataset name as the Series name

Examples:

>>> series = dataset.get_attributes_series()
>>> series.name, series['days']
('base', 5)
See Also
Source code in src/mesqual/datasets/dataset.py
1123
1124
1125
1126
1127
1128
1129
1130
1131
1132
1133
1134
1135
1136
1137
1138
1139
1140
1141
1142
1143
1144
1145
def get_attributes_series(self) -> pd.Series:
    """
    Convert dataset attributes to a pandas Series.

    Creates a Series with attribute names as the index and attribute
    values as data. The Series name is set to the dataset name, making
    it suitable for concatenation with other datasets' attribute series.

    Returns:
        pd.Series: Series containing attribute values, indexed by attribute
            names, with the dataset name as the Series name

    Examples:
        >>> series = dataset.get_attributes_series()
        >>> series.name, series['days']
        ('base', 5)

    See Also:
        - [`attributes`][mesqual.datasets.Dataset.attributes] - Access raw attributes dictionary
        - [`set_attributes()`][mesqual.datasets.Dataset.set_attributes] - Set attribute values
    """
    att_series = pd.Series(self.attributes, name=self.name)
    return att_series

set_attributes

set_attributes(**kwargs: bool | int | float | str) -> None

Set one or more metadata attributes on this dataset.

Attributes are key-value pairs that store scenario metadata. They must use string keys and primitive values (bool, int, float, str) to ensure serializability and consistent comparison behavior.

Parameters:

Name Type Description Default
**kwargs bool | int | float | str

Attribute key-value pairs to set. Keys must be strings, values must be bool, int, float, or str.

{}

Raises:

Type Description
TypeError

If any key is not a string

TypeError

If any value is not bool, int, float, or str

Examples:

>>> dataset.set_attributes(year=2030, includes_nuclear=True)
>>> dataset.attributes['year']
2030
See Also
Source code in src/mesqual/datasets/dataset.py
1147
1148
1149
1150
1151
1152
1153
1154
1155
1156
1157
1158
1159
1160
1161
1162
1163
1164
1165
1166
1167
1168
1169
1170
1171
1172
def set_attributes(self, **kwargs: bool | int | float | str) -> None:
    """
    Set one or more metadata attributes on this dataset.

    Attributes are key-value pairs that store scenario metadata. They must
    use string keys and primitive values (bool, int, float, str) to ensure
    serializability and consistent comparison behavior.

    Args:
        **kwargs: Attribute key-value pairs to set. Keys must be strings,
            values must be bool, int, float, or str.

    Raises:
        TypeError: If any key is not a string
        TypeError: If any value is not bool, int, float, or str

    Examples:
        >>> dataset.set_attributes(year=2030, includes_nuclear=True)
        >>> dataset.attributes['year']
        2030

    See Also:
        - [`attributes`][mesqual.datasets.Dataset.attributes] - Access attributes dictionary
        - [`get_attributes_series()`][mesqual.datasets.Dataset.get_attributes_series] - Convert to pandas Series
    """
    self._add_attributes(kwargs)

fetch_multiple_flags_and_concat

fetch_multiple_flags_and_concat(flags: Iterable[FlagType], flag_concat_axis: int = 1, concat_level_name: str = 'variable', concat_level_at_top: bool = True, config: dict | DatasetConfigType = None, *, granularity: str | Timedelta | None | Unset = UNSET, **kwargs: Any) -> Series | DataFrame

Fetch multiple flags and concatenate results into a single DataFrame.

Convenience method for retrieving data from multiple flags and combining them into a single DataFrame with a MultiIndex. Useful for comparative analysis of multiple variables or creating wide-format data structures.

All results must have the same shape of axes. Before the concat, each result is checked against the first one: both are a Series or both are a DataFrame, and each axis has the same number of levels and the same index type per level (for example DatetimeIndex). A Series has only its index to check.

Parameters:

Name Type Description Default
flags Iterable[FlagType]

Iterable of flags to fetch and concatenate

required
flag_concat_axis int

Axis along which to concatenate (0=rows, 1=columns). Default is 1 (columns). It is not the concat_axis of a concat collection, which is fixed when the collection is built.

1
concat_level_name str

Name for the new MultiIndex level identifying the source flag. Default is 'variable'.

'variable'
concat_level_at_top bool

If True, the flag level is the outermost level in the MultiIndex. If False, it's moved to the innermost level. Default is True.

True
config dict | DatasetConfigType

Optional configuration override (see fetch() for details)

None
granularity str | Timedelta | None | Unset

Target time resolution, as in fetch.

UNSET
**kwargs Any

Parameters. Each fetch gets the flag parameters that its spec declares, and the dataset parameters of this dataset.

{}

Returns:

Type Description
Series | DataFrame

DataFrame with concatenated data and a MultiIndex identifying the source flag for each section

Raises:

Type Description
ValueError

If two results differ in kind, number of levels or index type on an axis. The message names both flags.

ValueError

A keyword argument is no flag parameter of any of the flags and no dataset parameter of this dataset.

Examples:

>>> demand_and_gas = dataset.fetch_multiple_flags_and_concat(
...     ['Node.Results.demand_total', 'Node.Results.generation_gas'], concat_level_name='variable',
... )
>>> list(demand_and_gas.columns.names)
['variable', 'Node']
See Also
  • fetch() - Single flag data retrieval
Source code in src/mesqual/datasets/dataset.py
1356
1357
1358
1359
1360
1361
1362
1363
1364
1365
1366
1367
1368
1369
1370
1371
1372
1373
1374
1375
1376
1377
1378
1379
1380
1381
1382
1383
1384
1385
1386
1387
1388
1389
1390
1391
1392
1393
1394
1395
1396
1397
1398
1399
1400
1401
1402
1403
1404
1405
1406
1407
1408
1409
1410
1411
1412
1413
1414
1415
1416
1417
1418
1419
1420
1421
1422
1423
1424
1425
1426
1427
1428
1429
1430
def fetch_multiple_flags_and_concat(
    self,
    flags: Iterable[FlagType],
    flag_concat_axis: int = 1,
    concat_level_name: str = 'variable',
    concat_level_at_top: bool = True,
    config: dict | DatasetConfigType = None,
    *,
    granularity: str | pd.Timedelta | None | Unset = UNSET,
    **kwargs: Any,
) -> Union[pd.Series, pd.DataFrame]:
    """
    Fetch multiple flags and concatenate results into a single DataFrame.

    Convenience method for retrieving data from multiple flags and combining
    them into a single DataFrame with a MultiIndex. Useful for comparative
    analysis of multiple variables or creating wide-format data structures.

    All results must have the same shape of axes. Before the concat, each
    result is checked against the first one: both are a `Series` or both
    are a `DataFrame`, and each axis has the same number of levels and the
    same index type per level (for example `DatetimeIndex`). A `Series`
    has only its index to check.

    Args:
        flags: Iterable of flags to fetch and concatenate
        flag_concat_axis: Axis along which to concatenate (0=rows,
            1=columns). Default is 1 (columns). It is not the
            `concat_axis` of a concat collection, which is fixed when
            the collection is built.
        concat_level_name: Name for the new MultiIndex level identifying
            the source flag. Default is 'variable'.
        concat_level_at_top: If True, the flag level is the outermost level
            in the MultiIndex. If False, it's moved to the innermost level.
            Default is True.
        config: Optional configuration override (see fetch() for details)
        granularity: Target time resolution, as in `fetch`.
        **kwargs: Parameters. Each fetch gets the flag parameters that
            its spec declares, and the dataset parameters of this dataset.

    Returns:
        DataFrame with concatenated data and a MultiIndex identifying the
            source flag for each section

    Raises:
        ValueError: If two results differ in kind, number of levels or index
            type on an axis. The message names both flags.
        ValueError: A keyword argument is no flag parameter of any of the
            flags and no dataset parameter of this dataset.

    Examples:
        >>> demand_and_gas = dataset.fetch_multiple_flags_and_concat(
        ...     ['Node.Results.demand_total', 'Node.Results.generation_gas'], concat_level_name='variable',
        ... )
        >>> list(demand_and_gas.columns.names)
        ['variable', 'Node']

    See Also:
        - [`fetch()`][mesqual.datasets.Dataset.fetch] - Single flag data retrieval
    """
    flags = list(flags)
    self._check_declared_by_any(flags, kwargs)
    dfs = {
        str(flag): self.fetch(flag, config, granularity=granularity, **self._kwargs_for(flag, kwargs))
        for flag in flags
    }
    self._check_same_axes(dfs)
    df = pd.concat(
        dfs,
        axis=flag_concat_axis,
        names=[concat_level_name],
    )
    if not concat_level_at_top:
        df = move_top_level_last(df, flag_concat_axis)
    return df

get_flag_type classmethod

get_flag_type() -> Type[FlagType]

Get the flag type class for this dataset type.

Returns the type used for data flags in this dataset class. Subclasses can override to specify a custom flag type for type checking and validation.

Returns:

Type Description
Type[FlagType]

Type[FlagType]: The flag type class (default: FlagTypeProtocol)

Note

Override in subclasses that use custom flag types.

Source code in src/mesqual/datasets/dataset.py
1470
1471
1472
1473
1474
1475
1476
1477
1478
1479
1480
1481
1482
1483
1484
1485
1486
1487
@classmethod
def get_flag_type(cls) -> Type[FlagType]:
    """
    Get the flag type class for this dataset type.

    Returns the type used for data flags in this dataset class. Subclasses
    can override to specify a custom flag type for type checking and
    validation.

    Returns:
        Type[FlagType]: The flag type class (default: FlagTypeProtocol)

    Note:
        Override in subclasses that use custom flag types.
    """
    from mesqual.flag.flag import FlagTypeProtocol

    return FlagTypeProtocol

get_config_type classmethod

get_config_type() -> Type[DatasetConfigType]

Get the configuration type class for this dataset type.

Returns the DatasetConfig subclass used by this dataset. Platform interfaces typically override this to return their extended config class with platform-specific options.

Returns:

Type Description
Type[DatasetConfigType]

Type[DatasetConfigType]: The config type class (default: DatasetConfig)

Examples:

>>> from mesqual.testing import DummyDataset
>>> DummyDataset.get_config_type()
<class 'mesqual.testing.dataset.DummyConfig'>
Note

Override in platform dataset subclasses to return platform-specific config types with additional options.

Source code in src/mesqual/datasets/dataset.py
1489
1490
1491
1492
1493
1494
1495
1496
1497
1498
1499
1500
1501
1502
1503
1504
1505
1506
1507
1508
1509
1510
1511
1512
@classmethod
def get_config_type(cls) -> Type[DatasetConfigType]:
    """
    Get the configuration type class for this dataset type.

    Returns the DatasetConfig subclass used by this dataset. Platform
    interfaces typically override this to return their extended config
    class with platform-specific options.

    Returns:
        Type[DatasetConfigType]: The config type class (default: DatasetConfig)

    Examples:
        >>> from mesqual.testing import DummyDataset
        >>> DummyDataset.get_config_type()
        <class 'mesqual.testing.dataset.DummyConfig'>

    Note:
        Override in platform dataset subclasses to return platform-specific
        config types with additional options.
    """
    from mesqual.datasets.dataset_config import DatasetConfig

    return DatasetConfig

describe_config

describe_config() -> DataFrame

Describe the config fields of this dataset, one row per field.

Returns:

Type Description
DataFrame

DataFrame indexed by field name with the columns type, default, value (effective value without a fetch-time config), layer (the layer that set the value: 'default', 'class <Name>' or 'instance') and description.

Examples:

>>> dataset.describe_config().loc['use_cache', ['value', 'layer']].tolist()
[True, 'default']
Source code in src/mesqual/datasets/dataset.py
1544
1545
1546
1547
1548
1549
1550
1551
1552
1553
1554
1555
1556
1557
1558
1559
1560
1561
1562
1563
1564
1565
1566
1567
1568
1569
1570
1571
1572
1573
1574
def describe_config(self) -> pd.DataFrame:
    """Describe the config fields of this dataset, one row per field.

    Returns:
        DataFrame indexed by field name with the columns `type`,
            `default`, `value` (effective value without a fetch-time config),
            `layer` (the layer that set the value: `'default'`,
            `'class <Name>'` or `'instance'`) and `description`.

    Examples:
        >>> dataset.describe_config().loc['use_cache', ['value', 'layer']].tolist()
        [True, 'default']
    """
    from mesqual.datasets.dataset_config import DatasetConfigManager

    layers = DatasetConfigManager.get_layers(type(self), self._config)
    effective = self.instance_config
    rows = {}
    for field in dataclasses.fields(effective):
        setting_layer = 'default'
        for label, config in layers[1:]:
            if field.name in config.explicit_fields:
                setting_layer = label
        rows[field.name] = {
            'type': field.type if isinstance(field.type, str) else getattr(field.type, '__name__', str(field.type)),
            'default': effective._field_default(field.name),
            'value': getattr(effective, field.name),
            'layer': setting_layer,
            'description': field.metadata.get('description', ''),
        }
    return pd.DataFrame.from_dict(rows, orient='index')

set_instance_config

set_instance_config(config: DatasetConfigType) -> None

Replace the instance-level configuration for this dataset.

Sets the configuration that will be merged with class-level defaults to produce the effective configuration used by fetch().

Parameters:

Name Type Description Default
config DatasetConfigType

New configuration object to use for this instance

required

Examples:

>>> from mesqual.datasets import DatasetConfig
>>> dataset.set_instance_config(DatasetConfig(use_cache=False, auto_sort_index=False))
See Also
Source code in src/mesqual/datasets/dataset.py
1576
1577
1578
1579
1580
1581
1582
1583
1584
1585
1586
1587
1588
1589
1590
1591
1592
1593
1594
1595
def set_instance_config(self, config: DatasetConfigType) -> None:
    """
    Replace the instance-level configuration for this dataset.

    Sets the configuration that will be merged with class-level defaults
    to produce the effective configuration used by fetch().

    Args:
        config: New configuration object to use for this instance

    Examples:
        >>> from mesqual.datasets import DatasetConfig
        >>> dataset.set_instance_config(DatasetConfig(use_cache=False, auto_sort_index=False))

    See Also:
        - [`instance_config`][mesqual.datasets.Dataset.instance_config] - Get the effective configuration
        - [`set_instance_config_kwargs()`][mesqual.datasets.Dataset.set_instance_config_kwargs] - Update settings
        - [`set_class_config()`][mesqual.datasets.Dataset.set_class_config] - Set class-level defaults
    """
    self._config = config

merge_instance_config

merge_instance_config(config: DatasetConfigType) -> None

Merge config on top of the instance config of this dataset.

Uses DatasetConfig.merge on copies: a field that config sets explicitly wins, and the other explicit fields of the instance config stay. Without an instance config, a copy of config becomes the instance config. Later changes to config do not reach the dataset.

Parameters:

Name Type Description Default
config DatasetConfigType

Upper layer for the instance config.

required

Examples:

>>> from mesqual.datasets import DatasetConfig
>>> dataset.set_instance_config(DatasetConfig(use_cache=False))
>>> dataset.merge_instance_config(DatasetConfig(granularity='1h'))
>>> dataset.instance_config.use_cache, dataset.instance_config.granularity
(False, '1h')
See Also
Source code in src/mesqual/datasets/dataset.py
1597
1598
1599
1600
1601
1602
1603
1604
1605
1606
1607
1608
1609
1610
1611
1612
1613
1614
1615
1616
1617
1618
1619
1620
def merge_instance_config(self, config: DatasetConfigType) -> None:
    """Merge `config` on top of the instance config of this dataset.

    Uses `DatasetConfig.merge` on copies: a field that `config` sets
    explicitly wins, and the other explicit fields of the instance
    config stay. Without an instance config, a copy of `config` becomes
    the instance config. Later changes to `config` do not reach the
    dataset.

    Args:
        config: Upper layer for the instance config.

    Examples:
        >>> from mesqual.datasets import DatasetConfig
        >>> dataset.set_instance_config(DatasetConfig(use_cache=False))
        >>> dataset.merge_instance_config(DatasetConfig(granularity='1h'))
        >>> dataset.instance_config.use_cache, dataset.instance_config.granularity
        (False, '1h')

    See Also:
        - [`set_instance_config()`][mesqual.datasets.Dataset.set_instance_config] - Replace the config
    """
    upper = copy.deepcopy(config)
    self._config = upper if self._config is None else copy.deepcopy(self._config).merge(upper)

set_instance_config_kwargs

set_instance_config_kwargs(**kwargs: Any) -> None

Update individual configuration settings on this instance.

Modifies specific attributes of the existing instance configuration without replacing the entire config object. Useful for tweaking individual settings.

Parameters:

Name Type Description Default
**kwargs Any

Configuration attribute names and values to set

{}

Examples:

>>> dataset.set_instance_config_kwargs(use_cache=False)
>>> dataset.instance_config.use_cache
False

Creates an instance config of get_config_type() first if the dataset has none.

See Also
Source code in src/mesqual/datasets/dataset.py
1622
1623
1624
1625
1626
1627
1628
1629
1630
1631
1632
1633
1634
1635
1636
1637
1638
1639
1640
1641
1642
1643
1644
1645
1646
1647
1648
def set_instance_config_kwargs(self, **kwargs: Any) -> None:
    """
    Update individual configuration settings on this instance.

    Modifies specific attributes of the existing instance configuration
    without replacing the entire config object. Useful for tweaking
    individual settings.

    Args:
        **kwargs: Configuration attribute names and values to set

    Examples:
        >>> dataset.set_instance_config_kwargs(use_cache=False)
        >>> dataset.instance_config.use_cache
        False

    Creates an instance config of `get_config_type()` first if the
    dataset has none.

    See Also:
        - [`set_instance_config()`][mesqual.datasets.Dataset.set_instance_config] - Replace entire configuration
        - [`instance_config`][mesqual.datasets.Dataset.instance_config] - Get the effective configuration
    """
    if self._config is None:
        self._config = self.get_config_type()()
    for key, value in kwargs.items():
        setattr(self._config, key, value)

set_class_config classmethod

set_class_config(config: DatasetConfigType) -> None

Set the class-level configuration for all instances of this dataset type.

Class-level configuration serves as the default for all instances of this class. Instance-level configuration (set via set_instance_config) can override these defaults.

Parameters:

Name Type Description Default
config DatasetConfigType

Configuration object to use as class-level defaults

required

Examples:

>>> from mesqual.datasets import DatasetConfig
>>> from mesqual.testing import DummyDataset
>>> DummyDataset.set_class_config(DatasetConfig(use_cache=False))
>>> study.scen.get_dataset('high_demand').instance_config.use_cache
False
Note

This affects all instances of the class, including existing ones that haven't overridden the setting at instance level.

See Also
Source code in src/mesqual/datasets/dataset.py
1650
1651
1652
1653
1654
1655
1656
1657
1658
1659
1660
1661
1662
1663
1664
1665
1666
1667
1668
1669
1670
1671
1672
1673
1674
1675
1676
1677
1678
1679
@classmethod
def set_class_config(cls, config: DatasetConfigType) -> None:
    """
    Set the class-level configuration for all instances of this dataset type.

    Class-level configuration serves as the default for all instances of
    this class. Instance-level configuration (set via set_instance_config)
    can override these defaults.

    Args:
        config: Configuration object to use as class-level defaults

    Examples:
        >>> from mesqual.datasets import DatasetConfig
        >>> from mesqual.testing import DummyDataset
        >>> DummyDataset.set_class_config(DatasetConfig(use_cache=False))
        >>> study.scen.get_dataset('high_demand').instance_config.use_cache
        False

    Note:
        This affects all instances of the class, including existing ones
        that haven't overridden the setting at instance level.

    See Also:
        - [`set_instance_config()`][mesqual.datasets.Dataset.set_instance_config] - Override for specific instances
        - [`DatasetConfigManager`][mesqual.datasets.DatasetConfigManager] - Configuration management system
    """
    from mesqual.datasets.dataset_config import DatasetConfigManager

    DatasetConfigManager.set_class_config(cls, config)

is_flag_available

is_flag_available(flag: FlagType) -> bool | None

Return whether this dataset can answer a flag, without reading time series data.

A flag that the dataset does not accept, or a flag of a family with a variant value that is not a value on the dataset, is not available (False). Else:

  • Platform dataset: the _is_available of the answering interpreter decides. If it returns None, the flag is unknown (None) until its first successful fetch from the platform dataset.
  • Collection: the union of the children (available in one child, else unknown in one child, else False). Comparison: the intersection of both children (False if one child is False, else unknown if one child is unknown, else available).
  • Any other dataset: its own _is_available. None counts as available.

The interpreter of a layer flag implements _is_available with the availability of the flags that its fetch reads: the base flag, the base model or border model, and the input_flags of the aggregation. availability and flag_info give the reason.

Returns:

Type Description
bool | None

True, False, or None for unknown.

Source code in src/mesqual/datasets/dataset.py
1681
1682
1683
1684
1685
1686
1687
1688
1689
1690
1691
1692
1693
1694
1695
1696
1697
1698
1699
1700
1701
1702
1703
1704
1705
1706
1707
def is_flag_available(self, flag: FlagType) -> bool | None:
    """Return whether this dataset can answer a flag, without reading time series data.

    A flag that the dataset does not accept, or a flag of a family with
    a variant value that is not a value on the dataset, is not available
    (`False`). Else:

    - Platform dataset: the `_is_available` of the answering
      interpreter decides. If it returns `None`, the flag is unknown
      (`None`) until its first successful fetch from the platform
      dataset.
    - Collection: the union of the children (available in one child,
      else unknown in one child, else `False`). Comparison: the
      intersection of both children (`False` if one child is `False`,
      else unknown if one child is unknown, else available).
    - Any other dataset: its own `_is_available`. `None` counts as
      available.

    The interpreter of a layer flag implements `_is_available` with
    the availability of the flags that its fetch reads: the base flag,
    the base model or border model, and the `input_flags` of the
    aggregation. `availability` and `flag_info` give the reason.

    Returns:
        `True`, `False`, or `None` for unknown.
    """
    return self.availability(flag).available

availability

availability(flag: FlagType) -> Availability

Return whether this dataset can answer a flag, with the reason. Reads no time series data.

The same rules as is_flag_available, which returns the available field of the result. An interpreter uses it on its parent dataset to combine the availability of the flags that it fetches (Availability.of_inputs).

Examples:

>>> from mesqual.testing import DummyDataset, DummyScenario
>>> without_prices = DummyDataset(DummyScenario(missing_tables=('node_price',)))
>>> without_prices.availability('Country.Results.price').reason
"DummyAreaFlags._is_available: unavailable input Node.Results.price (DummyTimeSeriesInterpreter._is_available: table 'node_price' not in the source)"
Source code in src/mesqual/datasets/dataset.py
1709
1710
1711
1712
1713
1714
1715
1716
1717
1718
1719
1720
1721
1722
1723
def availability(self, flag: FlagType) -> Availability:
    """Return whether this dataset can answer a flag, with the reason. Reads no time series data.

    The same rules as `is_flag_available`, which returns the
    `available` field of the result. An interpreter uses it on its
    parent dataset to combine the availability of the flags that it
    fetches (`Availability.of_inputs`).

    Examples:
        >>> from mesqual.testing import DummyDataset, DummyScenario
        >>> without_prices = DummyDataset(DummyScenario(missing_tables=('node_price',)))
        >>> without_prices.availability('Country.Results.price').reason
        "DummyAreaFlags._is_available: unavailable input Node.Results.price (DummyTimeSeriesInterpreter._is_available: table 'node_price' not in the source)"
    """
    return self._availability(flag)

describe_class_flags classmethod

describe_class_flags(contains: str | None = None, layer: str | None = None, item_type: ItemTypeEnum | str | None = None, as_records: bool = False) -> DataFrame | list[dict]

Describe the flag specs that this class declares, one row per spec. Needs no source.

On a platform dataset class: the specs of its answering interpreters. On a dataset or interpreter class: its specs. Flag families show their template, and dimensions with values per dataset show 'resolved per dataset'.

Parameters:

Name Type Description Default
contains str | None

Keep specs whose name or one of whose flags contains this substring, case-insensitive.

None
layer str | None

Keep specs on this area or border layer.

None
item_type ItemTypeEnum | str | None

Keep specs of this item type.

None
as_records bool

Return a list of plain dicts, ready for JSON.

False

Returns:

Type Description
DataFrame | list[dict]

One row per spec, indexed by name, with the columns item_type (Model, TimeSeries or Other), object (the model flag; for a layer spec without model_flag, the model flags of its layers, in the order of layers), unit, quantity_type, params (flag and dataset parameters with their defaults), description, variants and layers.

Raises:

Type Description
TypeError

Called on a collection class other than a platform dataset class, or through an instance of one. A collection has no flags of its own: use describe_flags() on the instance.

See Also
  • describe_flags: the flags of one dataset instance, with resolved flag families and layers, effective parameter defaults and availability. Same index, and the same columns plus some more.

Examples:

>>> from mesqual.testing import DummyDataset
>>> DummyDataset.describe_class_flags(contains='loading')[['object', 'unit']]
                          object      unit
name
Line.Results.loading  Line.Model  per_unit
Source code in src/mesqual/datasets/dataset.py
1846
1847
1848
1849
1850
1851
1852
1853
1854
1855
1856
1857
1858
1859
1860
1861
1862
1863
1864
1865
1866
1867
1868
1869
1870
1871
1872
1873
1874
1875
1876
1877
1878
1879
1880
1881
1882
1883
1884
1885
1886
1887
1888
1889
1890
1891
1892
1893
1894
1895
@classmethod
def describe_class_flags(
    cls,
    contains: str | None = None,
    layer: str | None = None,
    item_type: ItemTypeEnum | str | None = None,
    as_records: bool = False,
) -> pd.DataFrame | list[dict]:
    """Describe the flag specs that this class declares, one row per spec. Needs no source.

    On a platform dataset class: the specs of its answering
    interpreters. On a dataset or interpreter class: its specs.
    Flag families show their template, and dimensions with values per
    dataset show `'resolved per dataset'`.

    Args:
        contains: Keep specs whose name or one of whose flags contains
            this substring, case-insensitive.
        layer: Keep specs on this area or border layer.
        item_type: Keep specs of this item type.
        as_records: Return a list of plain dicts, ready for JSON.

    Returns:
        One row per spec, indexed by name, with the columns `item_type`
            (`Model`, `TimeSeries` or `Other`), `object` (the model flag; for
            a layer spec without `model_flag`, the model flags of its
            layers, in the order of `layers`), `unit`, `quantity_type`, `params` (flag and dataset parameters
            with their defaults), `description`, `variants` and `layers`.

    Raises:
        TypeError: Called on a collection class other than a platform
            dataset class, or through an instance of one. A collection
            has no flags of its own: use `describe_flags()` on the
            instance.

    See Also:
        - [`describe_flags`][mesqual.datasets.Dataset.describe_flags]: the flags of one dataset
          instance, with resolved flag families and layers, effective parameter defaults and
          availability. Same index, and the same columns plus some more.

    Examples:
        >>> from mesqual.testing import DummyDataset
        >>> DummyDataset.describe_class_flags(contains='loading')[['object', 'unit']]
                                  object      unit
        name
        Line.Results.loading  Line.Model  per_unit
    """
    rows = FlagDescriber(cls).class_spec_rows()
    rows = FlagDescriber.filter_rows(rows, contains, layer, item_type)
    return FlagDescriber.output(rows, as_records, hidden=('flags_list',))

describe_flags

describe_flags(contains: str | None = None, layer: str | None = None, item_type: ItemTypeEnum | str | None = None, available_only: bool = True, as_records: bool = False) -> DataFrame | list[dict]

Describe the flags of this dataset, one row per spec. Loads no time series data.

As describe_class_flags, with the values of flag families and layers resolved for this dataset (from model flags), the effective parameter defaults with their source (spec, dataset, config, required, or per dataset on a collection whose children resolve different defaults), and the availability. A collection describes the union of its children.

Parameters:

Name Type Description Default
contains str | None

Keep specs whose name or one of whose flags contains this substring, case-insensitive.

None
layer str | None

Keep specs on this area or border layer.

None
item_type ItemTypeEnum | str | None

Keep specs of this item type.

None
available_only bool

Drop the specs whose flags are all unavailable. Specs with unknown availability stay.

True
as_records bool

Return a list of plain dicts, ready for JSON.

False

Returns:

Type Description
DataFrame | list[dict]

One row per spec, indexed by name, with the columns of describe_class_flags in the same order, and flags (number of flags that are available or unknown), availability (available, unknown, partial if some flags are unavailable, or unavailable) and datasets (for a collection: the children that have the flag, if not all of them).

See Also
  • describe_class_flags: the specs as declared on the class, without a source; no resolved values and no availability.

Examples:

>>> dataset.describe_flags(contains='loading')[['object', 'availability']]
                          object availability
name
Line.Results.loading  Line.Model    available
Source code in src/mesqual/datasets/dataset.py
1897
1898
1899
1900
1901
1902
1903
1904
1905
1906
1907
1908
1909
1910
1911
1912
1913
1914
1915
1916
1917
1918
1919
1920
1921
1922
1923
1924
1925
1926
1927
1928
1929
1930
1931
1932
1933
1934
1935
1936
1937
1938
1939
1940
1941
1942
1943
1944
1945
def describe_flags(
    self,
    contains: str | None = None,
    layer: str | None = None,
    item_type: ItemTypeEnum | str | None = None,
    available_only: bool = True,
    as_records: bool = False,
) -> pd.DataFrame | list[dict]:
    """Describe the flags of this dataset, one row per spec. Loads no time series data.

    As `describe_class_flags`, with the values of flag families and layers
    resolved for this dataset (from model flags), the effective parameter
    defaults with their source (`spec`, `dataset`, `config`, `required`,
    or `per dataset` on a collection whose children resolve different
    defaults), and the availability. A collection describes the union of
    its children.

    Args:
        contains: Keep specs whose name or one of whose flags contains
            this substring, case-insensitive.
        layer: Keep specs on this area or border layer.
        item_type: Keep specs of this item type.
        available_only: Drop the specs whose flags are all unavailable.
            Specs with unknown availability stay.
        as_records: Return a list of plain dicts, ready for JSON.

    Returns:
        One row per spec, indexed by name, with the columns of
            `describe_class_flags` in the same order, and `flags` (number of flags that are
            available or unknown), `availability` (`available`, `unknown`,
            `partial` if some flags are unavailable, or `unavailable`) and
            `datasets` (for a collection: the children that have the flag,
            if not all of them).

    See Also:
        - [`describe_class_flags`][mesqual.datasets.Dataset.describe_class_flags]: the specs as
          declared on the class, without a source; no resolved values and no availability.

    Examples:
        >>> dataset.describe_flags(contains='loading')[['object', 'availability']]
                                  object availability
        name
        Line.Results.loading  Line.Model    available
    """
    rows = FlagDescriber.filter_rows(FlagDescriber(self).flag_rows(), contains, layer, item_type)
    if available_only:
        rows = [row for row in rows if row['availability'] != 'unavailable']
    hidden = ('flags_list',) if self._describe_children() else ('flags_list', 'datasets')
    return FlagDescriber.output(rows, as_records, hidden=hidden)

flag_info

flag_info(flag: FlagType) -> FlagInfo

Return everything mesqual knows about one flag of this dataset, without its data.

Returns:

Type Description
FlagInfo

FlagInfo with the spec and its base, the aggregation, the variant values, the combined description, unit and quantity type with their source, the schema, the parameters with their effective defaults and source, fill_value, the model flag, the availability with its reason, the interpreter chain and the recorded dependencies of the last fetch without arguments.

Raises:

Type Description
UnknownFlagError

No spec declares the flag.

Examples:

>>> dataset.flag_info('Line.Results.flow_received_net').description
'Power flow on the line. Side: where the flow leaves the line, after losses. Direction: up minus down.'
Source code in src/mesqual/datasets/dataset.py
1947
1948
1949
1950
1951
1952
1953
1954
1955
1956
1957
1958
1959
1960
1961
1962
1963
1964
1965
def flag_info(self, flag: FlagType) -> FlagInfo:
    """Return everything mesqual knows about one flag of this dataset, without its data.

    Returns:
        `FlagInfo` with the spec and its base, the aggregation, the
            variant values, the combined description, unit and quantity type
            with their source, the schema, the parameters with their
            effective defaults and source, `fill_value`, the model flag, the
            availability with its reason, the interpreter chain and the
            recorded dependencies of the last fetch without arguments.

    Raises:
        UnknownFlagError: No spec declares the flag.

    Examples:
        >>> dataset.flag_info('Line.Results.flow_received_net').description
        'Power flow on the line. Side: where the flow leaves the line, after losses. Direction: up minus down.'
    """
    return FlagDescriber(self).flag_info(flag)

describe_class_area_layers classmethod

describe_class_area_layers(as_records: bool = False) -> DataFrame | list[dict]

Describe the area layers registered on this class, one row per layer.

Parameters:

Name Type Description Default
as_records bool

Return a list of plain dicts, ready for JSON.

False

Returns:

Type Description
DataFrame | list[dict]

One row per area layer, indexed by name, with the columns index, refers_to (base model and layer model), mapping_columns (the membership column of the base model), flags (number of declared flag specs on the layer) and description. Empty for a class without area layers.

Raises:

Type Description
TypeError

Called on a collection class other than a platform dataset class, or through an instance of one.

See Also
Source code in src/mesqual/datasets/dataset.py
1967
1968
1969
1970
1971
1972
1973
1974
1975
1976
1977
1978
1979
1980
1981
1982
1983
1984
1985
1986
1987
1988
1989
1990
@classmethod
def describe_class_area_layers(cls, as_records: bool = False) -> pd.DataFrame | list[dict]:
    """Describe the area layers registered on this class, one row per layer.

    Args:
        as_records: Return a list of plain dicts, ready for JSON.

    Returns:
        One row per area layer, indexed by name, with the columns
            `index`, `refers_to` (base model and layer model),
            `mapping_columns` (the membership column of the base model),
            `flags` (number of declared flag specs on the layer) and
            `description`. Empty for a class without area layers.

    Raises:
        TypeError: Called on a collection class other than a platform
            dataset class, or through an instance of one.

    See Also:
        - [`describe_class_border_layers`][mesqual.datasets.Dataset.describe_class_border_layers]: the border layers.
    """
    return FlagDescriber.output(
        FlagDescriber(cls).area_layer_rows(), as_records, columns=FlagDescriber.LAYER_COLUMNS
    )

describe_class_border_layers classmethod

describe_class_border_layers(as_records: bool = False) -> DataFrame | list[dict]

Describe the border layers registered on this class, one row per layer.

Parameters:

Name Type Description Default
as_records bool

Return a list of plain dicts, ready for JSON.

False

Returns:

Type Description
DataFrame | list[dict]

One row per border layer, indexed by name, with the columns index, refers_to (area layer and edge model), mapping_columns (the edge ends of the edge model), flags (number of declared flag specs on the layer) and description. Empty for a class without border layers.

Raises:

Type Description
TypeError

Called on a collection class other than a platform dataset class, or through an instance of one.

See Also
Source code in src/mesqual/datasets/dataset.py
1992
1993
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
@classmethod
def describe_class_border_layers(cls, as_records: bool = False) -> pd.DataFrame | list[dict]:
    """Describe the border layers registered on this class, one row per layer.

    Args:
        as_records: Return a list of plain dicts, ready for JSON.

    Returns:
        One row per border layer, indexed by name, with the columns
            `index`, `refers_to` (area layer and edge model),
            `mapping_columns` (the edge ends of the edge model), `flags`
            (number of declared flag specs on the layer) and `description`.
            Empty for a class without border layers.

    Raises:
        TypeError: Called on a collection class other than a platform
            dataset class, or through an instance of one.

    See Also:
        - [`describe_class_area_layers`][mesqual.datasets.Dataset.describe_class_area_layers]: the area layers.
    """
    return FlagDescriber.output(
        FlagDescriber(cls).border_layer_rows(), as_records, columns=FlagDescriber.LAYER_COLUMNS
    )

describe_datasets

describe_datasets(max_depth: int | None = None, as_records: bool = False) -> DataFrame | list[dict]

Describe the tree of this dataset and its children, one row per dataset.

A platform dataset is a leaf: its interpreters are not listed.

Parameters:

Name Type Description Default
max_depth int | None

Number of levels below this dataset. None: the full tree. 1: this dataset and its direct children. 0: this dataset only.

None
as_records bool

Return a list of plain dicts, ready for JSON.

False

Returns:

Type Description
DataFrame | list[dict]

One row per dataset, depth first, indexed by path (the names from this dataset down, joined with /), with the columns name, class, description, attributes, config (the explicit fields of the instance config) and children (all children, also below max_depth).

Raises:

Type Description
ValueError

max_depth is negative.

Examples:

>>> study.scen.describe_datasets(max_depth=1)['class'].tolist()
['DatasetConcatCollection', 'DatasetMergeCollection', 'DummyDataset', 'DummyDataset', 'DummyDataset']
Source code in src/mesqual/datasets/dataset.py
2017
2018
2019
2020
2021
2022
2023
2024
2025
2026
2027
2028
2029
2030
2031
2032
2033
2034
2035
2036
2037
2038
2039
2040
2041
2042
2043
2044
2045
2046
2047
2048
def describe_datasets(
    self,
    max_depth: int | None = None,
    as_records: bool = False,
) -> pd.DataFrame | list[dict]:
    """Describe the tree of this dataset and its children, one row per dataset.

    A platform dataset is a leaf: its interpreters are not listed.

    Args:
        max_depth: Number of levels below this dataset. `None`: the full
            tree. `1`: this dataset and its direct children. `0`: this
            dataset only.
        as_records: Return a list of plain dicts, ready for JSON.

    Returns:
        One row per dataset, depth first, indexed by `path` (the names
            from this dataset down, joined with `/`), with the columns
            `name`, `class`, `description`, `attributes`, `config` (the
            explicit fields of the instance config) and `children` (all
            children, also below `max_depth`).

    Raises:
        ValueError: `max_depth` is negative.

    Examples:
        >>> study.scen.describe_datasets(max_depth=1)['class'].tolist()
        ['DatasetConcatCollection', 'DatasetMergeCollection', 'DummyDataset', 'DummyDataset', 'DummyDataset']
    """
    if max_depth is not None and max_depth < 0:
        raise ValueError(f'max_depth must be None or at least 0, got {max_depth}.')
    return FlagDescriber.output(DatasetTreeDescriber(self).rows(max_depth), as_records, index='path')

DatasetCollection

Bases: Generic[DatasetType, DatasetConfigType, FlagType, FlagIndexType], Dataset[DatasetConfigType, FlagType, FlagIndexType], ABC

Abstract base class for collections of datasets.

A DatasetCollection is a Dataset with child datasets. It has the Dataset API, so a collection can be a child of another collection. It gives access to its children, accepts the flags of all children and runs KPI operations on all children. A collection can be used wherever a Dataset is expected.

Each collection class gives the schema of its results with transform_schema, so flag_info(flag).schema describes the fetched result.

Attributes:

Name Type Description
datasets list[DatasetType]

List of child datasets in this collection

Source code in src/mesqual/datasets/dataset_collection.py
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
class DatasetCollection(
    Generic[DatasetType, DatasetConfigType, FlagType, FlagIndexType],
    Dataset[DatasetConfigType, FlagType, FlagIndexType],
    ABC,
):
    """
    Abstract base class for collections of datasets.

    A DatasetCollection is a Dataset with child datasets. It has the Dataset
    API, so a collection can be a child of another collection. It gives access
    to its children, accepts the flags of all children and runs KPI operations
    on all children. A collection can be used wherever a Dataset is expected.

    Each collection class gives the schema of its results with
    `transform_schema`, so `flag_info(flag).schema` describes the fetched
    result.

    Attributes:
        datasets (list[DatasetType]): List of child datasets in this collection
    """

    _warns_about_unknown_config_fields = False
    _resolves_flag_params = False
    _reads_source = False
    _is_leaf: ClassVar[bool] = False
    """`True`: the describe methods and availability treat this collection as one dataset, without children (a platform dataset)."""

    def __init__(
        self,
        datasets: list[DatasetType] = None,
        name: str = None,
        parent_dataset: Dataset = None,
        attributes: dict = None,
        cache: Cache = None,
        config: DatasetConfigType = None,
    ):
        """Create a collection of child datasets.

        Args:
            datasets: The child datasets. Their names must be unique.
            name: Name of the collection.
            parent_dataset: Optional parent dataset.
            attributes: Attributes of the collection.
            cache: Optional cache of fetch results.
            config: Instance config.

        Raises:
            ValueError: Two children have the same name, or two children
                have different specs with the same name.
        """
        self._union_index: FlagIndex | None = None
        self._union_source: tuple[int, ...] | None = None
        self.datasets: list[DatasetType] = list(datasets) if datasets else []
        super().__init__(
            name=name,
            parent_dataset=parent_dataset,
            attributes=attributes,
            cache=cache,
            config=config,
        )
        self._check_unique_names()

    def _check_unique_names(self) -> None:
        duplicates = sorted(name for name, count in Counter(ds.name for ds in self.datasets).items() if count > 1)
        if duplicates:
            raise ValueError(f'Collection {self.name}: the dataset names {duplicates} are not unique.')

    @property
    def dataset_iterator(self) -> Iterator[DatasetType]:
        for ds in self.datasets:
            yield ds

    @property
    def flag_index(self) -> FlagIndex:
        """The flag index: the union of the flag indexes of the children.

        Raises:
            ValueError: Two children have different specs with the same name.
        """
        if self._flag_index is not None:
            return self._flag_index
        return self._union_of_child_indexes()

    def _union_of_child_indexes(self) -> FlagIndex:
        """The union of the flag indexes of the children, built again when a child index changes."""
        child_indexes = tuple(ds.flag_index for ds in self.datasets)
        if self._union_source != tuple(map(id, child_indexes)):
            self._union_index = FlagIndex.union(child_indexes)
            self._union_source = tuple(map(id, child_indexes))
        return self._union_index

    def _build_flag_index(self) -> FlagIndex | None:
        """Return `None`: the index of a collection is the union of the indexes of its children."""
        return None

    def version_identity(self, flag: FlagType) -> tuple:
        """Return one entry per child that a fetch of `flag` can read from, in order.

        Each entry is a `(class path, name)` pair. A collection gives no
        version of its own data; its children record their own identities.
        The pairs make an entry of the collection stale for an added or
        removed child, a child of another class or name, or a flag that
        becomes available or unavailable on a child. A child replaced by one
        of the same class and name keeps the entry: the name identifies the
        source (see [Cache](../architecture/other-contracts.md#cache)).
        `DatasetConcatCollection.version_identity` adds the concat layout
        first.
        """
        return tuple((ds._class_path(), ds.name) for ds in self._fetchable_children(flag))

    def transform_schema(self, schema: FrameSchema) -> FrameSchema:
        """Return the schema of a result of this collection, from the schema of the result of a child.

        Link, merge, sum and comparison collections keep the schema. A
        concat collection adds its level.
        """
        return schema

    def _result_schema(self, flag: FlagType) -> FrameSchema:
        """The schema of the result of a fetch of `flag`: `transform_schema` of the schema of the first child."""
        if self._is_leaf:
            return super()._result_schema(flag)
        children = self._fetchable_children(flag)
        if not children:
            return super()._result_schema(flag)
        return self.transform_schema(children[0]._result_schema(flag))

    def _check_param_name_clash(self) -> None:
        """Run `_check_children` on the children."""
        self._check_children(self.datasets)

    def _check_children(self, children: list[Dataset]) -> None:
        """Raise `ValueError` if `children`, as the children of this collection, clash.

        The checks of the constructor and of `add_dataset`: a name is a flag
        parameter of one child and a dataset parameter of this collection or
        of a child, or two children have different specs with the same name.
        """
        self._raise_on_param_name_clash(self._flag_param_names_with(children), set(self._dataset_params_with(children)))
        # The children of a platform dataset share one index: read each index object once.
        indexes = {id(ds.flag_index): ds.flag_index for ds in children}.values()
        FlagIndex.union(indexes)

    @property
    def _flag_param_names(self) -> set[str]:
        """Names of the flag parameters of the own specs and of all children, collected recursively."""
        return self._flag_param_names_with(self.datasets)

    def _flag_param_names_with(self, children: Iterable[Dataset]) -> set[str]:
        """Names of the flag parameters of the own specs and of `children`, collected recursively."""
        return super()._flag_param_names.union(*(ds._flag_param_names for ds in children))

    @property
    def _accepted_dataset_params(self) -> dict[str, Dimension]:
        """The `fetch_params` of this class and the dataset parameters of all children, collected recursively.

        `fetch` takes the dataset parameters of the children and passes them
        on, for example `comparison_type` through a concat collection of
        comparisons. A parameter of this class wins over one of a child with
        the same name.
        """
        return self._dataset_params_with(self.datasets)

    def _dataset_params_with(self, children: Iterable[Dataset]) -> dict[str, Dimension]:
        """The `fetch_params` of this class and the dataset parameters of `children`, collected recursively."""
        accepted: dict[str, Dimension] = {}
        for ds in children:
            for name, dimension in ds._accepted_dataset_params.items():
                accepted.setdefault(name, dimension)
        return {**accepted, **super()._accepted_dataset_params}

    def _kwargs_for_child(self, ds: Dataset, flag: FlagType) -> dict:
        """The keyword arguments for the fetch of `flag` from the child `ds`. Call it only in `_fetch` of this collection.

        They come from the parameters of the open fetch of this collection
        that the call or the `param_defaults` of this collection gave
        (`FetchRecord.given_params`): the flag parameters of the spec, and
        the dataset parameters that `ds` accepts. So a value in the call and
        the same value in `param_defaults` reach the child in the same way,
        also a dataset parameter of this collection, for example
        `comparison_type` of the comparisons of a comparison of comparisons.
        A dataset parameter that resolved to the default of its dimension
        does not reach the child, so the child resolves it with its own
        config. A plain child in a mixed collection of scenarios and
        comparisons gets no comparison parameters. A `param_defaults` entry
        for a parameter that this collection does not resolve reaches the
        child through the effective config.

        Raises:
            RuntimeError: No fetch of this collection is open, or a nested
                fetch is open (see `_open_frame`).
        """
        flag_params = {d.name for d in self.flag_index.get_spec(flag).params}
        child_params = ds._accepted_dataset_params
        given = self._open_frame().read_given_params()
        return {k: v for k, v in given.items() if k in flag_params or k in child_params}

    def _open_frame(self) -> FetchFrame:
        """The frame of the fetch of this collection that is open, from `_fetch` of this collection.

        Raises:
            RuntimeError: The innermost open frame is not one of this
                collection: `_fetch` runs outside of `fetch`, or in a thread
                started without `contextvars.copy_context().run(...)`.
        """
        frame = FetchContext.current()
        if frame is None or frame.dataset is not self:
            raise RuntimeError(
                f'{type(self).__name__} {self.name}: no open fetch of this collection. '
                f'Call _kwargs_for_child only in _fetch, in the context of fetch.'
            )
        return frame

    @property
    def attributes(self) -> dict:
        child_dataset_atts = [ds.attributes for ds in self.datasets]
        attributes_that_all_childs_have_in_common = get_intersection_of_dicts(child_dataset_atts)
        return {**attributes_that_all_childs_have_in_common, **self._attributes.copy()}

    def get_merged_kpi_collection(self, deep: bool = True) -> KPICollection:
        """
        Merge KPI collections from all child datasets.

        This method collects KPIs from all child datasets' kpi_collection
        properties and returns a unified collection. Optionally recurses into
        nested DatasetCollections.

        Args:
            deep: If True, recursively merge from nested DatasetCollections

        Returns:
            KPICollection containing all KPIs from all child datasets

        Examples:
            >>> from mesqual.kpis import Aggregations, FlagAggKPIBuilder
            >>> definitions = FlagAggKPIBuilder().for_flag('Country.Results.price').with_aggregation(Aggregations.Mean).build()
            >>> study.scen.add_kpis_from_definitions_to_all_child_datasets(definitions)
            >>> len(study.scen.get_merged_kpi_collection())  # 4 scenarios, 2 countries
            8
        """
        from mesqual.kpis.collection import KPICollection

        merged = KPICollection()

        for ds in self.datasets:
            # Add KPIs from this dataset
            merged.extend(ds.kpi_collection._kpis)

            # Recursively add from nested collections
            if deep and isinstance(ds, DatasetCollection):
                nested_merged = ds.get_merged_kpi_collection(deep=deep)
                merged.extend(nested_merged._kpis)

        return merged

    def add_kpis_from_definitions_to_all_child_datasets(self, kpi_definitions: KPIDefinition | list[KPIDefinition]):
        for ds in self.dataset_iterator:
            ds.add_kpis_from_definitions(kpi_definitions)

    def clear_kpi_collection_for_all_child_datasets(self, deep: bool = True):
        for ds in self.datasets:
            ds.clear_kpi_collection()
            if deep and isinstance(ds, DatasetCollection):
                ds.clear_kpi_collection_for_all_child_datasets(deep=deep)

    @abstractmethod
    def _fetch(self, flag: FlagType, effective_config: DatasetConfigType, **kwargs) -> pd.Series | pd.DataFrame:
        pass

    def flag_is_accepted(self, flag: FlagType) -> bool:
        return any(ds.flag_is_accepted(flag) for ds in self.datasets)

    @staticmethod
    def _fetches_from(ds: Dataset, flag: FlagType) -> bool:
        """`True` if the child `ds` accepts `flag` and the flag is not unavailable there."""
        return ds.flag_is_accepted(flag) and ds.is_flag_available(flag) is not False

    def _fetchable_children(self, flag: FlagType) -> list[DatasetType]:
        """The children that accept `flag` and on which it is not unavailable, in order. Empty if there are none."""
        return [ds for ds in self.datasets if self._fetches_from(ds, flag)]

    def _children_to_fetch(self, flag: FlagType) -> list[DatasetType]:
        """The children that a fetch of `flag` reads from, in order.

        `fetch` raises `ValueError` before this for a flag that no child
        accepts.

        Raises:
            FlagUnavailableError: Children accept the flag, but it is unavailable on all of them.
        """
        fetchable = self._fetchable_children(flag)
        if not fetchable:
            raise FlagUnavailableError(str(flag), self.name, self.availability(flag).reason)
        return fetchable

    def _availability(self, flag: FlagType) -> Availability:
        if self._is_leaf:
            return super()._availability(flag)
        return self._combine_availability({ds.name: ds._availability(flag) for ds in self.datasets})

    _combine_availability = staticmethod(Availability.union)
    """Combine the availability of the children: the union (available in one child, else unknown in one)."""

    def _flag_availability_by_spec(self) -> dict[str, dict[str, Availability]]:
        """Union over the children: per spec and flag, the availability in the children that have the spec."""
        if self._is_leaf:
            return super()._flag_availability_by_spec()
        by_flag: dict[str, dict[str, dict[str, Availability]]] = {}
        missing = Availability(False, 'no flag of the spec')
        for ds in self.datasets:
            for spec_name, flags in ds._flag_availability_by_spec().items():
                for flag, availability in flags.items():
                    by_flag.setdefault(spec_name, {}).setdefault(flag, {})[ds.name] = availability
        return {
            spec_name: {
                flag: self._combine_availability({ds.name: children.get(ds.name, missing) for ds in self.datasets})
                for flag, children in flags.items()
            }
            for spec_name, flags in by_flag.items()
        }

    def _values_dataset(self, flag: FlagType) -> Dataset:
        """The values dataset of the first child that fetches `flag`."""
        if self._is_leaf:
            return super()._values_dataset(flag)
        children = self._fetchable_children(flag)
        return children[0]._values_dataset(flag) if children else self

    @classmethod
    def _describes_class_flags(cls) -> bool:
        return cls._is_leaf

    def _describe_children(self) -> list[Dataset]:
        return [] if self._is_leaf else list(self._child_datasets())

    def _child_datasets(self) -> tuple[Dataset, ...]:
        return tuple(self.datasets)

    @property
    def accepted_flags(self) -> set[FlagType]:
        return nested_union([ds.accepted_flags for ds in self.datasets])

    def get_dataset(self, key: str = None) -> DatasetType:
        if key is None:
            if not self.datasets:
                raise ValueError('No datasets available')
            return self.datasets[0]

        for ds in self.datasets:
            if ds.name == key:
                return ds

        raise KeyError(f"Dataset with name '{key}' not found")

    def add_datasets(self, datasets: Iterable[DatasetType]):
        for ds in datasets:
            self.add_dataset(ds)

    def add_dataset(self, dataset: DatasetType):
        """Add a child dataset. A child with the same name replaces the existing one and logs a warning.

        The collection runs the checks of the constructor on the children
        with the new child. If a check fails, the collection does not change.

        Raises:
            TypeError: The dataset is not of `get_child_dataset_type()`.
            ValueError: A name is a flag parameter of one dataset and a
                dataset parameter (`fetch_params`) of another in the
                collection, or the new child and another child have different
                specs with the same name.
        """
        if not self._accepts_child_type(type(dataset)):
            raise TypeError(f'Can only add data sets of type {self.get_child_dataset_type().__name__}.')
        children = {ds.name: ds for ds in self.datasets}
        replaced = dataset.name in children
        children[dataset.name] = dataset
        self._check_children(list(children.values()))
        if replaced:
            logger.warning(f'Dataset {self.name}: dataset {dataset.name} already in this collection. Replacing it.')
        self.datasets[:] = children.values()

    @classmethod
    def get_child_dataset_type(cls) -> type[DatasetType]:
        return Dataset

    @classmethod
    def _accepts_child_type(cls, dataset_type: type[Dataset]) -> bool:
        """`True` if a child of this type may join the collection: a subclass of `get_child_dataset_type()`."""
        return issubclass(dataset_type, cls.get_child_dataset_type())

    def fetch_merged(
        self,
        flag: FlagType,
        config: dict | DatasetConfigType = None,
        precedence: Literal['first', 'last'] | Sequence[str] = 'first',
        **kwargs: Any,
    ) -> pd.Series | pd.DataFrame:
        """Fetch a flag from all child datasets and merge the results, as `DatasetMergeCollection` does.

        The fetch goes through a temporary `DatasetMergeCollection` of the
        children. It has no cache, so it caches nothing, and its record is
        not kept. The records of the children live on the child datasets.
        Dataset parameters of the children (for example `comparison_type`)
        pass through to them.

        Args:
            flag: The flag.
            config: Fetch-time config, as in `fetch`.
            precedence: Which child wins where two children overlap, as in
                `DatasetMergeCollection`.
            **kwargs: Flag parameters and dataset parameters of the children.
        """
        temp_merge_collection = self.get_merged_dataset_collection(precedence)
        return temp_merge_collection.fetch(flag, config, **kwargs)

    def get_merged_dataset_collection(
        self,
        precedence: Literal['first', 'last'] | Sequence[str] = 'first',
    ) -> 'DatasetMergeCollection':
        """Return a `DatasetMergeCollection` of the children of this collection.

        Args:
            precedence: Which child wins where two children overlap, as in
                `DatasetMergeCollection`.
        """
        return DatasetMergeCollection(
            datasets=self.datasets,
            name=f'{self.name} merged',
            precedence=precedence,
        )

flag_index property

flag_index: FlagIndex

The flag index: the union of the flag indexes of the children.

Raises:

Type Description
ValueError

Two children have different specs with the same name.

__init__

__init__(datasets: list[DatasetType] = None, name: str = None, parent_dataset: Dataset = None, attributes: dict = None, cache: Cache = None, config: DatasetConfigType = None)

Create a collection of child datasets.

Parameters:

Name Type Description Default
datasets list[DatasetType]

The child datasets. Their names must be unique.

None
name str

Name of the collection.

None
parent_dataset Dataset

Optional parent dataset.

None
attributes dict

Attributes of the collection.

None
cache Cache

Optional cache of fetch results.

None
config DatasetConfigType

Instance config.

None

Raises:

Type Description
ValueError

Two children have the same name, or two children have different specs with the same name.

Source code in src/mesqual/datasets/dataset_collection.py
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
def __init__(
    self,
    datasets: list[DatasetType] = None,
    name: str = None,
    parent_dataset: Dataset = None,
    attributes: dict = None,
    cache: Cache = None,
    config: DatasetConfigType = None,
):
    """Create a collection of child datasets.

    Args:
        datasets: The child datasets. Their names must be unique.
        name: Name of the collection.
        parent_dataset: Optional parent dataset.
        attributes: Attributes of the collection.
        cache: Optional cache of fetch results.
        config: Instance config.

    Raises:
        ValueError: Two children have the same name, or two children
            have different specs with the same name.
    """
    self._union_index: FlagIndex | None = None
    self._union_source: tuple[int, ...] | None = None
    self.datasets: list[DatasetType] = list(datasets) if datasets else []
    super().__init__(
        name=name,
        parent_dataset=parent_dataset,
        attributes=attributes,
        cache=cache,
        config=config,
    )
    self._check_unique_names()

version_identity

version_identity(flag: FlagType) -> tuple

Return one entry per child that a fetch of flag can read from, in order.

Each entry is a (class path, name) pair. A collection gives no version of its own data; its children record their own identities. The pairs make an entry of the collection stale for an added or removed child, a child of another class or name, or a flag that becomes available or unavailable on a child. A child replaced by one of the same class and name keeps the entry: the name identifies the source (see Cache). DatasetConcatCollection.version_identity adds the concat layout first.

Source code in src/mesqual/datasets/dataset_collection.py
131
132
133
134
135
136
137
138
139
140
141
142
143
144
def version_identity(self, flag: FlagType) -> tuple:
    """Return one entry per child that a fetch of `flag` can read from, in order.

    Each entry is a `(class path, name)` pair. A collection gives no
    version of its own data; its children record their own identities.
    The pairs make an entry of the collection stale for an added or
    removed child, a child of another class or name, or a flag that
    becomes available or unavailable on a child. A child replaced by one
    of the same class and name keeps the entry: the name identifies the
    source (see [Cache](../architecture/other-contracts.md#cache)).
    `DatasetConcatCollection.version_identity` adds the concat layout
    first.
    """
    return tuple((ds._class_path(), ds.name) for ds in self._fetchable_children(flag))

transform_schema

transform_schema(schema: FrameSchema) -> FrameSchema

Return the schema of a result of this collection, from the schema of the result of a child.

Link, merge, sum and comparison collections keep the schema. A concat collection adds its level.

Source code in src/mesqual/datasets/dataset_collection.py
146
147
148
149
150
151
152
def transform_schema(self, schema: FrameSchema) -> FrameSchema:
    """Return the schema of a result of this collection, from the schema of the result of a child.

    Link, merge, sum and comparison collections keep the schema. A
    concat collection adds its level.
    """
    return schema

get_merged_kpi_collection

get_merged_kpi_collection(deep: bool = True) -> KPICollection

Merge KPI collections from all child datasets.

This method collects KPIs from all child datasets' kpi_collection properties and returns a unified collection. Optionally recurses into nested DatasetCollections.

Parameters:

Name Type Description Default
deep bool

If True, recursively merge from nested DatasetCollections

True

Returns:

Type Description
KPICollection

KPICollection containing all KPIs from all child datasets

Examples:

>>> from mesqual.kpis import Aggregations, FlagAggKPIBuilder
>>> definitions = FlagAggKPIBuilder().for_flag('Country.Results.price').with_aggregation(Aggregations.Mean).build()
>>> study.scen.add_kpis_from_definitions_to_all_child_datasets(definitions)
>>> len(study.scen.get_merged_kpi_collection())  # 4 scenarios, 2 countries
8
Source code in src/mesqual/datasets/dataset_collection.py
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
def get_merged_kpi_collection(self, deep: bool = True) -> KPICollection:
    """
    Merge KPI collections from all child datasets.

    This method collects KPIs from all child datasets' kpi_collection
    properties and returns a unified collection. Optionally recurses into
    nested DatasetCollections.

    Args:
        deep: If True, recursively merge from nested DatasetCollections

    Returns:
        KPICollection containing all KPIs from all child datasets

    Examples:
        >>> from mesqual.kpis import Aggregations, FlagAggKPIBuilder
        >>> definitions = FlagAggKPIBuilder().for_flag('Country.Results.price').with_aggregation(Aggregations.Mean).build()
        >>> study.scen.add_kpis_from_definitions_to_all_child_datasets(definitions)
        >>> len(study.scen.get_merged_kpi_collection())  # 4 scenarios, 2 countries
        8
    """
    from mesqual.kpis.collection import KPICollection

    merged = KPICollection()

    for ds in self.datasets:
        # Add KPIs from this dataset
        merged.extend(ds.kpi_collection._kpis)

        # Recursively add from nested collections
        if deep and isinstance(ds, DatasetCollection):
            nested_merged = ds.get_merged_kpi_collection(deep=deep)
            merged.extend(nested_merged._kpis)

    return merged

add_dataset

add_dataset(dataset: DatasetType)

Add a child dataset. A child with the same name replaces the existing one and logs a warning.

The collection runs the checks of the constructor on the children with the new child. If a check fails, the collection does not change.

Raises:

Type Description
TypeError

The dataset is not of get_child_dataset_type().

ValueError

A name is a flag parameter of one dataset and a dataset parameter (fetch_params) of another in the collection, or the new child and another child have different specs with the same name.

Source code in src/mesqual/datasets/dataset_collection.py
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
def add_dataset(self, dataset: DatasetType):
    """Add a child dataset. A child with the same name replaces the existing one and logs a warning.

    The collection runs the checks of the constructor on the children
    with the new child. If a check fails, the collection does not change.

    Raises:
        TypeError: The dataset is not of `get_child_dataset_type()`.
        ValueError: A name is a flag parameter of one dataset and a
            dataset parameter (`fetch_params`) of another in the
            collection, or the new child and another child have different
            specs with the same name.
    """
    if not self._accepts_child_type(type(dataset)):
        raise TypeError(f'Can only add data sets of type {self.get_child_dataset_type().__name__}.')
    children = {ds.name: ds for ds in self.datasets}
    replaced = dataset.name in children
    children[dataset.name] = dataset
    self._check_children(list(children.values()))
    if replaced:
        logger.warning(f'Dataset {self.name}: dataset {dataset.name} already in this collection. Replacing it.')
    self.datasets[:] = children.values()

fetch_merged

fetch_merged(flag: FlagType, config: dict | DatasetConfigType = None, precedence: Literal['first', 'last'] | Sequence[str] = 'first', **kwargs: Any) -> Series | DataFrame

Fetch a flag from all child datasets and merge the results, as DatasetMergeCollection does.

The fetch goes through a temporary DatasetMergeCollection of the children. It has no cache, so it caches nothing, and its record is not kept. The records of the children live on the child datasets. Dataset parameters of the children (for example comparison_type) pass through to them.

Parameters:

Name Type Description Default
flag FlagType

The flag.

required
config dict | DatasetConfigType

Fetch-time config, as in fetch.

None
precedence Literal['first', 'last'] | Sequence[str]

Which child wins where two children overlap, as in DatasetMergeCollection.

'first'
**kwargs Any

Flag parameters and dataset parameters of the children.

{}
Source code in src/mesqual/datasets/dataset_collection.py
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
def fetch_merged(
    self,
    flag: FlagType,
    config: dict | DatasetConfigType = None,
    precedence: Literal['first', 'last'] | Sequence[str] = 'first',
    **kwargs: Any,
) -> pd.Series | pd.DataFrame:
    """Fetch a flag from all child datasets and merge the results, as `DatasetMergeCollection` does.

    The fetch goes through a temporary `DatasetMergeCollection` of the
    children. It has no cache, so it caches nothing, and its record is
    not kept. The records of the children live on the child datasets.
    Dataset parameters of the children (for example `comparison_type`)
    pass through to them.

    Args:
        flag: The flag.
        config: Fetch-time config, as in `fetch`.
        precedence: Which child wins where two children overlap, as in
            `DatasetMergeCollection`.
        **kwargs: Flag parameters and dataset parameters of the children.
    """
    temp_merge_collection = self.get_merged_dataset_collection(precedence)
    return temp_merge_collection.fetch(flag, config, **kwargs)

get_merged_dataset_collection

get_merged_dataset_collection(precedence: Literal['first', 'last'] | Sequence[str] = 'first') -> 'DatasetMergeCollection'

Return a DatasetMergeCollection of the children of this collection.

Parameters:

Name Type Description Default
precedence Literal['first', 'last'] | Sequence[str]

Which child wins where two children overlap, as in DatasetMergeCollection.

'first'
Source code in src/mesqual/datasets/dataset_collection.py
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
def get_merged_dataset_collection(
    self,
    precedence: Literal['first', 'last'] | Sequence[str] = 'first',
) -> 'DatasetMergeCollection':
    """Return a `DatasetMergeCollection` of the children of this collection.

    Args:
        precedence: Which child wins where two children overlap, as in
            `DatasetMergeCollection`.
    """
    return DatasetMergeCollection(
        datasets=self.datasets,
        name=f'{self.name} merged',
        precedence=precedence,
    )

DatasetLinkCollection

Bases: Generic[DatasetType, DatasetConfigType, FlagType, FlagIndexType], DatasetCollection[DatasetType, DatasetConfigType, FlagType, FlagIndexType]

Routes each flag to the one child dataset that accepts it.

A link collection combines child datasets that each answer a different subset of flags into one dataset. It does not combine scenarios; use DatasetConcatCollection or DatasetMergeCollection for that.

Routing

fetch(flag) goes to the one child that accepts the flag. If no child accepts the flag, or the flag is unavailable on that child, it raises ValueError. Two children that accept the same flag raise AmbiguousOverrideError at construction.

Children reach their siblings through parent_dataset:

  • self.parent_dataset.fetch('other_flag', effective_config)
  • self.parent_dataset.get_dataset_by_type(SomeDataset)

PlatformDataset builds on this class with an interpreter registry. To replace a platform interpreter, register an override with PlatformDataset.register_interpreter(Override, override_interpreter=Original, version=...).

The constructor checks the model_flag references of the union of the child specs. So a time series child may refer to the model spec of a sibling. Fetch such a child through the link collection: a child fetched directly checks its own flag index.

Examples:

class ModelData(Dataset):
    flag_specs = (GENERATOR_MODEL_SPEC,)  # 'Generator.Model'
    ...

class TimeSeriesData(Dataset):
    flag_specs = (GENERATION_SPEC,)  # 'Generator.Results.generation', model_flag='Generator.Model'
    ...

linked = DatasetLinkCollection([ModelData(), TimeSeriesData()], name='base')
linked.fetch('Generator.Results.generation')  # answered by TimeSeriesData
See Also
Source code in src/mesqual/datasets/dataset_collection.py
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
class DatasetLinkCollection(
    Generic[DatasetType, DatasetConfigType, FlagType, FlagIndexType],
    DatasetCollection[DatasetType, DatasetConfigType, FlagType, FlagIndexType],
):
    """
    Routes each flag to the one child dataset that accepts it.

    A link collection combines child datasets that each answer a different
    subset of flags into one dataset. It does not combine scenarios; use
    `DatasetConcatCollection` or `DatasetMergeCollection` for that.

    Routing:
        `fetch(flag)` goes to the one child that accepts the flag. If no
        child accepts the flag, or the flag is unavailable on that child,
        it raises `ValueError`. Two children that accept the same flag
        raise `AmbiguousOverrideError` at construction.

    Children reach their siblings through `parent_dataset`:

    - `self.parent_dataset.fetch('other_flag', effective_config)`
    - `self.parent_dataset.get_dataset_by_type(SomeDataset)`

    `PlatformDataset` builds on this class with an interpreter registry. To
    replace a platform interpreter, register an override with
    `PlatformDataset.register_interpreter(Override, override_interpreter=Original, version=...)`.

    The constructor checks the `model_flag` references of the union of the
    child specs. So a time series child may refer to the model spec of a
    sibling. Fetch such a child through the link collection: a child fetched
    directly checks its own flag index.

    Examples:
        ```python
        class ModelData(Dataset):
            flag_specs = (GENERATOR_MODEL_SPEC,)  # 'Generator.Model'
            ...

        class TimeSeriesData(Dataset):
            flag_specs = (GENERATION_SPEC,)  # 'Generator.Results.generation', model_flag='Generator.Model'
            ...

        linked = DatasetLinkCollection([ModelData(), TimeSeriesData()], name='base')
        linked.fetch('Generator.Results.generation')  # answered by TimeSeriesData
        ```

    See Also:
        - [`Dataset.parent_dataset`][mesqual.datasets.Dataset.parent_dataset]
        - [`PlatformDataset.register_interpreter`][mesqual.datasets.PlatformDataset.register_interpreter]
        - [`get_dataset_by_type()`][mesqual.datasets.DatasetLinkCollection.get_dataset_by_type]
    """

    def __init__(
        self,
        datasets: list[DatasetType],
        name: str = None,
        parent_dataset: Dataset = None,
        attributes: dict = None,
        cache: Cache = None,
        config: DatasetConfigType = None,
    ):
        """Create a link collection.

        The arguments are the same as for `DatasetCollection`.

        Raises:
            AmbiguousOverrideError: Two children accept the same flag.
            ValueError: Two children have the same name or different specs
                with the same name, or a `model_flag` names no model spec.
        """
        super().__init__(
            datasets=datasets,
            name=name,
            parent_dataset=parent_dataset,
            attributes=attributes,
            cache=cache,
            config=config,
        )
        self._raise_if_flags_overlap()
        if self.datasets:
            self.flag_index.check_references()

    def _fetch(self, flag: FlagType, effective_config: DatasetConfigType, **kwargs) -> pd.Series | pd.DataFrame:
        child = self._children_to_fetch(flag)[0]
        return child.fetch(flag, effective_config, **self._kwargs_for_child(child, flag))

    def _raise_if_flags_overlap(self) -> None:
        counts = Counter(flag for ds in self.datasets for flag in ds.accepted_flags)
        duplicates = sorted(str(flag) for flag, count in counts.items() if count > 1)
        if duplicates:
            raise AmbiguousOverrideError(
                f'Link collection {self.name}: several children accept the flags {duplicates}. '
                f'Each flag must route to one child. Use a merge or concat collection to combine '
                f'datasets with the same flags.'
            )

    def get_dataset_by_type(self, ds_type: type[Dataset]) -> DatasetType:
        """Returns instance of child dataset that matches the ds_type."""
        for ds in self.datasets:
            if isinstance(ds, ds_type):
                return ds
        raise KeyError(f'No Dataset of type {ds_type.__name__} found in {self.name}.')

__init__

__init__(datasets: list[DatasetType], name: str = None, parent_dataset: Dataset = None, attributes: dict = None, cache: Cache = None, config: DatasetConfigType = None)

Create a link collection.

The arguments are the same as for DatasetCollection.

Raises:

Type Description
AmbiguousOverrideError

Two children accept the same flag.

ValueError

Two children have the same name or different specs with the same name, or a model_flag names no model spec.

Source code in src/mesqual/datasets/dataset_collection.py
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
def __init__(
    self,
    datasets: list[DatasetType],
    name: str = None,
    parent_dataset: Dataset = None,
    attributes: dict = None,
    cache: Cache = None,
    config: DatasetConfigType = None,
):
    """Create a link collection.

    The arguments are the same as for `DatasetCollection`.

    Raises:
        AmbiguousOverrideError: Two children accept the same flag.
        ValueError: Two children have the same name or different specs
            with the same name, or a `model_flag` names no model spec.
    """
    super().__init__(
        datasets=datasets,
        name=name,
        parent_dataset=parent_dataset,
        attributes=attributes,
        cache=cache,
        config=config,
    )
    self._raise_if_flags_overlap()
    if self.datasets:
        self.flag_index.check_references()

get_dataset_by_type

get_dataset_by_type(ds_type: type[Dataset]) -> DatasetType

Returns instance of child dataset that matches the ds_type.

Source code in src/mesqual/datasets/dataset_collection.py
563
564
565
566
567
568
def get_dataset_by_type(self, ds_type: type[Dataset]) -> DatasetType:
    """Returns instance of child dataset that matches the ds_type."""
    for ds in self.datasets:
        if isinstance(ds, ds_type):
            return ds
    raise KeyError(f'No Dataset of type {ds_type.__name__} found in {self.name}.')

DatasetMergeCollection

Bases: Generic[DatasetType, DatasetConfigType, FlagType, FlagIndexType], DatasetCollection[DatasetType, DatasetConfigType, FlagType, FlagIndexType]

Merges the results of all children that have a flag into one result.

Use it for fragmented data of one scenario, for example simulation runs per month, or a model table from two sources.

The results combine pairwise (combine_dfs): disjoint columns with overlapping index concat on columns; shared columns with disjoint index concat on rows; else combine_first. Where two children overlap, precedence decides which values stay. With a name list as precedence, the columns of disjoint results also follow the order of the list, and so do their rows if auto_sort_index is False.

Examples:

Two periods of one scenario:

>>> from mesqual.testing import DummyDataset, DummyScenario
>>> first, second = DummyScenario(days=4).split('2024-01-03')
>>> merged = DatasetMergeCollection([DummyDataset(first), DummyDataset(second)], name='base')
>>> len(merged.fetch('Node.Results.price'))  # 4 days of hours
96

A rerun of the first day replaces the values of the base run there:

>>> base = DummyDataset(DummyScenario(days=4))
>>> rerun = DummyDataset(DummyScenario(name='rerun', days=1, demand_factor=1.2))
>>> patched = DatasetMergeCollection([base, rerun], precedence=['rerun', 'base'])
>>> patched.fetch('Node.Results.demand_total').iloc[0].equals(rerun.fetch('Node.Results.demand_total').iloc[0])
True
Source code in src/mesqual/datasets/dataset_collection.py
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
class DatasetMergeCollection(
    Generic[DatasetType, DatasetConfigType, FlagType, FlagIndexType],
    DatasetCollection[DatasetType, DatasetConfigType, FlagType, FlagIndexType],
):
    """Merges the results of all children that have a flag into one result.

    Use it for fragmented data of one scenario, for example simulation runs
    per month, or a model table from two sources.

    The results combine pairwise (`combine_dfs`): disjoint columns with
    overlapping index concat on columns; shared columns with disjoint index
    concat on rows; else `combine_first`. Where two children overlap,
    `precedence` decides which values stay. With a name list as
    `precedence`, the columns of disjoint results also follow the order of
    the list, and so do their rows if `auto_sort_index` is `False`.

    Examples:
        Two periods of one scenario:

        >>> from mesqual.testing import DummyDataset, DummyScenario
        >>> first, second = DummyScenario(days=4).split('2024-01-03')
        >>> merged = DatasetMergeCollection([DummyDataset(first), DummyDataset(second)], name='base')
        >>> len(merged.fetch('Node.Results.price'))  # 4 days of hours
        96

        A rerun of the first day replaces the values of the base run there:

        >>> base = DummyDataset(DummyScenario(days=4))
        >>> rerun = DummyDataset(DummyScenario(name='rerun', days=1, demand_factor=1.2))
        >>> patched = DatasetMergeCollection([base, rerun], precedence=['rerun', 'base'])
        >>> patched.fetch('Node.Results.demand_total').iloc[0].equals(rerun.fetch('Node.Results.demand_total').iloc[0])
        True
    """

    def __init__(
        self,
        datasets: list[DatasetType],
        name: str = None,
        parent_dataset: Dataset = None,
        attributes: dict = None,
        cache: Cache = None,
        config: DatasetConfigType = None,
        precedence: Literal['first', 'last'] | Sequence[str] = 'first',
    ):
        """Create a merge collection.

        Args:
            datasets: The child datasets.
            name: Name of the collection.
            parent_dataset: Optional parent dataset.
            attributes: Attributes of the collection.
            cache: Optional cache of fetch results.
            config: Instance config.
            precedence: Which child wins where two children overlap:
                `'first'` (default) the earlier child, `'last'` the later
                child, or a list (or other iterable) of all child names,
                highest precedence first.

        Raises:
            ValueError: `precedence` is not `'first'`, `'last'` or a list
                with each child name once.
        """
        super().__init__(
            datasets=datasets,
            name=name,
            parent_dataset=parent_dataset,
            attributes=attributes,
            cache=cache,
            config=config,
        )
        is_name_iterable = isinstance(precedence, Iterable) and not isinstance(precedence, str)
        self.precedence = list(precedence) if is_name_iterable else precedence
        self._check_precedence([ds.name for ds in self.datasets])

    def add_dataset(self, dataset: DatasetType):
        """Add a child dataset. A child with the same name replaces the existing one and logs a warning.

        With a name list as `precedence`, first set `precedence` to a list
        that also names the new child, then add the child.

        Raises:
            TypeError: The dataset is not of `get_child_dataset_type()`.
            ValueError: `precedence` is a name list that does not name each
                child once with the new child; a name is a flag parameter of
                one dataset and a dataset parameter (`fetch_params`) of
                another in the collection; or the new child and another child
                have different specs with the same name.
        """
        self._check_precedence({ds.name for ds in self.datasets} | {dataset.name})
        super().add_dataset(dataset)

    def _fetch(self, flag: FlagType, effective_config: DatasetConfigType, **kwargs) -> pd.Series | pd.DataFrame:
        children = self._children_by_precedence(self._children_to_fetch(flag))
        return combine_dfs(
            [ds.fetch(flag, effective_config, **self._kwargs_for_child(ds, flag)) for ds in children],
            precedence='last' if self.precedence == 'last' else 'first',
        )

    def _check_precedence(self, child_names: Iterable[str]) -> None:
        """Raise `ValueError` if `precedence` is not `'first'`, `'last'` or a list with each of `child_names` once."""
        if self.precedence in ('first', 'last'):
            return
        child_names = sorted(child_names)
        if not isinstance(self.precedence, list) or sorted(self.precedence) != child_names:
            raise ValueError(
                f"Merge collection {self.name}: precedence must be 'first', 'last' or a list of all child "
                f'names, got {self.precedence!r}. Children: {child_names}.'
            )

    def _children_by_precedence(self, children: list[DatasetType]) -> list[DatasetType]:
        """`children` in the order for `combine_dfs`: as given for `'first'` and `'last'`, else as in the name list.

        Raises:
            ValueError: `precedence` is not `'first'`, `'last'` or a list
                with each child name once.
        """
        self._check_precedence([ds.name for ds in self.datasets])
        if self.precedence in ('first', 'last'):
            return children
        rank = {name: position for position, name in enumerate(self.precedence)}
        return sorted(children, key=lambda ds: rank[ds.name])

__init__

__init__(datasets: list[DatasetType], name: str = None, parent_dataset: Dataset = None, attributes: dict = None, cache: Cache = None, config: DatasetConfigType = None, precedence: Literal['first', 'last'] | Sequence[str] = 'first')

Create a merge collection.

Parameters:

Name Type Description Default
datasets list[DatasetType]

The child datasets.

required
name str

Name of the collection.

None
parent_dataset Dataset

Optional parent dataset.

None
attributes dict

Attributes of the collection.

None
cache Cache

Optional cache of fetch results.

None
config DatasetConfigType

Instance config.

None
precedence Literal['first', 'last'] | Sequence[str]

Which child wins where two children overlap: 'first' (default) the earlier child, 'last' the later child, or a list (or other iterable) of all child names, highest precedence first.

'first'

Raises:

Type Description
ValueError

precedence is not 'first', 'last' or a list with each child name once.

Source code in src/mesqual/datasets/dataset_collection.py
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
def __init__(
    self,
    datasets: list[DatasetType],
    name: str = None,
    parent_dataset: Dataset = None,
    attributes: dict = None,
    cache: Cache = None,
    config: DatasetConfigType = None,
    precedence: Literal['first', 'last'] | Sequence[str] = 'first',
):
    """Create a merge collection.

    Args:
        datasets: The child datasets.
        name: Name of the collection.
        parent_dataset: Optional parent dataset.
        attributes: Attributes of the collection.
        cache: Optional cache of fetch results.
        config: Instance config.
        precedence: Which child wins where two children overlap:
            `'first'` (default) the earlier child, `'last'` the later
            child, or a list (or other iterable) of all child names,
            highest precedence first.

    Raises:
        ValueError: `precedence` is not `'first'`, `'last'` or a list
            with each child name once.
    """
    super().__init__(
        datasets=datasets,
        name=name,
        parent_dataset=parent_dataset,
        attributes=attributes,
        cache=cache,
        config=config,
    )
    is_name_iterable = isinstance(precedence, Iterable) and not isinstance(precedence, str)
    self.precedence = list(precedence) if is_name_iterable else precedence
    self._check_precedence([ds.name for ds in self.datasets])

add_dataset

add_dataset(dataset: DatasetType)

Add a child dataset. A child with the same name replaces the existing one and logs a warning.

With a name list as precedence, first set precedence to a list that also names the new child, then add the child.

Raises:

Type Description
TypeError

The dataset is not of get_child_dataset_type().

ValueError

precedence is a name list that does not name each child once with the new child; a name is a flag parameter of one dataset and a dataset parameter (fetch_params) of another in the collection; or the new child and another child have different specs with the same name.

Source code in src/mesqual/datasets/dataset_collection.py
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
def add_dataset(self, dataset: DatasetType):
    """Add a child dataset. A child with the same name replaces the existing one and logs a warning.

    With a name list as `precedence`, first set `precedence` to a list
    that also names the new child, then add the child.

    Raises:
        TypeError: The dataset is not of `get_child_dataset_type()`.
        ValueError: `precedence` is a name list that does not name each
            child once with the new child; a name is a flag parameter of
            one dataset and a dataset parameter (`fetch_params`) of
            another in the collection; or the new child and another child
            have different specs with the same name.
    """
    self._check_precedence({ds.name for ds in self.datasets} | {dataset.name})
    super().add_dataset(dataset)

DatasetSumCollection

Bases: Generic[DatasetType, DatasetConfigType, FlagType, FlagIndexType], DatasetCollection[DatasetType, DatasetConfigType, FlagType, FlagIndexType]

Sums the time series flags of its children and merges their model flags.

Time series of numeric data add element-wise. An object or row that one child lacks gets the fill_value of the spec before the addition; a spec without fill_value gives NaN there. Model flags merge as in DatasetMergeCollection, the first child wins.

Source code in src/mesqual/datasets/dataset_collection.py
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
class DatasetSumCollection(
    Generic[DatasetType, DatasetConfigType, FlagType, FlagIndexType],
    DatasetCollection[DatasetType, DatasetConfigType, FlagType, FlagIndexType],
):
    """Sums the time series flags of its children and merges their model flags.

    Time series of numeric data add element-wise. An object or row that one
    child lacks gets the `fill_value` of the spec before the addition; a
    spec without `fill_value` gives `NaN` there. Model flags merge as in
    `DatasetMergeCollection`, the first child wins.
    """

    def _fetch(self, flag: FlagType, effective_config: DatasetConfigType, **kwargs) -> pd.Series | pd.DataFrame:
        children = self._children_to_fetch(flag)
        item_type = self.flag_index.get_item_type(flag)
        if item_type not in (ItemTypeEnum.Model, ItemTypeEnum.TimeSeries):
            raise NotImplementedError(
                f'Sum collection {self.name}: flag {flag} has item type {item_type.name}. '
                f'A sum collection sums time series flags and merges model flags only.'
            )
        data = [ds.fetch(flag, effective_config, **self._kwargs_for_child(ds, flag)) for ds in children]
        if item_type is ItemTypeEnum.Model:
            return combine_dfs(data, precedence='first')
        not_numeric = [ds.name for ds, d in zip(children, data) if not pd_is_numeric(d)]
        if not_numeric:
            raise TypeError(
                f'Sum collection {self.name}: flag {flag} has non-numeric data in {not_numeric}. '
                f'A sum collection adds numeric time series only.'
            )
        fill_value = self.flag_index.get_fill_value(flag)
        return reduce(lambda a, b: a.add(b, fill_value=fill_value), data)

DatasetConcatCollection

Bases: Generic[DatasetType, DatasetConfigType, FlagType, FlagIndexType], DatasetCollection[DatasetType, DatasetConfigType, FlagType, FlagIndexType]

Concatenates the results of its child datasets, with a level that names the child.

It fetches the same flag from each child dataset and concatenates the results into one DataFrame or Series. The scenarios and the comparisons of a study are concat collections.

MultiIndex Structure

The result has a new level named concat_level_name (default dataset) with the child names, at the top of the axis concat_axis: 1 (default) concats the child results as columns, 0 stacks them as rows. The axis is fixed per collection. Each nested concat collection uses its own axis. concat_top=False puts it last. The join is outer, so children with different horizons give NaN outside their horizon.

The level names of the children must match on each axis. A child whose result lacks levels of the longest child result, with the other levels in the same order, gets these levels with the value ''. So a comparison with comparison_type=BOTH concats with scenarios. If the child result and the longest child result have unique level names, a child result with only level names of the longest child result, in another order, raises ValueError that names both children and both orders. Other differences raise NotImplementedError.

With auto_sort_index, a concat on the columns sorts the rows. A concat on the rows keeps the order of the children; each child result is sorted already.

Examples:

>>> prices = study.scen.fetch('Node.Results.price')
>>> list(prices.columns.names)
['dataset', 'Node']
>>> base_prices = prices['base']

Stack the scenarios as rows:

>>> stacked = DatasetConcatCollection(study.scen.datasets, name='stacked', concat_axis=0)
>>> list(stacked.fetch('Node.Results.price').index.names)
['dataset', 'time']
Child parameters

It passes each child the dataset parameters that the child accepts and that the call or param_defaults gave, for example comparison_type of comparisons.

Source code in src/mesqual/datasets/dataset_collection.py
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
class DatasetConcatCollection(
    Generic[DatasetType, DatasetConfigType, FlagType, FlagIndexType],
    DatasetCollection[DatasetType, DatasetConfigType, FlagType, FlagIndexType],
):
    """
    Concatenates the results of its child datasets, with a level that names the child.

    It fetches the same flag from each child dataset and concatenates the
    results into one DataFrame or Series. The scenarios and the comparisons
    of a study are concat collections.

    MultiIndex Structure:
        The result has a new level named `concat_level_name` (default
        `dataset`) with the child names, at the top of the axis
        `concat_axis`: `1` (default) concats the child results as columns,
        `0` stacks them as rows. The axis is fixed per collection. Each
        nested concat collection uses its own axis.
        `concat_top=False` puts it last. The join is outer, so children with
        different horizons give `NaN` outside their horizon.

        The level names of the children must match on each axis. A child
        whose result lacks levels of the longest child result, with the
        other levels in the same order, gets these levels with the value
        `''`. So a comparison with `comparison_type=BOTH` concats with
        scenarios. If the child result and the longest child result have
        unique level names, a child result with only level names of the
        longest child result, in another order, raises `ValueError` that
        names both children and both orders. Other differences raise
        `NotImplementedError`.

        With `auto_sort_index`, a concat on the columns sorts the rows. A
        concat on the rows keeps the order of the children; each child
        result is sorted already.

    Examples:
        >>> prices = study.scen.fetch('Node.Results.price')
        >>> list(prices.columns.names)
        ['dataset', 'Node']
        >>> base_prices = prices['base']

        Stack the scenarios as rows:

        >>> stacked = DatasetConcatCollection(study.scen.datasets, name='stacked', concat_axis=0)
        >>> list(stacked.fetch('Node.Results.price').index.names)
        ['dataset', 'time']

    Child parameters:
        It passes each child the dataset parameters that the child accepts
        and that the call or `param_defaults` gave, for example
        `comparison_type` of comparisons.
    """

    DEFAULT_CONCAT_LEVEL_NAME = 'dataset'
    DEFAULT_ATT_LEVEL_NAME = 'attribute'

    def __init__(
        self,
        datasets: list[DatasetType],
        name: str = None,
        parent_dataset: Dataset = None,
        attributes: dict = None,
        cache: Cache = None,
        config: DatasetConfigType = None,
        concat_axis: int = 1,
        concat_top: bool = True,
        concat_level_name: str = None,
    ):
        """Create a concat collection.

        Args:
            datasets: The child datasets. Their names must be unique.
            name: Name of the collection.
            parent_dataset: Optional parent dataset.
            attributes: Attributes of the collection.
            cache: Optional cache of fetch results.
            config: Instance config.
            concat_axis: Axis of the concat: `1` (default) concats the
                child results as columns, `0` stacks them as rows. A
                Python or numpy integer; `bool` and `float` values raise.
            concat_top: `True` (default) puts the new level at the top of
                the concat axis, `False` at the bottom. Stored as `bool`.
            concat_level_name: Name of the new level. Default `dataset`.

        Raises:
            ValueError: `concat_axis` is not `0` or `1`, two children have
                the same name, or two children have different specs with
                the same name.
        """
        is_integer = isinstance(concat_axis, (int, np.integer)) and not isinstance(concat_axis, (bool, np.bool_))
        if not is_integer or concat_axis not in (0, 1):
            raise ValueError(f'concat_axis must be 0 (rows) or 1 (columns), got {concat_axis!r}.')
        super().__init__(
            datasets=datasets,
            name=name,
            parent_dataset=parent_dataset,
            attributes=attributes,
            cache=cache,
            config=config,
        )
        self._concat_axis = int(concat_axis)
        self._concat_top = bool(concat_top)
        self._concat_level_name = concat_level_name or self.DEFAULT_CONCAT_LEVEL_NAME

    @property
    def concat_axis(self) -> int:
        """Axis of the concat, fixed at construction: `1` columns, `0` rows."""
        return self._concat_axis

    @property
    def concat_top(self) -> bool:
        """`True` if the new level is at the top of the concat axis, `False` if at the bottom. Fixed at construction."""
        return self._concat_top

    @property
    def concat_level_name(self) -> str:
        """Name of the new level with the child names. Fixed at construction."""
        return self._concat_level_name

    def transform_schema(self, schema: FrameSchema) -> FrameSchema:
        """Add `Level(concat_level_name, 'dataset')` to the axis of the concat.

        The axis is `concat_axis` of the collection. The level is at the
        top, or at the bottom with `concat_top=False`.
        """
        level = Level(self.concat_level_name, 'dataset')
        axis = 'index' if self.concat_axis == 0 else 'columns'
        levels = getattr(schema, axis)
        levels = (level, *levels) if self.concat_top else (*levels, level)
        return dataclasses.replace(schema, **{axis: levels})

    def _sort_index(self, data: pd.Series | pd.DataFrame) -> pd.Series | pd.DataFrame:
        """Sort the rows, unless the collection concats on rows (`concat_axis=0`): the concat level keeps the order of the children."""
        if self.concat_axis == 0:
            return data
        return super()._sort_index(data)

    def version_identity(self, flag: FlagType) -> tuple:
        """As `DatasetCollection.version_identity`, with `('concat', concat_axis, concat_top, concat_level_name)` first.

        So a collection with the same name and another layout of its result
        does not reuse a cache entry.
        """
        layout = ('concat', self.concat_axis, self.concat_top, self.concat_level_name)
        return (layout, *super().version_identity(flag))

    def get_attributes_concat_df(self) -> pd.DataFrame:
        if all(isinstance(ds, DatasetConcatCollection) for ds in self.datasets):
            use_att_df_instead_of_series = True
        else:
            use_att_df_instead_of_series = False

        atts_per_dataset = dict()
        for ds in self.datasets:
            atts = ds.get_attributes_concat_df().T if use_att_df_instead_of_series else ds.get_attributes_series()
            atts_per_dataset[ds.name] = atts

        return (
            pd.concat(atts_per_dataset, axis=1, names=[self.concat_level_name])
            .rename_axis(self.DEFAULT_ATT_LEVEL_NAME)
            .T
        )

    def _fetch(self, flag: FlagType, effective_config: DatasetConfigType, **kwargs) -> pd.Series | pd.DataFrame:
        dfs = {
            ds.name: ds.fetch(flag, effective_config, **self._kwargs_for_child(ds, flag))
            for ds in self._children_to_fetch(flag)
        }
        dfs = self._align_levels(dfs)
        df = pd.concat(dfs, join='outer', axis=self.concat_axis, names=[self.concat_level_name])
        return df if self.concat_top else move_top_level_last(df, self.concat_axis)

    @classmethod
    def _align_levels(cls, dfs: dict[str, pd.Series | pd.DataFrame]) -> dict[str, pd.Series | pd.DataFrame]:
        """Give all results the same level names on each axis.

        A result whose level names are the names of the longest result
        without some levels, in the same order, gets the missing levels with
        the value `''`.

        Raises:
            ValueError: A result has only level names of the longest result,
                but in another order. The result and the longest result have
                unique level names. The message names both children and both
                orders.
            NotImplementedError: The results differ in kind, or their level
                names differ in another way.
        """
        first = next(iter(dfs.values()))
        if not all(len(df.axes) == len(first.axes) for df in dfs.values()):
            raise NotImplementedError('Axes lengths do not match between dfs.')
        for axis in range(len(first.axes)):
            names = {key: list(df.axes[axis].names) for key, df in dfs.items()}
            target_key = max(names, key=lambda key: len(names[key]))
            target = names[target_key]
            target_is_unique = len(set(target)) == len(target)
            for key, own in names.items():
                if own == target:
                    continue
                own_is_unique = len(set(own)) == len(own)
                if not (target_is_unique and own_is_unique and set(own) <= set(target)):
                    raise NotImplementedError(f'Axes names do not match between dfs: {own} and {target}.')
                if not cls._is_subsequence(own, target):
                    raise ValueError(
                        f'Child {key!r} has the levels {own} on axis {axis} in another order than child '
                        f'{target_key!r} {target}. Give the child results the same level order.'
                    )
                dfs[key] = dfs[key].set_axis(cls._padded_axis(dfs[key].axes[axis], target), axis=axis)
        return dfs

    @staticmethod
    def _is_subsequence(names: list, target: list) -> bool:
        remaining = iter(target)
        return all(name in remaining for name in names)

    @staticmethod
    def _padded_axis(ax: pd.Index, target: list) -> pd.MultiIndex:
        """`ax` with the levels of `target` that it lacks, each with the value `''`."""
        own = list(ax.names)
        empty = np.full(len(ax), '', dtype=object)
        arrays = [ax.get_level_values(own.index(name)) if name in own else empty for name in target]
        return pd.MultiIndex.from_arrays(arrays, names=target)

concat_axis property

concat_axis: int

Axis of the concat, fixed at construction: 1 columns, 0 rows.

concat_top property

concat_top: bool

True if the new level is at the top of the concat axis, False if at the bottom. Fixed at construction.

concat_level_name property

concat_level_name: str

Name of the new level with the child names. Fixed at construction.

__init__

__init__(datasets: list[DatasetType], name: str = None, parent_dataset: Dataset = None, attributes: dict = None, cache: Cache = None, config: DatasetConfigType = None, concat_axis: int = 1, concat_top: bool = True, concat_level_name: str = None)

Create a concat collection.

Parameters:

Name Type Description Default
datasets list[DatasetType]

The child datasets. Their names must be unique.

required
name str

Name of the collection.

None
parent_dataset Dataset

Optional parent dataset.

None
attributes dict

Attributes of the collection.

None
cache Cache

Optional cache of fetch results.

None
config DatasetConfigType

Instance config.

None
concat_axis int

Axis of the concat: 1 (default) concats the child results as columns, 0 stacks them as rows. A Python or numpy integer; bool and float values raise.

1
concat_top bool

True (default) puts the new level at the top of the concat axis, False at the bottom. Stored as bool.

True
concat_level_name str

Name of the new level. Default dataset.

None

Raises:

Type Description
ValueError

concat_axis is not 0 or 1, two children have the same name, or two children have different specs with the same name.

Source code in src/mesqual/datasets/dataset_collection.py
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
def __init__(
    self,
    datasets: list[DatasetType],
    name: str = None,
    parent_dataset: Dataset = None,
    attributes: dict = None,
    cache: Cache = None,
    config: DatasetConfigType = None,
    concat_axis: int = 1,
    concat_top: bool = True,
    concat_level_name: str = None,
):
    """Create a concat collection.

    Args:
        datasets: The child datasets. Their names must be unique.
        name: Name of the collection.
        parent_dataset: Optional parent dataset.
        attributes: Attributes of the collection.
        cache: Optional cache of fetch results.
        config: Instance config.
        concat_axis: Axis of the concat: `1` (default) concats the
            child results as columns, `0` stacks them as rows. A
            Python or numpy integer; `bool` and `float` values raise.
        concat_top: `True` (default) puts the new level at the top of
            the concat axis, `False` at the bottom. Stored as `bool`.
        concat_level_name: Name of the new level. Default `dataset`.

    Raises:
        ValueError: `concat_axis` is not `0` or `1`, two children have
            the same name, or two children have different specs with
            the same name.
    """
    is_integer = isinstance(concat_axis, (int, np.integer)) and not isinstance(concat_axis, (bool, np.bool_))
    if not is_integer or concat_axis not in (0, 1):
        raise ValueError(f'concat_axis must be 0 (rows) or 1 (columns), got {concat_axis!r}.')
    super().__init__(
        datasets=datasets,
        name=name,
        parent_dataset=parent_dataset,
        attributes=attributes,
        cache=cache,
        config=config,
    )
    self._concat_axis = int(concat_axis)
    self._concat_top = bool(concat_top)
    self._concat_level_name = concat_level_name or self.DEFAULT_CONCAT_LEVEL_NAME

transform_schema

transform_schema(schema: FrameSchema) -> FrameSchema

Add Level(concat_level_name, 'dataset') to the axis of the concat.

The axis is concat_axis of the collection. The level is at the top, or at the bottom with concat_top=False.

Source code in src/mesqual/datasets/dataset_collection.py
812
813
814
815
816
817
818
819
820
821
822
def transform_schema(self, schema: FrameSchema) -> FrameSchema:
    """Add `Level(concat_level_name, 'dataset')` to the axis of the concat.

    The axis is `concat_axis` of the collection. The level is at the
    top, or at the bottom with `concat_top=False`.
    """
    level = Level(self.concat_level_name, 'dataset')
    axis = 'index' if self.concat_axis == 0 else 'columns'
    levels = getattr(schema, axis)
    levels = (level, *levels) if self.concat_top else (*levels, level)
    return dataclasses.replace(schema, **{axis: levels})

version_identity

version_identity(flag: FlagType) -> tuple

As DatasetCollection.version_identity, with ('concat', concat_axis, concat_top, concat_level_name) first.

So a collection with the same name and another layout of its result does not reuse a cache entry.

Source code in src/mesqual/datasets/dataset_collection.py
830
831
832
833
834
835
836
837
def version_identity(self, flag: FlagType) -> tuple:
    """As `DatasetCollection.version_identity`, with `('concat', concat_axis, concat_top, concat_level_name)` first.

    So a collection with the same name and another layout of its result
    does not reuse a cache entry.
    """
    layout = ('concat', self.concat_axis, self.concat_top, self.concat_level_name)
    return (layout, *super().version_identity(flag))

DatasetComparison

Bases: Generic[DatasetType, DatasetConfigType, FlagType, FlagIndexType], DatasetCollection[DatasetType, DatasetConfigType, FlagType, FlagIndexType]

The difference between a variation dataset and a reference dataset, with the Dataset interface.

fetch fetches the flag from both datasets and compares the results. Numeric and non-numeric data, frames and series are handled.

Comparison Types
  • DELTA: Variation - Reference (default). For non-numeric data, a changed cell becomes "<var> (was <ref>)". An entry (row or column) that only the variation has becomes "<var> (new)"; one that only the reference has becomes "DELETED: <ref>". A NaN value in an entry of one side only stays NaN and gets no marker.
  • VARIATION: Returns variation data with optional NaN for unchanged values
  • BOTH: Side-by-side variation and reference data, with a new last column level compared (BOTH_LEVEL_NAME) that holds the names of the two datasets.

All comparison types work for frames and series. For two series, DELTA and VARIATION return a series and BOTH a frame with the compared columns.

Attributes:

Name Type Description
variation_dataset

The dataset representing the scenario being compared

reference_dataset

The dataset representing the baseline for comparison

Dataset parameters (fetch_params):

  • comparison_type: DELTA (default), VARIATION or BOTH.
  • replace_unchanged_values_by_nan: True replaces values that are equal in both datasets by NaN. Default False. Only for flags that are not time series, for example model flags, so the comparison is empty where nothing changed. For a time series flag, True raises ValueError, also inside a collection. Mask the final result instead: for a numeric DELTA, for example delta.mask(delta == 0) after any conversion; for VARIATION or BOTH, compare with the reference.
  • fill_value: Value for missing entries before the subtraction of DELTA. Default None: the fill_value of the flag spec. float('nan'): no fill, so an entry missing on one side gives NaN, also if the spec has a fill_value.

Each parameter resolves like a flag parameter (Dataset.fetch). Flag parameters of the call go to both children. A dataset parameter that the call or param_defaults gave also goes to a child that accepts it, for example comparison_type to the children of a comparison of comparisons.

Examples:

The model change of a scenario without node N4:

>>> base = study.scen.get_dataset('base')
>>> without_n4 = DatasetComparison(study.scen.get_dataset('without_N4'), base)
>>> without_n4.fetch('Node.Model', replace_unchanged_values_by_nan=True)['country']
Node
N1            NaN
N2            NaN
N3            NaN
N4    DELETED: C2
Name: country, dtype: object

comparison_type from the call, fill_value from param_defaults and replace_unchanged_values_by_nan from its default:

>>> from mesqual.datasets import DatasetConfig
>>> from mesqual.enums import ComparisonTypeEnum
>>> high_demand = DatasetComparison(
...     study.scen.get_dataset('high_demand'), base, config=DatasetConfig(param_defaults={'fill_value': 0}),
... )
>>> both = high_demand.fetch('Node.Results.price', comparison_type=ComparisonTypeEnum.BOTH)
>>> list(both.columns.names)
['Node', 'compared']
>>> high_demand.get_fetch_record('Node.Results.price', comparison_type=ComparisonTypeEnum.BOTH).params
{'comparison_type': <ComparisonTypeEnum.BOTH: 'both'>, 'replace_unchanged_values_by_nan': False, 'fill_value': 0}
Source code in src/mesqual/datasets/dataset_comparison.py
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
class DatasetComparison(
    Generic[DatasetType, DatasetConfigType, FlagType, FlagIndexType],
    DatasetCollection[DatasetType, DatasetConfigType, FlagType, FlagIndexType],
):
    """
    The difference between a variation dataset and a reference dataset, with the `Dataset` interface.

    `fetch` fetches the flag from both datasets and compares the results.
    Numeric and non-numeric data, frames and series are handled.

    Comparison Types:
        - DELTA: Variation - Reference (default). For non-numeric data, a
          changed cell becomes `"<var> (was <ref>)"`. An entry (row or
          column) that only the variation has becomes `"<var> (new)"`; one
          that only the reference has becomes `"DELETED: <ref>"`. A NaN
          value in an entry of one side only stays NaN and gets no marker.
        - VARIATION: Returns variation data with optional NaN for unchanged values
        - BOTH: Side-by-side variation and reference data, with a new last
          column level `compared` (`BOTH_LEVEL_NAME`) that holds the names
          of the two datasets.

    All comparison types work for frames and series. For two series,
    `DELTA` and `VARIATION` return a series and `BOTH` a frame with the
    `compared` columns.

    Attributes:
        variation_dataset: The dataset representing the scenario being compared
        reference_dataset: The dataset representing the baseline for comparison

    Dataset parameters (`fetch_params`):

    - `comparison_type`: `DELTA` (default), `VARIATION` or `BOTH`.
    - `replace_unchanged_values_by_nan`: `True` replaces values that are
      equal in both datasets by NaN. Default `False`. Only for flags
      that are not time series, for example model flags, so the
      comparison is empty where nothing changed. For a time series
      flag, `True` raises `ValueError`, also inside a collection. Mask
      the final result instead: for a numeric `DELTA`, for example
      `delta.mask(delta == 0)` after any conversion; for `VARIATION`
      or `BOTH`, compare with the reference.
    - `fill_value`: Value for missing entries before the subtraction of
      `DELTA`. Default `None`: the `fill_value` of the flag spec.
      `float('nan')`: no fill, so an entry missing on one side gives
      NaN, also if the spec has a `fill_value`.

    Each parameter resolves like a flag parameter (`Dataset.fetch`).
    Flag parameters of the call go to both children. A dataset parameter that the call or
    `param_defaults` gave also goes to a child that accepts it, for
    example `comparison_type` to the children of a comparison of
    comparisons.

    Examples:
        The model change of a scenario without node N4:

        >>> base = study.scen.get_dataset('base')
        >>> without_n4 = DatasetComparison(study.scen.get_dataset('without_N4'), base)
        >>> without_n4.fetch('Node.Model', replace_unchanged_values_by_nan=True)['country']
        Node
        N1            NaN
        N2            NaN
        N3            NaN
        N4    DELETED: C2
        Name: country, dtype: object

        `comparison_type` from the call, `fill_value` from `param_defaults`
        and `replace_unchanged_values_by_nan` from its default:

        >>> from mesqual.datasets import DatasetConfig
        >>> from mesqual.enums import ComparisonTypeEnum
        >>> high_demand = DatasetComparison(
        ...     study.scen.get_dataset('high_demand'), base, config=DatasetConfig(param_defaults={'fill_value': 0}),
        ... )
        >>> both = high_demand.fetch('Node.Results.price', comparison_type=ComparisonTypeEnum.BOTH)
        >>> list(both.columns.names)
        ['Node', 'compared']
        >>> high_demand.get_fetch_record('Node.Results.price', comparison_type=ComparisonTypeEnum.BOTH).params
        {'comparison_type': <ComparisonTypeEnum.BOTH: 'both'>, 'replace_unchanged_values_by_nan': False, 'fill_value': 0}
    """

    COMPARISON_ATTRIBUTES_SOURCE = ComparisonAttributesSourceEnum.USE_VARIATION_ATTS
    COMPARISON_NAME_JOIN = ' vs '
    BOTH_LEVEL_NAME = 'compared'
    """Name of the last column level that `BOTH` adds. Its values are the names of the variation and the reference."""
    fetch_params = (
        Dimension(
            'comparison_type',
            'How to compare the variation with the reference.',
            {
                ComparisonTypeEnum.DELTA: 'variation minus reference; for non-numeric values the changes as text',
                ComparisonTypeEnum.VARIATION: 'the variation data',
                ComparisonTypeEnum.BOTH: 'variation and reference side by side',
            },
            default=ComparisonTypeEnum.DELTA,
        ),
        Dimension(
            'replace_unchanged_values_by_nan',
            'Replace values that are equal in both datasets by NaN. Not for time series flags.',
            {
                False: 'keep unchanged values',
                True: 'replace unchanged values by NaN',
            },
            default=False,
        ),
        Dimension(
            'fill_value',
            'Value for missing entries before the subtraction of a delta. None: the fill_value of the spec. '
            "float('nan'): no fill.",
            None,
            default=None,
        ),
    )

    VARIATION_DS_ATT_KEY = 'variation_dataset'
    REFERENCE_DS_ATT_KEY = 'reference_dataset'

    def __init__(
        self,
        variation_dataset: Dataset,
        reference_dataset: Dataset,
        name: str = None,
        attributes: dict = None,
        config: DatasetConfigType = None,
    ):
        name = name or self._get_auto_generated_name(variation_dataset, reference_dataset)

        super().__init__([reference_dataset, variation_dataset], name=name, attributes=attributes, config=config)

        self.variation_dataset = variation_dataset
        self.reference_dataset = reference_dataset

    def _get_auto_generated_name(self, variation_dataset: Dataset, reference_dataset: Dataset) -> str:
        return variation_dataset.name + self.COMPARISON_NAME_JOIN + reference_dataset.name

    @property
    def attributes(self) -> dict:
        match self.COMPARISON_ATTRIBUTES_SOURCE:
            case ComparisonAttributesSourceEnum.USE_VARIATION_ATTS:
                atts = self.variation_dataset.attributes.copy()
            case ComparisonAttributesSourceEnum.USE_REFERENCE_ATTS:
                atts = self.reference_dataset.attributes.copy()
            case _:
                atts = super().attributes
        atts[self.VARIATION_DS_ATT_KEY] = self.variation_dataset.name
        atts[self.REFERENCE_DS_ATT_KEY] = self.reference_dataset.name
        return atts

    def fetch(
        self,
        flag: FlagType,
        config: dict | DatasetConfigType = None,
        *,
        comparison_type: ComparisonTypeEnum | Unset = UNSET,
        replace_unchanged_values_by_nan: bool | Unset = UNSET,
        fill_value: float | None | Unset = UNSET,
        **kwargs: Any,
    ) -> pd.Series | pd.DataFrame:
        """Fetch a flag from both datasets and compare them, as `Dataset.fetch`.

        A dataset parameter left out of the call resolves from
        `param_defaults` of the effective config, else from the default
        given below.

        Args:
            flag: The flag.
            config: Config overrides for this call, as in `Dataset.fetch`.
            comparison_type: `DELTA` (default), `VARIATION` or `BOTH`.
            replace_unchanged_values_by_nan: `True` replaces values that are
                equal in both datasets by NaN. Default `False`. Only for flags
                that are not time series; a time series flag raises
                `ValueError`. Mask the final result instead: for a numeric
                `DELTA`, for example `delta.mask(delta == 0)` after any
                conversion; for `VARIATION` or `BOTH`, compare with the
                reference.
            fill_value: Value for missing entries before the subtraction of
                `DELTA`. Default `None`: the `fill_value` of the flag spec.
                `float('nan')`: no fill.
            **kwargs: Flag parameters, which go to both datasets, and the `granularity` argument of `Dataset.fetch`.

        Raises:
            ValueError: As `Dataset.fetch`, and for
                `replace_unchanged_values_by_nan=True` on a time series flag.
        """
        given = self._given(
            comparison_type=comparison_type,
            replace_unchanged_values_by_nan=replace_unchanged_values_by_nan,
            fill_value=fill_value,
        )
        return super().fetch(flag, config, **given, **kwargs)

    def transform_schema(self, schema: FrameSchema) -> FrameSchema:
        """Add `Level(BOTH_LEVEL_NAME, 'dataset')` as the last column level if the default `comparison_type` is `BOTH`.

        The default comes from the `param_defaults` of the instance config,
        else from the dimension.
        """
        comparison_type = (self.instance_config.param_defaults or {}).get('comparison_type')
        if comparison_type is None:
            comparison_type = next(d.default for d in self.fetch_params if d.name == 'comparison_type')
        if comparison_type is not ComparisonTypeEnum.BOTH:
            return schema
        return dataclasses.replace(schema, columns=(*schema.columns, Level(self.BOTH_LEVEL_NAME, 'dataset')))

    def _check_dataset_values(self, spec: FlagSpec, flag: FlagType, params: dict) -> None:
        """Also raise `ValueError` for `replace_unchanged_values_by_nan=True` on a time series flag."""
        super()._check_dataset_values(spec, flag, params)
        if params['replace_unchanged_values_by_nan'] and spec.item_type is ItemTypeEnum.TimeSeries:
            raise ValueError(
                f'Comparison {self.name}, flag {flag}: replace_unchanged_values_by_nan=True is only for flags that '
                f'are not time series. Mask the final result instead: for a numeric DELTA, for example '
                f'delta.mask(delta == 0) after any conversion; for VARIATION or BOTH, compare with the reference.'
            )

    def _fetch(
        self,
        flag: FlagType,
        effective_config: DatasetConfigType,
        *,
        comparison_type: ComparisonTypeEnum,
        replace_unchanged_values_by_nan: bool,
        fill_value: float | None,
        **kwargs,
    ) -> pd.Series | pd.DataFrame:
        df_var = self.variation_dataset.fetch(
            flag, effective_config, **self._kwargs_for_child(self.variation_dataset, flag)
        )
        df_ref = self.reference_dataset.fetch(
            flag, effective_config, **self._kwargs_for_child(self.reference_dataset, flag)
        )
        if fill_value is None:
            fill_value = self.flag_index.get_fill_value(flag)
        var, ref = self._as_frame(df_var), self._as_frame(df_ref)
        match comparison_type:
            case ComparisonTypeEnum.VARIATION:
                result = self._get_variation_comparison(var, ref, replace_unchanged_values_by_nan)
            case ComparisonTypeEnum.BOTH:
                result = self._get_both_comparison(var, ref, replace_unchanged_values_by_nan)
            case ComparisonTypeEnum.DELTA:
                result = self._get_delta_comparison(var, ref, replace_unchanged_values_by_nan, fill_value)
            case _:
                raise ValueError(f'Unsupported comparison_type: {comparison_type}')
        if isinstance(df_var, pd.Series) and isinstance(df_ref, pd.Series):
            return self._from_frame(result, df_var.name)
        return result

    _combine_availability = staticmethod(Availability.intersection)
    """Combine the availability of both children: the intersection (unavailable if one is unavailable)."""

    _SERIES_COLUMN = '__series__'

    @classmethod
    def _as_frame(cls, data: pd.Series | pd.DataFrame) -> pd.DataFrame:
        """A `Series` as a frame with one column, so all comparison types work for series."""
        return data.to_frame(cls._SERIES_COLUMN) if isinstance(data, pd.Series) else data

    @classmethod
    def _from_frame(cls, result: pd.DataFrame, name) -> pd.Series | pd.DataFrame:
        """The result for two `Series`: a `Series`, or for `BOTH` a frame with the `BOTH_LEVEL_NAME` columns."""
        if result.columns.nlevels > 1:
            return result.droplevel(0, axis=1)
        return result[cls._SERIES_COLUMN].rename(name)

    @staticmethod
    def _unchanged(df: pd.DataFrame, other: pd.DataFrame) -> pd.DataFrame:
        """`True` where `df` and `other` have the same value, or both NaN, in rows and columns that both have."""
        aligned = other.reindex(index=df.index, columns=df.columns)
        equal = df.eq(aligned) | (df.isna() & aligned.isna())
        in_both = np.outer(df.index.isin(other.index), df.columns.isin(other.columns))
        return equal & in_both

    def _get_variation_comparison(
        self, df_var: pd.DataFrame, df_ref: pd.DataFrame, replace_unchanged_values_by_nan: bool
    ) -> pd.DataFrame:
        if replace_unchanged_values_by_nan:
            return df_var.mask(self._unchanged(df_var, df_ref))
        return df_var.copy()

    def _get_both_comparison(
        self, df_var: pd.DataFrame, df_ref: pd.DataFrame, replace_unchanged_values_by_nan: bool
    ) -> pd.DataFrame:
        """Variation and reference side by side: a new last column level `BOTH_LEVEL_NAME` with the dataset names.

        Each column of the variation is followed by the same column of the
        reference. Columns of one side only appear once.
        """
        var_name, ref_name = self.variation_dataset.name, self.reference_dataset.name
        if replace_unchanged_values_by_nan:
            df_var, df_ref = df_var.mask(self._unchanged(df_var, df_ref)), df_ref.mask(self._unchanged(df_ref, df_var))
        result = move_top_level_last(
            pd.concat({var_name: df_var, ref_name: df_ref}, axis=1, names=[self.BOTH_LEVEL_NAME]),
            axis=1,
        )
        order = [
            (*self._as_tuple(column), side)
            for column in df_var.columns.union(df_ref.columns, sort=False)
            for side, frame in ((var_name, df_var), (ref_name, df_ref))
            if column in frame.columns
        ]
        return result[order]

    @staticmethod
    def _as_tuple(column) -> tuple:
        return column if isinstance(column, tuple) else (column,)

    def _get_delta_comparison(
        self,
        df_var: pd.DataFrame,
        df_ref: pd.DataFrame,
        replace_unchanged_values_by_nan: bool,
        fill_value: float | int | None,
    ) -> pd.DataFrame:
        if pd_is_numeric(df_var) and pd_is_numeric(df_ref):
            result = df_var.subtract(df_ref, fill_value=fill_value)
            return result.mask(result == 0) if replace_unchanged_values_by_nan else result

        index = df_var.index.union(df_ref.index, sort=False)
        columns = df_var.columns.union(df_ref.columns, sort=False)
        result = pd.DataFrame(index=index, columns=columns, dtype=object)
        empty = pd.Series(index=index[:0], dtype=object)
        for column in columns:
            result[column] = self._column_delta(
                df_var.get(column, empty),
                df_ref.get(column, empty),
                replace_unchanged_values_by_nan,
                fill_value,
            )
        return result

    def _column_delta(
        self,
        var: pd.Series,
        ref: pd.Series,
        replace_unchanged_values_by_nan: bool,
        fill_value: float | int | None,
    ) -> pd.Series:
        """Delta of one column: a difference for numeric data, else the changes as text.

        For non-numeric data, an entry of one side only gets a marker; a NaN
        value there stays NaN.
        """
        if pd_is_numeric(var) and pd_is_numeric(ref):
            delta = var.subtract(ref, fill_value=fill_value)
            return delta.mask(delta == 0) if replace_unchanged_values_by_nan else delta
        common = var.index.intersection(ref.index, sort=False)
        var_common, ref_common = var.loc[common], ref.loc[common]
        unchanged = (var_common.eq(ref_common) | (var_common.isna() & ref_common.isna())).to_numpy()
        delta = pd.Series(index=var.index.union(ref.index, sort=False), dtype=object)
        delta.loc[common[~unchanged]] = [
            f'{v} (was {r})' for v, r in zip(var_common[~unchanged], ref_common[~unchanged])
        ]
        if not replace_unchanged_values_by_nan:
            delta.loc[common[unchanged]] = var_common[unchanged].to_numpy()
        var_only = var.loc[var.index.difference(ref.index, sort=False)].dropna()
        delta.loc[var_only.index] = [f'{value} (new)' for value in var_only]
        ref_only = ref.loc[ref.index.difference(var.index, sort=False)].dropna()
        delta.loc[ref_only.index] = [f'DELETED: {value}' for value in ref_only]
        return delta

BOTH_LEVEL_NAME class-attribute instance-attribute

BOTH_LEVEL_NAME = 'compared'

Name of the last column level that BOTH adds. Its values are the names of the variation and the reference.

fetch

fetch(flag: FlagType, config: dict | DatasetConfigType = None, *, comparison_type: ComparisonTypeEnum | Unset = UNSET, replace_unchanged_values_by_nan: bool | Unset = UNSET, fill_value: float | None | Unset = UNSET, **kwargs: Any) -> Series | DataFrame

Fetch a flag from both datasets and compare them, as Dataset.fetch.

A dataset parameter left out of the call resolves from param_defaults of the effective config, else from the default given below.

Parameters:

Name Type Description Default
flag FlagType

The flag.

required
config dict | DatasetConfigType

Config overrides for this call, as in Dataset.fetch.

None
comparison_type ComparisonTypeEnum | Unset

DELTA (default), VARIATION or BOTH.

UNSET
replace_unchanged_values_by_nan bool | Unset

True replaces values that are equal in both datasets by NaN. Default False. Only for flags that are not time series; a time series flag raises ValueError. Mask the final result instead: for a numeric DELTA, for example delta.mask(delta == 0) after any conversion; for VARIATION or BOTH, compare with the reference.

UNSET
fill_value float | None | Unset

Value for missing entries before the subtraction of DELTA. Default None: the fill_value of the flag spec. float('nan'): no fill.

UNSET
**kwargs Any

Flag parameters, which go to both datasets, and the granularity argument of Dataset.fetch.

{}

Raises:

Type Description
ValueError

As Dataset.fetch, and for replace_unchanged_values_by_nan=True on a time series flag.

Source code in src/mesqual/datasets/dataset_comparison.py
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
def fetch(
    self,
    flag: FlagType,
    config: dict | DatasetConfigType = None,
    *,
    comparison_type: ComparisonTypeEnum | Unset = UNSET,
    replace_unchanged_values_by_nan: bool | Unset = UNSET,
    fill_value: float | None | Unset = UNSET,
    **kwargs: Any,
) -> pd.Series | pd.DataFrame:
    """Fetch a flag from both datasets and compare them, as `Dataset.fetch`.

    A dataset parameter left out of the call resolves from
    `param_defaults` of the effective config, else from the default
    given below.

    Args:
        flag: The flag.
        config: Config overrides for this call, as in `Dataset.fetch`.
        comparison_type: `DELTA` (default), `VARIATION` or `BOTH`.
        replace_unchanged_values_by_nan: `True` replaces values that are
            equal in both datasets by NaN. Default `False`. Only for flags
            that are not time series; a time series flag raises
            `ValueError`. Mask the final result instead: for a numeric
            `DELTA`, for example `delta.mask(delta == 0)` after any
            conversion; for `VARIATION` or `BOTH`, compare with the
            reference.
        fill_value: Value for missing entries before the subtraction of
            `DELTA`. Default `None`: the `fill_value` of the flag spec.
            `float('nan')`: no fill.
        **kwargs: Flag parameters, which go to both datasets, and the `granularity` argument of `Dataset.fetch`.

    Raises:
        ValueError: As `Dataset.fetch`, and for
            `replace_unchanged_values_by_nan=True` on a time series flag.
    """
    given = self._given(
        comparison_type=comparison_type,
        replace_unchanged_values_by_nan=replace_unchanged_values_by_nan,
        fill_value=fill_value,
    )
    return super().fetch(flag, config, **given, **kwargs)

transform_schema

transform_schema(schema: FrameSchema) -> FrameSchema

Add Level(BOTH_LEVEL_NAME, 'dataset') as the last column level if the default comparison_type is BOTH.

The default comes from the param_defaults of the instance config, else from the dimension.

Source code in src/mesqual/datasets/dataset_comparison.py
231
232
233
234
235
236
237
238
239
240
241
242
def transform_schema(self, schema: FrameSchema) -> FrameSchema:
    """Add `Level(BOTH_LEVEL_NAME, 'dataset')` as the last column level if the default `comparison_type` is `BOTH`.

    The default comes from the `param_defaults` of the instance config,
    else from the dimension.
    """
    comparison_type = (self.instance_config.param_defaults or {}).get('comparison_type')
    if comparison_type is None:
        comparison_type = next(d.default for d in self.fetch_params if d.name == 'comparison_type')
    if comparison_type is not ComparisonTypeEnum.BOTH:
        return schema
    return dataclasses.replace(schema, columns=(*schema.columns, Level(self.BOTH_LEVEL_NAME, 'dataset')))

DatasetConcatCollectionOfComparisons

Bases: Generic[DatasetConfigType, FlagType, FlagIndexType], DatasetConcatCollection[DatasetComparison, DatasetConfigType, FlagType, FlagIndexType]

Concat collection of comparisons.

Constructor as DatasetConcatCollection, including concat_axis.

Child parameters

It passes each child the dataset parameters that the child accepts and that the call or param_defaults gave, for example comparison_type of comparisons.

A comparison parameter that the call or the param_defaults of the collection does not give reaches no comparison. Each comparison then resolves it with its own config, so a comparison gives the same numbers inside the collection as alone.

Examples:

Each comparison gets comparison_type=BOTH as keyword argument and resolves replace_unchanged_values_by_nan and fill_value itself:

>>> from mesqual.enums import ComparisonTypeEnum
>>> both = study.comp.fetch('Node.Results.price', comparison_type=ComparisonTypeEnum.BOTH)
>>> study.comp.get_fetch_record('Node.Results.price', comparison_type=ComparisonTypeEnum.BOTH).params
{'comparison_type': <ComparisonTypeEnum.BOTH: 'both'>}
Source code in src/mesqual/datasets/dataset_comparison.py
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
class DatasetConcatCollectionOfComparisons(
    Generic[DatasetConfigType, FlagType, FlagIndexType],
    DatasetConcatCollection[DatasetComparison, DatasetConfigType, FlagType, FlagIndexType],
):
    """Concat collection of comparisons.

    Constructor as `DatasetConcatCollection`, including `concat_axis`.

    Child parameters:
        It passes each child the dataset parameters that the child accepts
        and that the call or `param_defaults` gave, for example
        `comparison_type` of comparisons.

    A comparison parameter that the call or the `param_defaults` of the
    collection does not give reaches no comparison. Each comparison then resolves it with its own config, so a
    comparison gives the same numbers inside the collection as alone.

    Examples:
        Each comparison gets `comparison_type=BOTH` as keyword argument and
        resolves `replace_unchanged_values_by_nan` and `fill_value` itself:

        >>> from mesqual.enums import ComparisonTypeEnum
        >>> both = study.comp.fetch('Node.Results.price', comparison_type=ComparisonTypeEnum.BOTH)
        >>> study.comp.get_fetch_record('Node.Results.price', comparison_type=ComparisonTypeEnum.BOTH).params
        {'comparison_type': <ComparisonTypeEnum.BOTH: 'both'>}
    """

    def fetch(
        self,
        flag: FlagType,
        config: dict | DatasetConfigType = None,
        *,
        comparison_type: ComparisonTypeEnum | Unset = UNSET,
        replace_unchanged_values_by_nan: bool | Unset = UNSET,
        fill_value: float | None | Unset = UNSET,
        **kwargs: Any,
    ) -> pd.Series | pd.DataFrame:
        """Fetch a flag from each comparison and concat the results, as `Dataset.fetch`.

        A dataset parameter left out of the call resolves from
        `param_defaults` of the effective config, else from the default
        given below.

        Args:
            flag: The flag.
            config: Config overrides for this call, as in `Dataset.fetch`.
            comparison_type: `DELTA` (default), `VARIATION` or `BOTH`.
            replace_unchanged_values_by_nan: `True` replaces values that are
                equal in both datasets by NaN. Default `False`. Only for flags
                that are not time series; a time series flag raises
                `ValueError`. Mask the final result instead: for a numeric
                `DELTA`, for example `delta.mask(delta == 0)` after any
                conversion; for `VARIATION` or `BOTH`, compare with the
                reference.
            fill_value: Value for missing entries before the subtraction of
                `DELTA`. Default `None`: the `fill_value` of the flag spec.
                `float('nan')`: no fill.
            **kwargs: Flag parameters, which go to all comparisons, and the `granularity` argument of `Dataset.fetch`.
        """
        given = self._given(
            comparison_type=comparison_type,
            replace_unchanged_values_by_nan=replace_unchanged_values_by_nan,
            fill_value=fill_value,
        )
        return super().fetch(flag, config, **given, **kwargs)

    @classmethod
    def get_child_dataset_type(cls) -> type[DatasetType]:
        return DatasetComparison

fetch

fetch(flag: FlagType, config: dict | DatasetConfigType = None, *, comparison_type: ComparisonTypeEnum | Unset = UNSET, replace_unchanged_values_by_nan: bool | Unset = UNSET, fill_value: float | None | Unset = UNSET, **kwargs: Any) -> Series | DataFrame

Fetch a flag from each comparison and concat the results, as Dataset.fetch.

A dataset parameter left out of the call resolves from param_defaults of the effective config, else from the default given below.

Parameters:

Name Type Description Default
flag FlagType

The flag.

required
config dict | DatasetConfigType

Config overrides for this call, as in Dataset.fetch.

None
comparison_type ComparisonTypeEnum | Unset

DELTA (default), VARIATION or BOTH.

UNSET
replace_unchanged_values_by_nan bool | Unset

True replaces values that are equal in both datasets by NaN. Default False. Only for flags that are not time series; a time series flag raises ValueError. Mask the final result instead: for a numeric DELTA, for example delta.mask(delta == 0) after any conversion; for VARIATION or BOTH, compare with the reference.

UNSET
fill_value float | None | Unset

Value for missing entries before the subtraction of DELTA. Default None: the fill_value of the flag spec. float('nan'): no fill.

UNSET
**kwargs Any

Flag parameters, which go to all comparisons, and the granularity argument of Dataset.fetch.

{}
Source code in src/mesqual/datasets/dataset_comparison.py
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
def fetch(
    self,
    flag: FlagType,
    config: dict | DatasetConfigType = None,
    *,
    comparison_type: ComparisonTypeEnum | Unset = UNSET,
    replace_unchanged_values_by_nan: bool | Unset = UNSET,
    fill_value: float | None | Unset = UNSET,
    **kwargs: Any,
) -> pd.Series | pd.DataFrame:
    """Fetch a flag from each comparison and concat the results, as `Dataset.fetch`.

    A dataset parameter left out of the call resolves from
    `param_defaults` of the effective config, else from the default
    given below.

    Args:
        flag: The flag.
        config: Config overrides for this call, as in `Dataset.fetch`.
        comparison_type: `DELTA` (default), `VARIATION` or `BOTH`.
        replace_unchanged_values_by_nan: `True` replaces values that are
            equal in both datasets by NaN. Default `False`. Only for flags
            that are not time series; a time series flag raises
            `ValueError`. Mask the final result instead: for a numeric
            `DELTA`, for example `delta.mask(delta == 0)` after any
            conversion; for `VARIATION` or `BOTH`, compare with the
            reference.
        fill_value: Value for missing entries before the subtraction of
            `DELTA`. Default `None`: the `fill_value` of the flag spec.
            `float('nan')`: no fill.
        **kwargs: Flag parameters, which go to all comparisons, and the `granularity` argument of `Dataset.fetch`.
    """
    given = self._given(
        comparison_type=comparison_type,
        replace_unchanged_values_by_nan=replace_unchanged_values_by_nan,
        fill_value=fill_value,
    )
    return super().fetch(flag, config, **given, **kwargs)

PlatformDataset

Bases: Generic[DatasetType, DatasetConfigType, FlagType, FlagIndexType], DatasetLinkCollection[DatasetType, DatasetConfigType, FlagType, FlagIndexType], ABC

Base class for platform-specific datasets with an interpreter registry.

A platform dataset holds one instance of each interpreter that answers (per original interpreter, the original or the override that replaces it) and routes each flag to the interpreter that declares its spec.

Registry rules
  • Each subclass has its own registry. register_interpreter on a class does not change its parent or sibling classes.
  • get_registered_interpreters() resolves the registry at call time along the MRO, so a parent registration made after the subclass definition reaches the subclass.
  • An instance resolves the registry once, in __init__, and builds the interpreter chains and the flag index there. register_interpreter raises for invalid arguments and for conflicts between overrides in the own registry of the class. All other spec and override errors are raised at instance creation and in the class lookups. It asks the interpreters for their accepted_flags at call time (fetch, flag_is_accepted, accepted_flags, get_flags_by_interpreter), so accepted_flags of an interpreter can read its siblings.
  • datasets and get_registered_interpreters() list the answering interpreters of base classes first, then in registration order.
  • Any Dataset subclass can be registered as interpreter, for example an AreaFlagInterpreter. get_child_dataset_type() gives the constructor arguments that the platform passes to them.
  • Each spec has exactly one answering interpreter. Two answering interpreters that declare the same spec name raise AmbiguousOverrideError at instance creation; the message names both interpreters.
  • A platform dataset class has no specs of its own. A subclass that sets flag_specs raises TypeError at class creation.
  • Each interpreter is named f'{platform.name}.{Interpreter.__qualname__}'. Two registered interpreters with the same __qualname__ raise ValueError at instance creation; the message names their modules.
Source inventory and default attributes

A subclass implements build_source_inventory with cheap reads of its source. source_inventory builds it once per instance. The interpreters read it in their _is_available. _default_attributes adds attributes from the inventory or model flags; explicit attributes win. _default_attributes runs in __init__: set the fields that build_source_inventory reads before calling super().__init__.

Overrides

register_interpreter(Override, override_interpreter=Original, version=1) replaces a whole interpreter. The override answers its own flag_specs and reuses code by subclassing and super()._fetch. Layer flags that fetch a flag of the override through self.parent_dataset.fetch(...) get the override result. get_answering_interpreter(flag) returns the interpreter that answers now, so a package can subclass it to build on it.

Processing after the interpreter chain

_fetch is the hook to process the result of each flag after its interpreter chain: a subclass overrides it and changes the result of super()._fetch(flag, effective_config, **params). The rules are in Interpreter overrides, item "After the chain".

Examples:

class MyPlatformDataset(PlatformDataset[MyInterpreter, ...]):
    def build_source_inventory(self):
        return MyInventory(fingerprint=hash_of_file_keys(self.folder), files=...)

    @classmethod
    def get_child_dataset_type(cls):
        return MyInterpreter

@MyPlatformDataset.register_interpreter
class NodeInterpreter(MyInterpreter):
    flag_specs = (NODE_MODEL_SPEC, NODE_PRICE_SPEC)
    ...

A study dataset that rounds the node price after the interpreter chain:

>>> from mesqual.testing import DummyDataset, DummyScenario
>>> class RoundedDataset(DummyDataset):
...     def _fetch(self, flag, effective_config, **params):
...         result = super()._fetch(flag, effective_config, **params)
...         return result.round(0) if flag == 'Node.Results.price' else result
>>> RoundedDataset(DummyScenario()).fetch('Node.Results.price').iloc[0].tolist()
[38.0, 44.0, 46.0, 55.0]
Source code in src/mesqual/datasets/platform_dataset.py
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
class PlatformDataset(
    Generic[DatasetType, DatasetConfigType, FlagType, FlagIndexType],
    DatasetLinkCollection[DatasetType, DatasetConfigType, FlagType, FlagIndexType],
    ABC,
):
    """
    Base class for platform-specific datasets with an interpreter registry.

    A platform dataset holds one instance of each interpreter that answers
    (per original interpreter, the original or the override that replaces
    it) and routes each flag to the interpreter that declares its spec.

    Registry rules:
        - Each subclass has its own registry. `register_interpreter` on a
          class does not change its parent or sibling classes.
        - `get_registered_interpreters()` resolves the registry at call time
          along the MRO, so a parent registration made after the subclass
          definition reaches the subclass.
        - An instance resolves the registry once, in `__init__`, and builds
          the interpreter chains and the flag index there.
          `register_interpreter` raises for invalid arguments and for
          conflicts between overrides in the own registry of the class. All
          other spec and override errors are raised at instance creation
          and in the class lookups. It asks the interpreters
          for their `accepted_flags` at call time (`fetch`,
          `flag_is_accepted`, `accepted_flags`, `get_flags_by_interpreter`),
          so `accepted_flags` of an interpreter can read its siblings.
        - `datasets` and `get_registered_interpreters()` list the answering
          interpreters of base classes first, then in registration order.
        - Any `Dataset` subclass can be registered as interpreter, for
          example an `AreaFlagInterpreter`. `get_child_dataset_type()` gives
          the constructor arguments that the platform passes to them.
        - Each spec has exactly one answering interpreter. Two answering
          interpreters that declare the same spec name raise
          `AmbiguousOverrideError` at instance creation; the message names
          both interpreters.
        - A platform dataset class has no specs of its own. A subclass that
          sets `flag_specs` raises `TypeError` at class creation.
        - Each interpreter is named `f'{platform.name}.{Interpreter.__qualname__}'`.
          Two registered interpreters with the same `__qualname__` raise
          `ValueError` at instance creation; the message names their modules.

    Source inventory and default attributes:
        A subclass implements `build_source_inventory` with cheap reads of
        its source. `source_inventory` builds it once per instance. The
        interpreters read it in their `_is_available`. `_default_attributes`
        adds attributes from the inventory or model flags; explicit
        attributes win. `_default_attributes` runs in `__init__`: set the
        fields that `build_source_inventory` reads before calling
        `super().__init__`.

    Overrides:
        `register_interpreter(Override, override_interpreter=Original, version=1)`
        replaces a whole interpreter. The override answers its own
        `flag_specs` and reuses code by subclassing and `super()._fetch`.
        Layer flags that fetch a flag of the override through
        `self.parent_dataset.fetch(...)` get the override result.
        `get_answering_interpreter(flag)` returns the interpreter that
        answers now, so a package can subclass it to build on it.

    Processing after the interpreter chain:
        `_fetch` is the hook to process the result of each flag after its
        interpreter chain: a subclass overrides it and changes the result
        of `super()._fetch(flag, effective_config, **params)`. The rules
        are in [Interpreter overrides](../architecture/other-contracts.md#interpreter-overrides),
        item "After the chain".

    Examples:
        ```python
        class MyPlatformDataset(PlatformDataset[MyInterpreter, ...]):
            def build_source_inventory(self):
                return MyInventory(fingerprint=hash_of_file_keys(self.folder), files=...)

            @classmethod
            def get_child_dataset_type(cls):
                return MyInterpreter

        @MyPlatformDataset.register_interpreter
        class NodeInterpreter(MyInterpreter):
            flag_specs = (NODE_MODEL_SPEC, NODE_PRICE_SPEC)
            ...
        ```

        A study dataset that rounds the node price after the interpreter chain:

        >>> from mesqual.testing import DummyDataset, DummyScenario
        >>> class RoundedDataset(DummyDataset):
        ...     def _fetch(self, flag, effective_config, **params):
        ...         result = super()._fetch(flag, effective_config, **params)
        ...         return result.round(0) if flag == 'Node.Results.price' else result
        >>> RoundedDataset(DummyScenario()).fetch('Node.Results.price').iloc[0].tolist()
        [38.0, 44.0, 46.0, 55.0]
    """

    _resolves_flag_params = True
    _is_leaf = True
    _reads_source = True

    def __init_subclass__(cls, **kwargs):
        super().__init_subclass__(**kwargs)
        if 'flag_specs' in cls.__dict__:
            raise TypeError(
                f'{cls.__name__} sets flag_specs. A platform dataset has no specs of its own; '
                f'declare them on a registered interpreter.'
            )

    def __init__(
        self,
        name: str = None,
        attributes: dict = None,
        cache: Cache = None,
        config: DatasetConfigType = None,
        **kwargs: Any,
    ):
        """Create the interpreters of this class and build the flag index.

        The keyword arguments are the own arguments below and the constructor
        arguments of the interpreter base `get_child_dataset_type()`, without
        `parent_dataset`. The platform dataset passes
        these to every interpreter it creates: the given value, else the
        default. A base argument with the name of an own argument (`name`,
        `attributes`, `cache`, `config`) binds to the own argument, so the
        interpreters get its default. Any other keyword argument raises
        `TypeError`.

        A subclass with arguments of its own takes them in its own `__init__`
        and passes only the others to `super().__init__`, or makes them
        constructor arguments of its interpreter base. For example,
        `DummyDataset` takes a scenario and passes the interpreter argument
        `source`.

        Args:
            name: Dataset name. Default: the class name and the object id.
            attributes: Attributes of the dataset. Default attributes from
                `_default_attributes` are added; explicit attributes win.
            cache: Optional cache of fetch results.
            config: Optional instance config.
            **kwargs: The constructor arguments of the interpreter base.

        Raises:
            TypeError: A keyword argument is neither an own argument nor a
                constructor argument of the interpreter base. The message
                names the unknown and the accepted arguments.
        """
        signature = InterpreterSignature.from_interpreter(self.get_child_dataset_type())
        self._raise_for_unknown_arguments(signature, kwargs)
        chains = self._resolved_chains()
        self._routes = _Routes(chains)
        self._interpreter_instances: dict[type[DatasetType], DatasetType] = {}
        self._source_inventory: SourceInventory | None = None
        super().__init__(
            datasets=[],
            name=name,
            attributes=attributes,
            cache=cache,
            config=config,
        )
        interpreter_args = signature.interpreter_arguments(kwargs)
        self._interpreter_instances = {
            chain[-1].interpreter: self._new_interpreter(chain[-1].interpreter, interpreter_args) for chain in chains
        }
        self.add_datasets(self._interpreter_instances.values())
        self.flag_index.check_references(self)
        self._add_default_attributes()

    @property
    def flag_index(self) -> FlagIndex:
        """The flag index: built at construction from the specs of the answering interpreters and the layers of the class."""
        return super().flag_index

    def _build_flag_index(self) -> FlagIndex:
        """Return the index of the specs of the answering interpreters and the layers of the class."""
        return FlagIndex(self._routes.flag_specs(), self._class_layers())

    @abstractmethod
    def build_source_inventory(self) -> SourceInventory:
        """Build the source inventory of this dataset. Read file names and small tables only.

        Returns:
            A `SourceInventory`, usually of a platform subclass with the
                fields that the `_is_available` implementations of the
                interpreters and `_default_attributes` need.
        """

    @property
    def source_inventory(self) -> SourceInventory:
        """The source inventory, built once per instance with `build_source_inventory`, then kept in memory."""
        if self._source_inventory is None:
            self._source_inventory = self.build_source_inventory()
        return self._source_inventory

    def _default_attributes(self) -> dict:
        """Hook: attributes that the platform adds to each dataset, for example the number of runs or the first date.

        Computed from the source inventory or model flags, never from time
        series flags. Values must be `bool`, `int`, `float` or `str`.
        Explicit attributes win. Default: none.
        """
        return {}

    def _add_default_attributes(self) -> None:
        explicit = self._attributes
        self._attributes = {}
        self._add_attributes({**self._default_attributes(), **explicit})

    def _checked_availability(self, flag: FlagType) -> Availability:
        """Availability from the `_is_available` of the answering interpreter.

        If it returns `None`, the flag is unknown, and available after its
        first successful fetch from this dataset.
        """
        interpreter = self._answering_instance(flag)
        name = type(interpreter).__qualname__
        checked = self._availability_from_hook(interpreter._is_available(flag), f'{name}._is_available')
        if checked is not None:
            return checked
        if str(flag) in self._fetched_flags:
            return Availability(True, f'fetched successfully from {self.name}')
        return Availability(None, f'no availability check in {name} and the flag was not fetched yet')

    def _flag_chain_names(self, flag: FlagType) -> tuple[str, ...]:
        return tuple(registration.interpreter.__qualname__ for registration in self._chain_of(flag))

    @classmethod
    def _declared_flag_specs(cls) -> tuple[FlagSpec, ...]:
        """The specs of the answering interpreters. A platform class has no specs of its own."""
        return tuple(cls._class_routes().flag_specs())

    @classmethod
    def _class_routes(cls) -> _Routes:
        return _Routes(cls._resolved_chains())

    @staticmethod
    def _check_unique_qualnames(chains: list[Chain]) -> None:
        """Raise `ValueError` if two interpreters of the chains have the same `__qualname__`.

        The qualname names the interpreter instance and its version identity link.
        """
        seen: dict[str, type] = {}
        for interpreter in (link.interpreter for chain in chains for link in chain):
            other = seen.setdefault(interpreter.__qualname__, interpreter)
            if other is not interpreter:
                if other.__module__ == interpreter.__module__:
                    origin = f'two different classes in module {interpreter.__module__!r}, for example from a factory function'
                else:
                    origin = f'one in module {other.__module__!r}, one in module {interpreter.__module__!r}'
                raise ValueError(
                    f'Two interpreters have the qualname {interpreter.__qualname__!r}: {origin}. Rename one of them.'
                )

    def _new_interpreter(self, interpreter: type[DatasetType], interpreter_args: dict) -> DatasetType:
        """Create the instance of an interpreter, named `f'{self.name}.{interpreter.__qualname__}'`.

        The name is stable across sessions, so records of two platform
        datasets with the same name have equal keys.
        """
        instance = interpreter(**interpreter_args, parent_dataset=self)
        instance.name = f'{self.name}.{interpreter.__qualname__}'
        return instance

    @classmethod
    def register_interpreter(
        cls,
        interpreter: Type[DatasetType] | None = None,
        *,
        override_interpreter: Type[DatasetType] | None = None,
        version: int | None = None,
    ) -> Type[DatasetType] | Callable[[Type[DatasetType]], Type[DatasetType]]:
        """Register an interpreter, or an override that replaces a whole interpreter.

        Usable as a decorator, with or without keyword arguments. The
        registration applies to this class and its subclasses only.

        An override replaces `override_interpreter` on the dataset: the
        replaced interpreter is not created, and the override answers its
        own `flag_specs`. A subclass of the replaced interpreter inherits
        its specs and can declare a spec again by name with other content.
        Per original interpreter, the registration on the most derived
        class answers. Reuse code by subclassing and `super()._fetch`; to
        build on what answers now, subclass `get_answering_interpreter(flag)`.

        Args:
            interpreter: The interpreter class.
            override_interpreter: The interpreter that the override replaces:
                an original, or the override that answers on this class or
                a parent class.
            version: Version of the override logic. Required with
                `override_interpreter`, not allowed without it. Part of the
                version identity of each flag of the override.

        Returns:
            The interpreter class, or a decorator if `interpreter` is `None`.

        Raises:
            TypeError: The interpreter or `override_interpreter` is not a
                `Dataset` subclass.
            ValueError: `version` is missing with `override_interpreter`, or
                given without it.
            ValueError: The interpreter names itself as `override_interpreter`.
            ValueError: The interpreter is already registered on this class
                with other arguments.
            ValueError: The override overrides an override on this class,
                or an override on this class overrides it.
            AmbiguousOverrideError: Another override on this class names the
                same `override_interpreter`.

        A failed registration does not change the registry. At instance
        creation and in the class lookups, `ValueError` if
        `override_interpreter` does not answer on the class or a parent
        class (the message names the interpreter to override instead), and
        `AmbiguousOverrideError` if two overrides on one class replace one
        original through different interpreters, or two overrides on
        sibling classes replace one original without an override on a
        common subclass.

        Examples:
            The override declares the price spec of `DummyTimeSeriesInterpreter`
            again, with the extra parameter `cap`, and keeps its other specs:

            >>> import dataclasses
            >>> from mesqual.flag import Dimension
            >>> from mesqual.testing import DummyDataset, DummyScenario
            >>> from mesqual.testing.dataset import DummyTimeSeriesInterpreter as Original
            >>> class StudyDataset(DummyDataset):
            ...     pass
            >>> CAP = Dimension('cap', 'Price cap in EUR/MWh.', default=1000)
            >>> CAPPED_PRICE_SPEC = dataclasses.replace(Original.PRICE_SPEC, params=Original.PRICE_SPEC.params + (CAP,))
            >>> @StudyDataset.register_interpreter(override_interpreter=Original, version=1)
            ... class CappedPrices(Original):
            ...     flag_specs = (CAPPED_PRICE_SPEC,) + Original.flag_specs[1:]
            ...     def _fetch(self, flag, effective_config, cap=1000, **params):
            ...         result = super()._fetch(flag, effective_config, **params)
            ...         return result.clip(upper=cap) if flag == 'Node.Results.price' else result
            >>> float(StudyDataset(DummyScenario()).fetch('Node.Results.price', cap=40).max().max())
            40.0
        """
        if interpreter is None:

            def decorator(interpreter_class: Type[DatasetType]) -> Type[DatasetType]:
                return cls.register_interpreter(
                    interpreter_class, override_interpreter=override_interpreter, version=version
                )

            return decorator

        registration = cls._build_registration(interpreter, override_interpreter, version)
        own_registrations = cls._own_registrations()
        for existing in own_registrations:
            if existing.interpreter is interpreter:
                if existing != registration:
                    raise ValueError(
                        f'{interpreter.__name__} is already registered on {cls.__name__} with other arguments.'
                    )
                return interpreter
        cls._check_own_overrides(registration, own_registrations)
        own_registrations.append(registration)
        return interpreter

    @classmethod
    def register_area_layer(cls, layer: AreaLayer) -> None:
        """Register an area layer on this class and its subclasses.

        The layer applies to every layer spec whose base is on
        `layer.base_model`. An instance checks the layer against the model
        specs at creation: the spec of the base model needs exactly one
        membership (`ModelFlagSpec.memberships`) to the layer model, else
        `ValueError`.

        Raises:
            TypeError: `layer` is not an `AreaLayer`.

        Examples:
            A layer of price zones, here one zone per country through the
            membership `country` of `'Node.Model'`:

            >>> from mesqual.flag import AreaLayer
            >>> from mesqual.testing import DummyDataset, DummyScenario
            >>> class StudyDataset(DummyDataset):
            ...     pass
            >>> StudyDataset.register_area_layer(AreaLayer(
            ...     name='Zone', index='Zone', base_model='Node.Model', layer_model='Country.Model',
            ...     description='Price zones, one per country.',
            ... ))
            >>> StudyDataset(DummyScenario()).fetch('Zone.Results.price').columns.tolist()
            ['C1', 'C2']
        """
        cls._register_layer(layer, AreaLayer, '_area_layer_registry')

    @classmethod
    def register_border_layer(cls, layer: BorderLayer) -> None:
        """Register a border layer on this class and its subclasses.

        Raises:
            TypeError: `layer` is not a `BorderLayer`.

        Examples:
            >>> from mesqual.flag import BorderLayer
            >>> from mesqual.testing import DummyDataset, DummyScenario
            >>> class StudyDataset(DummyDataset):
            ...     pass
            >>> StudyDataset.register_border_layer(BorderLayer(
            ...     name='CountryPair', index='CountryPair', area_layer='Country', edge_model='Line.Model',
            ...     description='Borders between two countries, crossed by lines.',
            ... ))
            >>> StudyDataset(DummyScenario()).fetch('CountryPair.Model').index.tolist()
            ['C1 - C2', 'C2 - C1']
        """
        cls._register_layer(layer, BorderLayer, '_border_layer_registry')

    @classmethod
    def get_area_layers(cls) -> list[AreaLayer]:
        """Return the area layers of this class and its parents, base classes first."""
        return cls._resolve_layers('_area_layer_registry')

    @classmethod
    def get_border_layers(cls) -> list[BorderLayer]:
        """Return the border layers of this class and its parents, base classes first."""
        return cls._resolve_layers('_border_layer_registry')

    @classmethod
    def _class_layers(cls) -> LayerSet:
        return LayerSet(cls.get_area_layers(), cls.get_border_layers())

    @classmethod
    def _register_layer(cls, layer: AreaLayer | BorderLayer, layer_type: type, registry: str) -> None:
        if not isinstance(layer, layer_type):
            raise TypeError(f'Expected a {layer_type.__name__}, got {type(layer).__name__}.')
        if registry not in cls.__dict__:
            setattr(cls, registry, [])
        own = cls.__dict__[registry]
        if layer not in own:
            own.append(layer)

    @classmethod
    def _resolve_layers(cls, registry: str) -> list:
        layers = []
        for klass in reversed(cls.__mro__):
            layers += [layer for layer in klass.__dict__.get(registry, []) if layer not in layers]
        return layers

    @classmethod
    def get_registered_interpreters(cls) -> list[Type[DatasetType]]:
        """Return the interpreters that a dataset of this class creates: per original, the one that answers.

        Resolved at call time along the MRO, base classes first. A replaced
        interpreter is not in the list. Raises as at instance creation.
        """
        return [chain[-1].interpreter for chain in cls._resolved_chains()]

    @classmethod
    def get_answering_interpreter(cls, flag: FlagType) -> Type[DatasetType]:
        """Return the interpreter class that answers a flag on this class, after all registered overrides.

        The lookup reads the specs and the registry, resolved along the MRO
        at call time. It does not check if a source provides the flag: an
        interpreter whose `accepted_flags` leaves out the flag on an
        instance still answers it here. Subclass the result to build an
        override on what answers now.

        Args:
            flag: A flag of a spec of a registered interpreter.

        Returns:
            The interpreter that answers the spec of the flag.

        Raises:
            UnknownFlagError: No registered interpreter declares the flag.
            AmbiguousOverrideError: Two interpreters declare the spec, or
                overrides conflict, as at instance creation.
            ValueError: An override is not valid, as at instance creation.

        Examples:
            >>> from mesqual.testing import DummyDataset
            >>> class StudyDataset(DummyDataset):
            ...     pass
            >>> answering = StudyDataset.get_answering_interpreter('Node.Results.price')
            >>> @StudyDataset.register_interpreter(override_interpreter=answering, version=1)
            ... class PriceFloor(answering):
            ...     def _fetch(self, flag, effective_config, **params):
            ...         return super()._fetch(flag, effective_config, **params).clip(lower=0)
            >>> StudyDataset.get_answering_interpreter('Node.Results.price').__name__
            'PriceFloor'
        """
        routes = cls._class_routes()
        chain = routes.chain_of(flag, FlagIndex(routes.flag_specs(), cls._class_layers()))
        if not chain:
            raise UnknownFlagError(f'No interpreter registered on {cls.__name__} declares the flag {flag!r}.')
        return chain[-1].interpreter

    def get_interpreter_instance(self, interpreter_type: Type[DatasetType]) -> DatasetType:
        """Return the interpreter instance of exactly this class (`type(instance) is interpreter_type`).

        Args:
            interpreter_type: Interpreter class.

        Returns:
            The interpreter instance of this dataset.

        Raises:
            ValueError: No interpreter of this class is registered.
        """
        interpreter = self._interpreter_instances.get(interpreter_type)
        if interpreter is None:
            raise ValueError(f'No interpreter instance found for type {interpreter_type.__name__}')
        return interpreter

    def get_flags_by_interpreter(self) -> dict[Type[DatasetType], set[FlagType]]:
        """Return the flags each interpreter answers. A replaced interpreter is not on the dataset."""
        return {
            type(interpreter): set(interpreter.accepted_flags) for interpreter in self._interpreter_instances.values()
        }

    @property
    def accepted_flags(self) -> set[FlagType]:
        """The flags in the `accepted_flags` of each interpreter."""
        return set().union(*self.get_flags_by_interpreter().values())

    def flag_is_accepted(self, flag: FlagType) -> bool:
        """`True` if the interpreter that declares the spec of the flag accepts it."""
        interpreter = self._answering_instance(flag)
        return interpreter is not None and interpreter.flag_is_accepted(flag)

    def _fetch(self, flag: FlagType, effective_config: DatasetConfigType, **kwargs) -> pd.Series | pd.DataFrame:
        child = self._answering_instance(flag)
        if child is None:
            raise KeyError(f"Key '{flag}' not recognized by any interpreter of {self.name}.")
        return child.fetch(flag, effective_config, **self._kwargs_for_child(child, flag))

    def version_identity(self, flag: FlagType) -> tuple:
        """Return one `(class name, version)` pair per link of the interpreter chain of the flag.

        The first link is the original interpreter with the version of the
        spec in the flag index. Each override along the dataset class
        hierarchy adds its `register_interpreter(version=)`. A layer spec
        adds the identity of its aggregation after the first link.

        Examples:
            >>> high_demand = study.scen.get_dataset('high_demand')
            >>> high_demand.version_identity('Node.Results.price')
            (('DummyTimeSeriesInterpreter', 1),)
            >>> high_demand.version_identity('Country.Results.price')
            (('DummyAreaFlags', 1), ('WeightedMean', {'weight': 'Node.Results.demand_total', 'version': 2}))
        """
        chain = self._chain_of(flag)
        if not chain:
            return ()
        spec = self.flag_index.get_spec(flag)
        first = ((chain[0].interpreter.__qualname__, spec.version),) + self._aggregation_links(spec)
        return first + tuple(
            (registration.interpreter.__qualname__, registration.version) for registration in chain[1:]
        )

    @staticmethod
    def _aggregation_links(spec: FlagSpec) -> tuple:
        """The aggregation link of a layer spec, right after its interpreter, else nothing."""
        return (spec.aggregation.identity(),) if isinstance(spec, LayerFlagSpec) else ()

    def _answering_instance(self, flag: FlagType) -> DatasetType | None:
        chain = self._chain_of(flag)
        return self._interpreter_instances[chain[-1].interpreter] if chain else None

    def _chain_of(self, flag: FlagType) -> Chain:
        """The interpreter chain of `flag`, from the original to the answering interpreter. Empty without one."""
        return self._routes.chain_of(flag, self.flag_index)

    @classmethod
    def _own_registrations(cls) -> list[InterpreterRegistration]:
        if '_interpreter_registry' not in cls.__dict__:
            cls._interpreter_registry = []
        return cls._interpreter_registry

    @classmethod
    def _check_own_overrides(cls, registration: InterpreterRegistration, own: list[InterpreterRegistration]) -> None:
        """Raises if `registration` conflicts with an override in the own registry of the class."""
        if not registration.is_override:
            return
        for existing in (r for r in own if r.is_override):
            if existing.interpreter is registration.override_interpreter:
                raise _ChainResolver._override_of_override_error(registration, existing, cls)
            if existing.override_interpreter is registration.interpreter:
                raise _ChainResolver._override_of_override_error(existing, registration, cls)
            if existing.override_interpreter is registration.override_interpreter:
                raise _ChainResolver._replaced_twice_error(
                    existing, registration, registration.override_interpreter, cls
                )

    @classmethod
    def _resolved_chains(cls) -> list[Chain]:
        """One chain per original interpreter, resolved along the MRO at call time. Raises on conflicts."""
        registrations = {
            klass: klass._own_registrations() for klass in cls.__mro__ if issubclass(klass, PlatformDataset)
        }
        chains = _ChainResolver(cls, registrations).resolve()
        cls._check_unique_qualnames(chains)
        return chains

    @classmethod
    def _build_registration(
        cls,
        interpreter: Type[DatasetType],
        override_interpreter: Type[DatasetType] | None,
        version: int | None,
    ) -> InterpreterRegistration:
        cls._validate_interpreter_type(interpreter)
        if override_interpreter is not None:
            cls._validate_interpreter_type(override_interpreter)
        if override_interpreter is not None and version is None:
            raise ValueError(f'{interpreter.__name__}: version is required with override_interpreter.')
        if override_interpreter is None and version is not None:
            raise ValueError(f'{interpreter.__name__}: version needs override_interpreter.')
        if override_interpreter is interpreter:
            raise ValueError(
                f'{interpreter.__name__} names itself as override_interpreter. Register it without '
                f'override_interpreter, or name the interpreter that it replaces.'
            )
        return InterpreterRegistration(interpreter, cls, override_interpreter=override_interpreter, version=version)

    def _raise_for_unknown_arguments(self, signature: InterpreterSignature, kwargs: dict[str, Any]) -> None:
        """Raise `TypeError` for a keyword argument that is neither an own argument nor an interpreter argument."""
        unknown = sorted(set(kwargs) - set(signature.args))
        if unknown:
            own = inspect.signature(PlatformDataset.__init__).parameters.values()
            named = {p.name for p in own if p.name != 'self' and p.kind is not p.VAR_KEYWORD}
            accepted = sorted(named | set(signature.args))
            raise TypeError(
                f'{type(self).__name__} got unknown keyword arguments {unknown}. '
                f'Accepted by PlatformDataset.__init__: {accepted} (its own arguments and the constructor '
                f'arguments of {self.get_child_dataset_type().__name__}).'
            )

    @classmethod
    def _validate_interpreter_type(cls, interpreter: Type[DatasetType]) -> None:
        if not cls._accepts_child_type(interpreter):
            raise TypeError(f'Interpreter must be a Dataset subclass, got {interpreter!r}.')

    @classmethod
    def _accepts_child_type(cls, dataset_type: type[Dataset]) -> bool:
        """`True` for any `Dataset` subclass: an interpreter need not subclass `get_child_dataset_type()`."""
        return isinstance(dataset_type, type) and issubclass(dataset_type, Dataset)

flag_index property

flag_index: FlagIndex

The flag index: built at construction from the specs of the answering interpreters and the layers of the class.

source_inventory property

source_inventory: SourceInventory

The source inventory, built once per instance with build_source_inventory, then kept in memory.

accepted_flags property

accepted_flags: set[FlagType]

The flags in the accepted_flags of each interpreter.

__init__

__init__(name: str = None, attributes: dict = None, cache: Cache = None, config: DatasetConfigType = None, **kwargs: Any)

Create the interpreters of this class and build the flag index.

The keyword arguments are the own arguments below and the constructor arguments of the interpreter base get_child_dataset_type(), without parent_dataset. The platform dataset passes these to every interpreter it creates: the given value, else the default. A base argument with the name of an own argument (name, attributes, cache, config) binds to the own argument, so the interpreters get its default. Any other keyword argument raises TypeError.

A subclass with arguments of its own takes them in its own __init__ and passes only the others to super().__init__, or makes them constructor arguments of its interpreter base. For example, DummyDataset takes a scenario and passes the interpreter argument source.

Parameters:

Name Type Description Default
name str

Dataset name. Default: the class name and the object id.

None
attributes dict

Attributes of the dataset. Default attributes from _default_attributes are added; explicit attributes win.

None
cache Cache

Optional cache of fetch results.

None
config DatasetConfigType

Optional instance config.

None
**kwargs Any

The constructor arguments of the interpreter base.

{}

Raises:

Type Description
TypeError

A keyword argument is neither an own argument nor a constructor argument of the interpreter base. The message names the unknown and the accepted arguments.

Source code in src/mesqual/datasets/platform_dataset.py
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
def __init__(
    self,
    name: str = None,
    attributes: dict = None,
    cache: Cache = None,
    config: DatasetConfigType = None,
    **kwargs: Any,
):
    """Create the interpreters of this class and build the flag index.

    The keyword arguments are the own arguments below and the constructor
    arguments of the interpreter base `get_child_dataset_type()`, without
    `parent_dataset`. The platform dataset passes
    these to every interpreter it creates: the given value, else the
    default. A base argument with the name of an own argument (`name`,
    `attributes`, `cache`, `config`) binds to the own argument, so the
    interpreters get its default. Any other keyword argument raises
    `TypeError`.

    A subclass with arguments of its own takes them in its own `__init__`
    and passes only the others to `super().__init__`, or makes them
    constructor arguments of its interpreter base. For example,
    `DummyDataset` takes a scenario and passes the interpreter argument
    `source`.

    Args:
        name: Dataset name. Default: the class name and the object id.
        attributes: Attributes of the dataset. Default attributes from
            `_default_attributes` are added; explicit attributes win.
        cache: Optional cache of fetch results.
        config: Optional instance config.
        **kwargs: The constructor arguments of the interpreter base.

    Raises:
        TypeError: A keyword argument is neither an own argument nor a
            constructor argument of the interpreter base. The message
            names the unknown and the accepted arguments.
    """
    signature = InterpreterSignature.from_interpreter(self.get_child_dataset_type())
    self._raise_for_unknown_arguments(signature, kwargs)
    chains = self._resolved_chains()
    self._routes = _Routes(chains)
    self._interpreter_instances: dict[type[DatasetType], DatasetType] = {}
    self._source_inventory: SourceInventory | None = None
    super().__init__(
        datasets=[],
        name=name,
        attributes=attributes,
        cache=cache,
        config=config,
    )
    interpreter_args = signature.interpreter_arguments(kwargs)
    self._interpreter_instances = {
        chain[-1].interpreter: self._new_interpreter(chain[-1].interpreter, interpreter_args) for chain in chains
    }
    self.add_datasets(self._interpreter_instances.values())
    self.flag_index.check_references(self)
    self._add_default_attributes()

build_source_inventory abstractmethod

build_source_inventory() -> SourceInventory

Build the source inventory of this dataset. Read file names and small tables only.

Returns:

Type Description
SourceInventory

A SourceInventory, usually of a platform subclass with the fields that the _is_available implementations of the interpreters and _default_attributes need.

Source code in src/mesqual/datasets/platform_dataset.py
449
450
451
452
453
454
455
456
457
@abstractmethod
def build_source_inventory(self) -> SourceInventory:
    """Build the source inventory of this dataset. Read file names and small tables only.

    Returns:
        A `SourceInventory`, usually of a platform subclass with the
            fields that the `_is_available` implementations of the
            interpreters and `_default_attributes` need.
    """

register_interpreter classmethod

register_interpreter(interpreter: Type[DatasetType] | None = None, *, override_interpreter: Type[DatasetType] | None = None, version: int | None = None) -> Type[DatasetType] | Callable[[Type[DatasetType]], Type[DatasetType]]

Register an interpreter, or an override that replaces a whole interpreter.

Usable as a decorator, with or without keyword arguments. The registration applies to this class and its subclasses only.

An override replaces override_interpreter on the dataset: the replaced interpreter is not created, and the override answers its own flag_specs. A subclass of the replaced interpreter inherits its specs and can declare a spec again by name with other content. Per original interpreter, the registration on the most derived class answers. Reuse code by subclassing and super()._fetch; to build on what answers now, subclass get_answering_interpreter(flag).

Parameters:

Name Type Description Default
interpreter Type[DatasetType] | None

The interpreter class.

None
override_interpreter Type[DatasetType] | None

The interpreter that the override replaces: an original, or the override that answers on this class or a parent class.

None
version int | None

Version of the override logic. Required with override_interpreter, not allowed without it. Part of the version identity of each flag of the override.

None

Returns:

Type Description
Type[DatasetType] | Callable[[Type[DatasetType]], Type[DatasetType]]

The interpreter class, or a decorator if interpreter is None.

Raises:

Type Description
TypeError

The interpreter or override_interpreter is not a Dataset subclass.

ValueError

version is missing with override_interpreter, or given without it.

ValueError

The interpreter names itself as override_interpreter.

ValueError

The interpreter is already registered on this class with other arguments.

ValueError

The override overrides an override on this class, or an override on this class overrides it.

AmbiguousOverrideError

Another override on this class names the same override_interpreter.

A failed registration does not change the registry. At instance creation and in the class lookups, ValueError if override_interpreter does not answer on the class or a parent class (the message names the interpreter to override instead), and AmbiguousOverrideError if two overrides on one class replace one original through different interpreters, or two overrides on sibling classes replace one original without an override on a common subclass.

Examples:

The override declares the price spec of DummyTimeSeriesInterpreter again, with the extra parameter cap, and keeps its other specs:

>>> import dataclasses
>>> from mesqual.flag import Dimension
>>> from mesqual.testing import DummyDataset, DummyScenario
>>> from mesqual.testing.dataset import DummyTimeSeriesInterpreter as Original
>>> class StudyDataset(DummyDataset):
...     pass
>>> CAP = Dimension('cap', 'Price cap in EUR/MWh.', default=1000)
>>> CAPPED_PRICE_SPEC = dataclasses.replace(Original.PRICE_SPEC, params=Original.PRICE_SPEC.params + (CAP,))
>>> @StudyDataset.register_interpreter(override_interpreter=Original, version=1)
... class CappedPrices(Original):
...     flag_specs = (CAPPED_PRICE_SPEC,) + Original.flag_specs[1:]
...     def _fetch(self, flag, effective_config, cap=1000, **params):
...         result = super()._fetch(flag, effective_config, **params)
...         return result.clip(upper=cap) if flag == 'Node.Results.price' else result
>>> float(StudyDataset(DummyScenario()).fetch('Node.Results.price', cap=40).max().max())
40.0
Source code in src/mesqual/datasets/platform_dataset.py
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
@classmethod
def register_interpreter(
    cls,
    interpreter: Type[DatasetType] | None = None,
    *,
    override_interpreter: Type[DatasetType] | None = None,
    version: int | None = None,
) -> Type[DatasetType] | Callable[[Type[DatasetType]], Type[DatasetType]]:
    """Register an interpreter, or an override that replaces a whole interpreter.

    Usable as a decorator, with or without keyword arguments. The
    registration applies to this class and its subclasses only.

    An override replaces `override_interpreter` on the dataset: the
    replaced interpreter is not created, and the override answers its
    own `flag_specs`. A subclass of the replaced interpreter inherits
    its specs and can declare a spec again by name with other content.
    Per original interpreter, the registration on the most derived
    class answers. Reuse code by subclassing and `super()._fetch`; to
    build on what answers now, subclass `get_answering_interpreter(flag)`.

    Args:
        interpreter: The interpreter class.
        override_interpreter: The interpreter that the override replaces:
            an original, or the override that answers on this class or
            a parent class.
        version: Version of the override logic. Required with
            `override_interpreter`, not allowed without it. Part of the
            version identity of each flag of the override.

    Returns:
        The interpreter class, or a decorator if `interpreter` is `None`.

    Raises:
        TypeError: The interpreter or `override_interpreter` is not a
            `Dataset` subclass.
        ValueError: `version` is missing with `override_interpreter`, or
            given without it.
        ValueError: The interpreter names itself as `override_interpreter`.
        ValueError: The interpreter is already registered on this class
            with other arguments.
        ValueError: The override overrides an override on this class,
            or an override on this class overrides it.
        AmbiguousOverrideError: Another override on this class names the
            same `override_interpreter`.

    A failed registration does not change the registry. At instance
    creation and in the class lookups, `ValueError` if
    `override_interpreter` does not answer on the class or a parent
    class (the message names the interpreter to override instead), and
    `AmbiguousOverrideError` if two overrides on one class replace one
    original through different interpreters, or two overrides on
    sibling classes replace one original without an override on a
    common subclass.

    Examples:
        The override declares the price spec of `DummyTimeSeriesInterpreter`
        again, with the extra parameter `cap`, and keeps its other specs:

        >>> import dataclasses
        >>> from mesqual.flag import Dimension
        >>> from mesqual.testing import DummyDataset, DummyScenario
        >>> from mesqual.testing.dataset import DummyTimeSeriesInterpreter as Original
        >>> class StudyDataset(DummyDataset):
        ...     pass
        >>> CAP = Dimension('cap', 'Price cap in EUR/MWh.', default=1000)
        >>> CAPPED_PRICE_SPEC = dataclasses.replace(Original.PRICE_SPEC, params=Original.PRICE_SPEC.params + (CAP,))
        >>> @StudyDataset.register_interpreter(override_interpreter=Original, version=1)
        ... class CappedPrices(Original):
        ...     flag_specs = (CAPPED_PRICE_SPEC,) + Original.flag_specs[1:]
        ...     def _fetch(self, flag, effective_config, cap=1000, **params):
        ...         result = super()._fetch(flag, effective_config, **params)
        ...         return result.clip(upper=cap) if flag == 'Node.Results.price' else result
        >>> float(StudyDataset(DummyScenario()).fetch('Node.Results.price', cap=40).max().max())
        40.0
    """
    if interpreter is None:

        def decorator(interpreter_class: Type[DatasetType]) -> Type[DatasetType]:
            return cls.register_interpreter(
                interpreter_class, override_interpreter=override_interpreter, version=version
            )

        return decorator

    registration = cls._build_registration(interpreter, override_interpreter, version)
    own_registrations = cls._own_registrations()
    for existing in own_registrations:
        if existing.interpreter is interpreter:
            if existing != registration:
                raise ValueError(
                    f'{interpreter.__name__} is already registered on {cls.__name__} with other arguments.'
                )
            return interpreter
    cls._check_own_overrides(registration, own_registrations)
    own_registrations.append(registration)
    return interpreter

register_area_layer classmethod

register_area_layer(layer: AreaLayer) -> None

Register an area layer on this class and its subclasses.

The layer applies to every layer spec whose base is on layer.base_model. An instance checks the layer against the model specs at creation: the spec of the base model needs exactly one membership (ModelFlagSpec.memberships) to the layer model, else ValueError.

Raises:

Type Description
TypeError

layer is not an AreaLayer.

Examples:

A layer of price zones, here one zone per country through the membership country of 'Node.Model':

>>> from mesqual.flag import AreaLayer
>>> from mesqual.testing import DummyDataset, DummyScenario
>>> class StudyDataset(DummyDataset):
...     pass
>>> StudyDataset.register_area_layer(AreaLayer(
...     name='Zone', index='Zone', base_model='Node.Model', layer_model='Country.Model',
...     description='Price zones, one per country.',
... ))
>>> StudyDataset(DummyScenario()).fetch('Zone.Results.price').columns.tolist()
['C1', 'C2']
Source code in src/mesqual/datasets/platform_dataset.py
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
@classmethod
def register_area_layer(cls, layer: AreaLayer) -> None:
    """Register an area layer on this class and its subclasses.

    The layer applies to every layer spec whose base is on
    `layer.base_model`. An instance checks the layer against the model
    specs at creation: the spec of the base model needs exactly one
    membership (`ModelFlagSpec.memberships`) to the layer model, else
    `ValueError`.

    Raises:
        TypeError: `layer` is not an `AreaLayer`.

    Examples:
        A layer of price zones, here one zone per country through the
        membership `country` of `'Node.Model'`:

        >>> from mesqual.flag import AreaLayer
        >>> from mesqual.testing import DummyDataset, DummyScenario
        >>> class StudyDataset(DummyDataset):
        ...     pass
        >>> StudyDataset.register_area_layer(AreaLayer(
        ...     name='Zone', index='Zone', base_model='Node.Model', layer_model='Country.Model',
        ...     description='Price zones, one per country.',
        ... ))
        >>> StudyDataset(DummyScenario()).fetch('Zone.Results.price').columns.tolist()
        ['C1', 'C2']
    """
    cls._register_layer(layer, AreaLayer, '_area_layer_registry')

register_border_layer classmethod

register_border_layer(layer: BorderLayer) -> None

Register a border layer on this class and its subclasses.

Raises:

Type Description
TypeError

layer is not a BorderLayer.

Examples:

>>> from mesqual.flag import BorderLayer
>>> from mesqual.testing import DummyDataset, DummyScenario
>>> class StudyDataset(DummyDataset):
...     pass
>>> StudyDataset.register_border_layer(BorderLayer(
...     name='CountryPair', index='CountryPair', area_layer='Country', edge_model='Line.Model',
...     description='Borders between two countries, crossed by lines.',
... ))
>>> StudyDataset(DummyScenario()).fetch('CountryPair.Model').index.tolist()
['C1 - C2', 'C2 - C1']
Source code in src/mesqual/datasets/platform_dataset.py
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
@classmethod
def register_border_layer(cls, layer: BorderLayer) -> None:
    """Register a border layer on this class and its subclasses.

    Raises:
        TypeError: `layer` is not a `BorderLayer`.

    Examples:
        >>> from mesqual.flag import BorderLayer
        >>> from mesqual.testing import DummyDataset, DummyScenario
        >>> class StudyDataset(DummyDataset):
        ...     pass
        >>> StudyDataset.register_border_layer(BorderLayer(
        ...     name='CountryPair', index='CountryPair', area_layer='Country', edge_model='Line.Model',
        ...     description='Borders between two countries, crossed by lines.',
        ... ))
        >>> StudyDataset(DummyScenario()).fetch('CountryPair.Model').index.tolist()
        ['C1 - C2', 'C2 - C1']
    """
    cls._register_layer(layer, BorderLayer, '_border_layer_registry')

get_area_layers classmethod

get_area_layers() -> list[AreaLayer]

Return the area layers of this class and its parents, base classes first.

Source code in src/mesqual/datasets/platform_dataset.py
684
685
686
687
@classmethod
def get_area_layers(cls) -> list[AreaLayer]:
    """Return the area layers of this class and its parents, base classes first."""
    return cls._resolve_layers('_area_layer_registry')

get_border_layers classmethod

get_border_layers() -> list[BorderLayer]

Return the border layers of this class and its parents, base classes first.

Source code in src/mesqual/datasets/platform_dataset.py
689
690
691
692
@classmethod
def get_border_layers(cls) -> list[BorderLayer]:
    """Return the border layers of this class and its parents, base classes first."""
    return cls._resolve_layers('_border_layer_registry')

get_registered_interpreters classmethod

get_registered_interpreters() -> list[Type[DatasetType]]

Return the interpreters that a dataset of this class creates: per original, the one that answers.

Resolved at call time along the MRO, base classes first. A replaced interpreter is not in the list. Raises as at instance creation.

Source code in src/mesqual/datasets/platform_dataset.py
715
716
717
718
719
720
721
722
@classmethod
def get_registered_interpreters(cls) -> list[Type[DatasetType]]:
    """Return the interpreters that a dataset of this class creates: per original, the one that answers.

    Resolved at call time along the MRO, base classes first. A replaced
    interpreter is not in the list. Raises as at instance creation.
    """
    return [chain[-1].interpreter for chain in cls._resolved_chains()]

get_answering_interpreter classmethod

get_answering_interpreter(flag: FlagType) -> Type[DatasetType]

Return the interpreter class that answers a flag on this class, after all registered overrides.

The lookup reads the specs and the registry, resolved along the MRO at call time. It does not check if a source provides the flag: an interpreter whose accepted_flags leaves out the flag on an instance still answers it here. Subclass the result to build an override on what answers now.

Parameters:

Name Type Description Default
flag FlagType

A flag of a spec of a registered interpreter.

required

Returns:

Type Description
Type[DatasetType]

The interpreter that answers the spec of the flag.

Raises:

Type Description
UnknownFlagError

No registered interpreter declares the flag.

AmbiguousOverrideError

Two interpreters declare the spec, or overrides conflict, as at instance creation.

ValueError

An override is not valid, as at instance creation.

Examples:

>>> from mesqual.testing import DummyDataset
>>> class StudyDataset(DummyDataset):
...     pass
>>> answering = StudyDataset.get_answering_interpreter('Node.Results.price')
>>> @StudyDataset.register_interpreter(override_interpreter=answering, version=1)
... class PriceFloor(answering):
...     def _fetch(self, flag, effective_config, **params):
...         return super()._fetch(flag, effective_config, **params).clip(lower=0)
>>> StudyDataset.get_answering_interpreter('Node.Results.price').__name__
'PriceFloor'
Source code in src/mesqual/datasets/platform_dataset.py
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
@classmethod
def get_answering_interpreter(cls, flag: FlagType) -> Type[DatasetType]:
    """Return the interpreter class that answers a flag on this class, after all registered overrides.

    The lookup reads the specs and the registry, resolved along the MRO
    at call time. It does not check if a source provides the flag: an
    interpreter whose `accepted_flags` leaves out the flag on an
    instance still answers it here. Subclass the result to build an
    override on what answers now.

    Args:
        flag: A flag of a spec of a registered interpreter.

    Returns:
        The interpreter that answers the spec of the flag.

    Raises:
        UnknownFlagError: No registered interpreter declares the flag.
        AmbiguousOverrideError: Two interpreters declare the spec, or
            overrides conflict, as at instance creation.
        ValueError: An override is not valid, as at instance creation.

    Examples:
        >>> from mesqual.testing import DummyDataset
        >>> class StudyDataset(DummyDataset):
        ...     pass
        >>> answering = StudyDataset.get_answering_interpreter('Node.Results.price')
        >>> @StudyDataset.register_interpreter(override_interpreter=answering, version=1)
        ... class PriceFloor(answering):
        ...     def _fetch(self, flag, effective_config, **params):
        ...         return super()._fetch(flag, effective_config, **params).clip(lower=0)
        >>> StudyDataset.get_answering_interpreter('Node.Results.price').__name__
        'PriceFloor'
    """
    routes = cls._class_routes()
    chain = routes.chain_of(flag, FlagIndex(routes.flag_specs(), cls._class_layers()))
    if not chain:
        raise UnknownFlagError(f'No interpreter registered on {cls.__name__} declares the flag {flag!r}.')
    return chain[-1].interpreter

get_interpreter_instance

get_interpreter_instance(interpreter_type: Type[DatasetType]) -> DatasetType

Return the interpreter instance of exactly this class (type(instance) is interpreter_type).

Parameters:

Name Type Description Default
interpreter_type Type[DatasetType]

Interpreter class.

required

Returns:

Type Description
DatasetType

The interpreter instance of this dataset.

Raises:

Type Description
ValueError

No interpreter of this class is registered.

Source code in src/mesqual/datasets/platform_dataset.py
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
def get_interpreter_instance(self, interpreter_type: Type[DatasetType]) -> DatasetType:
    """Return the interpreter instance of exactly this class (`type(instance) is interpreter_type`).

    Args:
        interpreter_type: Interpreter class.

    Returns:
        The interpreter instance of this dataset.

    Raises:
        ValueError: No interpreter of this class is registered.
    """
    interpreter = self._interpreter_instances.get(interpreter_type)
    if interpreter is None:
        raise ValueError(f'No interpreter instance found for type {interpreter_type.__name__}')
    return interpreter

get_flags_by_interpreter

get_flags_by_interpreter() -> dict[Type[DatasetType], set[FlagType]]

Return the flags each interpreter answers. A replaced interpreter is not on the dataset.

Source code in src/mesqual/datasets/platform_dataset.py
781
782
783
784
785
def get_flags_by_interpreter(self) -> dict[Type[DatasetType], set[FlagType]]:
    """Return the flags each interpreter answers. A replaced interpreter is not on the dataset."""
    return {
        type(interpreter): set(interpreter.accepted_flags) for interpreter in self._interpreter_instances.values()
    }

flag_is_accepted

flag_is_accepted(flag: FlagType) -> bool

True if the interpreter that declares the spec of the flag accepts it.

Source code in src/mesqual/datasets/platform_dataset.py
792
793
794
795
def flag_is_accepted(self, flag: FlagType) -> bool:
    """`True` if the interpreter that declares the spec of the flag accepts it."""
    interpreter = self._answering_instance(flag)
    return interpreter is not None and interpreter.flag_is_accepted(flag)

version_identity

version_identity(flag: FlagType) -> tuple

Return one (class name, version) pair per link of the interpreter chain of the flag.

The first link is the original interpreter with the version of the spec in the flag index. Each override along the dataset class hierarchy adds its register_interpreter(version=). A layer spec adds the identity of its aggregation after the first link.

Examples:

>>> high_demand = study.scen.get_dataset('high_demand')
>>> high_demand.version_identity('Node.Results.price')
(('DummyTimeSeriesInterpreter', 1),)
>>> high_demand.version_identity('Country.Results.price')
(('DummyAreaFlags', 1), ('WeightedMean', {'weight': 'Node.Results.demand_total', 'version': 2}))
Source code in src/mesqual/datasets/platform_dataset.py
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
def version_identity(self, flag: FlagType) -> tuple:
    """Return one `(class name, version)` pair per link of the interpreter chain of the flag.

    The first link is the original interpreter with the version of the
    spec in the flag index. Each override along the dataset class
    hierarchy adds its `register_interpreter(version=)`. A layer spec
    adds the identity of its aggregation after the first link.

    Examples:
        >>> high_demand = study.scen.get_dataset('high_demand')
        >>> high_demand.version_identity('Node.Results.price')
        (('DummyTimeSeriesInterpreter', 1),)
        >>> high_demand.version_identity('Country.Results.price')
        (('DummyAreaFlags', 1), ('WeightedMean', {'weight': 'Node.Results.demand_total', 'version': 2}))
    """
    chain = self._chain_of(flag)
    if not chain:
        return ()
    spec = self.flag_index.get_spec(flag)
    first = ((chain[0].interpreter.__qualname__, spec.version),) + self._aggregation_links(spec)
    return first + tuple(
        (registration.interpreter.__qualname__, registration.version) for registration in chain[1:]
    )

SourceInventory dataclass

What the source of a platform dataset holds, read with cheap reads only.

A platform dataset builds its inventory once (PlatformDataset.build_source_inventory), for example from the file list and small tables of the source. mesqual reads no field of the inventory. A platform subclass adds the fields that its _is_available implementations and its default attributes need.

Attributes:

Name Type Description
fingerprint str

Changes when the source changes, for example a hash of the relative name and the FileRead.key of each source file.

Examples:

>>> from dataclasses import dataclass
>>> @dataclass(frozen=True)
... class MyInventory(SourceInventory):
...     files: frozenset[str] = frozenset()
>>> MyInventory(fingerprint='3f2a', files=frozenset({'prices.csv'}))
MyInventory(fingerprint='3f2a', files=frozenset({'prices.csv'}))
Source code in src/mesqual/datasets/source_inventory.py
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
@dataclass(frozen=True)
class SourceInventory:
    """What the source of a platform dataset holds, read with cheap reads only.

    A platform dataset builds its inventory once (`PlatformDataset.build_source_inventory`),
    for example from the file list and small tables of the source. mesqual
    reads no field of the inventory. A platform subclass adds the fields
    that its `_is_available` implementations and its default attributes
    need.

    Attributes:
        fingerprint: Changes when the source changes, for example a hash of
            the relative name and the `FileRead.key` of each source file.

    Examples:
        >>> from dataclasses import dataclass
        >>> @dataclass(frozen=True)
        ... class MyInventory(SourceInventory):
        ...     files: frozenset[str] = frozenset()
        >>> MyInventory(fingerprint='3f2a', files=frozenset({'prices.csv'}))
        MyInventory(fingerprint='3f2a', files=frozenset({'prices.csv'}))
    """

    fingerprint: str

Availability dataclass

Whether a dataset can answer a flag.

Attributes:

Name Type Description
available bool | None

True or False, or None for unknown.

reason str

Where the answer comes from.

datasets tuple[str, ...] | None

For a collection: names of the children on which the flag is available or unknown. None for other datasets.

Source code in src/mesqual/datasets/describe.py
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
@dataclass(frozen=True)
class Availability:
    """Whether a dataset can answer a flag.

    Attributes:
        available: `True` or `False`, or `None` for unknown.
        reason: Where the answer comes from.
        datasets: For a collection: names of the children on which the flag
            is available or unknown. `None` for other datasets.
    """

    available: bool | None
    reason: str
    datasets: tuple[str, ...] | None = None

    @property
    def label(self) -> str:
        """`'available'`, `'unavailable'` or `'unknown'`."""
        return {True: 'available', False: 'unavailable', None: 'unknown'}[self.available]

    @classmethod
    def union(cls, by_child: Mapping[str, Availability]) -> Availability:
        """Combine the availability of the children of a collection.

        Available if one child has the flag available, else unknown if one
        child does not know, else unavailable.
        """
        states = {availability.available for availability in by_child.values()}
        available = True if True in states else (None if None in states else False)
        groups = {
            label: [name for name, availability in by_child.items() if availability.label == label]
            for label in ('available', 'unknown', 'unavailable')
        }
        reason = '; '.join(f'{label} in {", ".join(names)}' for label, names in groups.items() if names)
        datasets = tuple(name for name, availability in by_child.items() if availability.available is not False)
        return cls(available, reason or 'no children', datasets)

    @classmethod
    def intersection(cls, by_child: Mapping[str, Availability]) -> Availability:
        """Combine the availability of children that are all needed, for example of a comparison.

        Unavailable if one child has the flag unavailable, else unknown if
        one child does not know, else available.
        """
        available = cls._weakest_state(by_child.values()) if by_child else False
        reason = '; '.join(f'{availability.label} in {name}' for name, availability in by_child.items())
        datasets = () if available is False else tuple(by_child)
        return cls(available, reason or 'no children', datasets)

    @classmethod
    def of_inputs(cls, by_flag: Mapping[str, Availability]) -> Availability:
        """Combine the availability of the flags that a flag needs.

        Unavailable if one input is unavailable, else unknown if one input
        is unknown, else available. The reason names the unavailable or the
        unknown inputs with their reasons.

        Args:
            by_flag: The availability of each input flag.

        Examples:
            >>> Availability.of_inputs({
            ...     'Node.Results.price': Availability(True, 'prices in the source'),
            ...     'Node.Results.demand': Availability(False, 'no demand table'),
            ... }).reason
            'unavailable input Node.Results.demand (no demand table)'
        """
        available = cls._weakest_state(by_flag.values())
        if available is True:
            return cls(True, f'inputs available: {", ".join(by_flag)}' if by_flag else 'no inputs')
        label = 'unavailable' if available is False else 'unknown'
        reason = '; '.join(
            f'{label} input {flag} ({availability.reason})'
            for flag, availability in by_flag.items()
            if availability.available is available
        )
        return cls(available, reason)

    @staticmethod
    def _weakest_state(availabilities: Iterable[Availability]) -> bool | None:
        """`False` if one is unavailable, else `None` if one is unknown, else `True`."""
        states = {availability.available for availability in availabilities}
        return False if False in states else (None if None in states else True)

label property

label: str

'available', 'unavailable' or 'unknown'.

union classmethod

union(by_child: Mapping[str, Availability]) -> Availability

Combine the availability of the children of a collection.

Available if one child has the flag available, else unknown if one child does not know, else unavailable.

Source code in src/mesqual/datasets/describe.py
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
@classmethod
def union(cls, by_child: Mapping[str, Availability]) -> Availability:
    """Combine the availability of the children of a collection.

    Available if one child has the flag available, else unknown if one
    child does not know, else unavailable.
    """
    states = {availability.available for availability in by_child.values()}
    available = True if True in states else (None if None in states else False)
    groups = {
        label: [name for name, availability in by_child.items() if availability.label == label]
        for label in ('available', 'unknown', 'unavailable')
    }
    reason = '; '.join(f'{label} in {", ".join(names)}' for label, names in groups.items() if names)
    datasets = tuple(name for name, availability in by_child.items() if availability.available is not False)
    return cls(available, reason or 'no children', datasets)

intersection classmethod

intersection(by_child: Mapping[str, Availability]) -> Availability

Combine the availability of children that are all needed, for example of a comparison.

Unavailable if one child has the flag unavailable, else unknown if one child does not know, else available.

Source code in src/mesqual/datasets/describe.py
66
67
68
69
70
71
72
73
74
75
76
@classmethod
def intersection(cls, by_child: Mapping[str, Availability]) -> Availability:
    """Combine the availability of children that are all needed, for example of a comparison.

    Unavailable if one child has the flag unavailable, else unknown if
    one child does not know, else available.
    """
    available = cls._weakest_state(by_child.values()) if by_child else False
    reason = '; '.join(f'{availability.label} in {name}' for name, availability in by_child.items())
    datasets = () if available is False else tuple(by_child)
    return cls(available, reason or 'no children', datasets)

of_inputs classmethod

of_inputs(by_flag: Mapping[str, Availability]) -> Availability

Combine the availability of the flags that a flag needs.

Unavailable if one input is unavailable, else unknown if one input is unknown, else available. The reason names the unavailable or the unknown inputs with their reasons.

Parameters:

Name Type Description Default
by_flag Mapping[str, Availability]

The availability of each input flag.

required

Examples:

>>> Availability.of_inputs({
...     'Node.Results.price': Availability(True, 'prices in the source'),
...     'Node.Results.demand': Availability(False, 'no demand table'),
... }).reason
'unavailable input Node.Results.demand (no demand table)'
Source code in src/mesqual/datasets/describe.py
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
@classmethod
def of_inputs(cls, by_flag: Mapping[str, Availability]) -> Availability:
    """Combine the availability of the flags that a flag needs.

    Unavailable if one input is unavailable, else unknown if one input
    is unknown, else available. The reason names the unavailable or the
    unknown inputs with their reasons.

    Args:
        by_flag: The availability of each input flag.

    Examples:
        >>> Availability.of_inputs({
        ...     'Node.Results.price': Availability(True, 'prices in the source'),
        ...     'Node.Results.demand': Availability(False, 'no demand table'),
        ... }).reason
        'unavailable input Node.Results.demand (no demand table)'
    """
    available = cls._weakest_state(by_flag.values())
    if available is True:
        return cls(True, f'inputs available: {", ".join(by_flag)}' if by_flag else 'no inputs')
    label = 'unavailable' if available is False else 'unknown'
    reason = '; '.join(
        f'{label} input {flag} ({availability.reason})'
        for flag, availability in by_flag.items()
        if availability.available is available
    )
    return cls(available, reason)

FlagInfo dataclass

Everything mesqual knows about one flag of a dataset, without its data.

Attributes:

Name Type Description
flag str

The flag.

spec FlagSpec

The flag spec.

base FlagSpec | None

The base spec of a layer spec, else None.

aggregation Any

The aggregation of a layer spec, else None.

item_type ItemTypeEnum

Item type of the flag.

variant dict[str, Any]

The variant values of the flag.

description str

Spec description and one line per variant value.

unit Unit | None

Unit of the values, None for a model flag.

unit_source str

'spec', or 'base' for a layer spec.

quantity_type QuantityTypeEnum

Quantity type of the values.

quantity_type_source str

'spec' (an ambiguous unit), 'unit', or 'base' for a layer spec.

schema FrameSchema | None

Index and column levels of the result.

params tuple[ParamInfo, ...]

The flag parameters and the dataset parameters, with their effective defaults.

fill_value float | None

Value for objects or rows that one side lacks, None for no fill.

model_flag str | None

The linked model flag, or None.

available bool | None

True, False or None for unknown.

availability_reason str

Where the availability comes from.

chain tuple[str, ...]

For a platform dataset: the interpreter chain of the flag, from the original interpreter to the one that answers.

dependencies tuple[FetchRecord, ...]

The recorded dependencies of the last fetch of the flag without arguments. Empty before the first fetch.

Source code in src/mesqual/datasets/describe.py
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
@dataclass(frozen=True)
class FlagInfo:
    """Everything mesqual knows about one flag of a dataset, without its data.

    Attributes:
        flag: The flag.
        spec: The flag spec.
        base: The base spec of a layer spec, else `None`.
        aggregation: The aggregation of a layer spec, else `None`.
        item_type: Item type of the flag.
        variant: The variant values of the flag.
        description: Spec description and one line per variant value.
        unit: Unit of the values, `None` for a model flag.
        unit_source: `'spec'`, or `'base'` for a layer spec.
        quantity_type: Quantity type of the values.
        quantity_type_source: `'spec'` (an ambiguous unit), `'unit'`, or `'base'` for a layer spec.
        schema: Index and column levels of the result.
        params: The flag parameters and the dataset parameters, with their effective defaults.
        fill_value: Value for objects or rows that one side lacks, `None` for no fill.
        model_flag: The linked model flag, or `None`.
        available: `True`, `False` or `None` for unknown.
        availability_reason: Where the availability comes from.
        chain: For a platform dataset: the interpreter chain of the flag,
            from the original interpreter to the one that answers.
        dependencies: The recorded dependencies of the last fetch of the
            flag without arguments. Empty before the first fetch.
    """

    flag: str
    spec: FlagSpec
    base: FlagSpec | None
    aggregation: Any
    item_type: ItemTypeEnum
    variant: dict[str, Any]
    description: str
    unit: Units.Unit | None
    unit_source: str
    quantity_type: QuantityTypeEnum
    quantity_type_source: str
    schema: FrameSchema | None
    params: tuple[ParamInfo, ...]
    fill_value: float | None
    model_flag: str | None
    available: bool | None
    availability_reason: str
    chain: tuple[str, ...] = ()
    dependencies: tuple[FetchRecord, ...] = field(default=(), repr=False)

    def as_record(self) -> dict:
        """Return the info as a plain dict, ready for JSON.

        Spec, base and aggregation appear by name, the schema as
        `{'index': [level, ...], 'columns': [level, ...], 'dtype': ...}`
        with one dict per level (`name`, `role`).
        """
        record = {k: v for k, v in self.__dict__.items() if k not in ('spec', 'base', 'aggregation', 'dependencies')}
        record['spec'] = self.spec.name
        record['base'] = None if self.base is None else self.base.name
        record['aggregation'] = None if self.aggregation is None else type(self.aggregation).__name__
        record['params'] = [p.as_record() for p in self.params]
        record['dependencies'] = [f'{d.dataset_name}: {d.flag}' for d in self.dependencies]
        return _Plain.of(record)

as_record

as_record() -> dict

Return the info as a plain dict, ready for JSON.

Spec, base and aggregation appear by name, the schema as {'index': [level, ...], 'columns': [level, ...], 'dtype': ...} with one dict per level (name, role).

Source code in src/mesqual/datasets/describe.py
203
204
205
206
207
208
209
210
211
212
213
214
215
216
def as_record(self) -> dict:
    """Return the info as a plain dict, ready for JSON.

    Spec, base and aggregation appear by name, the schema as
    `{'index': [level, ...], 'columns': [level, ...], 'dtype': ...}`
    with one dict per level (`name`, `role`).
    """
    record = {k: v for k, v in self.__dict__.items() if k not in ('spec', 'base', 'aggregation', 'dependencies')}
    record['spec'] = self.spec.name
    record['base'] = None if self.base is None else self.base.name
    record['aggregation'] = None if self.aggregation is None else type(self.aggregation).__name__
    record['params'] = [p.as_record() for p in self.params]
    record['dependencies'] = [f'{d.dataset_name}: {d.flag}' for d in self.dependencies]
    return _Plain.of(record)

ParamInfo dataclass

One parameter of a flag: a flag parameter of the spec or a dataset parameter of the dataset class.

Attributes:

Name Type Description
name str

Keyword argument of fetch.

owner str

'spec' for a flag parameter, 'dataset' for a dataset parameter (fetch_params).

description str

What the parameter selects.

values Mapping[Any, str] | str | None

Allowed values with their descriptions, None for an open parameter, or 'resolved per dataset' on a dataset class.

default Any

Effective default. None for a required parameter.

default_source str

'spec', 'dataset' (the default of the dimension), 'config' (param_defaults), 'required' (no default), or 'per dataset' on a collection whose children resolve different defaults (default is then None).

Source code in src/mesqual/datasets/describe.py
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
@dataclass(frozen=True)
class ParamInfo:
    """One parameter of a flag: a flag parameter of the spec or a dataset parameter of the dataset class.

    Attributes:
        name: Keyword argument of `fetch`.
        owner: `'spec'` for a flag parameter, `'dataset'` for a dataset parameter (`fetch_params`).
        description: What the parameter selects.
        values: Allowed values with their descriptions, `None` for an open
            parameter, or `'resolved per dataset'` on a dataset class.
        default: Effective default. `None` for a required parameter.
        default_source: `'spec'`, `'dataset'` (the default of the dimension),
            `'config'` (`param_defaults`), `'required'` (no default), or
            `'per dataset'` on a collection whose children resolve
            different defaults (`default` is then `None`).
    """

    name: str
    owner: str
    description: str
    values: Mapping[Any, str] | str | None
    default: Any
    default_source: str

    def as_record(self) -> dict:
        """Return the parameter as a plain dict, ready for JSON.

        `values` is a list of `{'value': ..., 'description': ...}`, so the
        default is equal to the `value` of one entry.
        """
        record = _Plain.of(self.__dict__)
        if isinstance(self.values, Mapping):
            record['values'] = [
                {'value': _Plain.of(value), 'description': description} for value, description in self.values.items()
            ]
        return record

    def __str__(self) -> str:
        return f'{self.name}={self.default!r} ({self.default_source})'

as_record

as_record() -> dict

Return the parameter as a plain dict, ready for JSON.

values is a list of {'value': ..., 'description': ...}, so the default is equal to the value of one entry.

Source code in src/mesqual/datasets/describe.py
138
139
140
141
142
143
144
145
146
147
148
149
def as_record(self) -> dict:
    """Return the parameter as a plain dict, ready for JSON.

    `values` is a list of `{'value': ..., 'description': ...}`, so the
    default is equal to the `value` of one entry.
    """
    record = _Plain.of(self.__dict__)
    if isinstance(self.values, Mapping):
        record['values'] = [
            {'value': _Plain.of(value), 'description': description} for value, description in self.values.items()
        ]
    return record

DatasetConfig dataclass

Base config for the fetch behavior of a dataset.

A platform or a study subclasses it as a dataclass and adds fields with config_field. A dataset declares its config class with Dataset.get_config_type().

Layers resolve in this order, later wins: config class defaults, class config of each class in the MRO (base first), instance config, fetch-time config. A layer overrides only its explicit fields: the constructor arguments, fields assigned after construction, or the keys of a dict. An explicit None overrides like any other value. Set a field to DatasetConfig.CLEAR to reset it to its class default and override the lower layers.

Examples:

>>> prices = dataset.fetch('Node.Results.price', config=dict(auto_sort_index=False))
Source code in src/mesqual/datasets/dataset_config.py
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
@dataclass
class DatasetConfig(metaclass=_DatasetConfigMeta):
    """Base config for the fetch behavior of a dataset.

    A platform or a study subclasses it as a dataclass and adds fields with
    `config_field`. A dataset declares its config class with
    `Dataset.get_config_type()`.

    Layers resolve in this order, later wins: config class defaults, class
    config of each class in the MRO (base first), instance config, fetch-time
    config. A layer overrides only its explicit fields: the constructor
    arguments, fields assigned after construction, or the keys of a dict.
    An explicit `None` overrides like any other value. Set a field to
    `DatasetConfig.CLEAR` to reset it to its class default and override the
    lower layers.

    Examples:
        >>> prices = dataset.fetch('Node.Results.price', config=dict(auto_sort_index=False))
    """

    CLEAR: ClassVar[_ClearType] = _ClearType()

    use_cache: bool = config_field(True, 'Read from and write to the cache of the dataset.')
    auto_sort_index: bool = config_field(True, 'Sort a result by its row index, of any kind, time or not.')
    keep_duplicate_indices: Literal['first', 'last', 'all'] = config_field(
        'first',
        "Rows to keep for duplicate index entries: 'first' or 'last' drops the other rows "
        "with a warning, 'all' keeps every row.",
    )
    time_index_filter: TimeIndexFilter | tuple[TimeIndexFilter, ...] | None = config_field(
        None,
        'Row filter on one level of the time index of time series flags, or several filters '
        'that combine with AND. `level=None` means the datetime level. A result without the '
        'level is returned unchanged, so model flags pass through. '
        'Applies only in the outermost fetch, after the cache.',
    )
    param_defaults: dict[str, Any] | None = config_field(
        None,
        'Default values of flag parameters, by parameter name. A value applies to every flag that '
        'declares a parameter with that name. The outermost fetch raises for a name in the instance or '
        'fetch-time config that no parameter of the dataset declares; a class config may also name a '
        'dataset parameter of any dataset class. Layers merge per key.',
        merge_per_key=True,
    )
    granularity: str | pd.Timedelta | None = config_field(
        None,
        "Target time resolution of time series flags, a pandas frequency such as '15min' or '1h'. "
        '`None`: the native resolution. Applies only in the outermost fetch, after the cache.',
        kw_only=True,
    )

    def __getattribute__(self, name: str) -> Any:
        value = object.__getattribute__(self, name)
        if name[0] != '_' or name == '__dict__':
            recorder = object.__getattribute__(self, '__dict__').get('_read_recorder')
            if recorder is not None:
                self._record_read(recorder, name, value)
        return value

    @property
    def _state(self) -> dict:
        """The instance dict, read without recording."""
        return object.__getattribute__(self, '__dict__')

    def _record_read(self, recorder, name: str, value: Any) -> None:
        if name == '__dict__':
            for field_name in self._field_names():
                recorder(field_name, object.__getattribute__(self, field_name))
        elif name in type(self).__dataclass_fields__:
            recorder(name, value)

    def _watched(self, recorder) -> DatasetConfig:
        """Return a copy that reports each field read to `recorder(name, value)`."""
        watched = copy.copy(self)
        watched._state['_read_recorder'] = recorder
        return watched

    def __getstate__(self) -> dict:
        state = dict(self._state)
        recorder = state.pop('_read_recorder', None)
        if recorder is not None:
            self._record_read(recorder, '__dict__', None)
        return state

    def __setattr__(self, name: str, value: Any) -> None:
        if '_read_recorder' in self._state:
            raise AttributeError(
                f'The effective config of a fetch is read-only; cannot set {name!r}. '
                f'To pass a changed config on, use config.merge({{{name!r}: ...}}).'
            )
        super().__setattr__(name, value)
        if not name.startswith('_') and '_explicit_fields' in self._state:
            super().__setattr__('_explicit_fields', self._explicit_fields | {name})
            super().__setattr__('_cleared_fields', self._cleared_fields - {name})

    @property
    def _cleared_fields(self) -> frozenset[str]:
        return self._state.get('_cleared_fields_set', frozenset())

    @_cleared_fields.setter
    def _cleared_fields(self, names: frozenset[str]) -> None:
        self._state['_cleared_fields_set'] = names

    @property
    def explicit_fields(self) -> frozenset[str]:
        """Names of the fields that this config sets explicitly."""
        return self._state.get('_explicit_fields', frozenset())

    @classmethod
    def _merges_per_key(cls, name: str) -> bool:
        field = cls.__dataclass_fields__.get(name)
        return field is not None and field.metadata.get('merge_per_key', False)

    def _unknown_fields(self) -> frozenset[str]:
        return self.explicit_fields - self._field_names()

    @classmethod
    def _field_names(cls) -> frozenset[str]:
        return frozenset(f.name for f in dataclasses.fields(cls))

    @classmethod
    def _field_default(cls, name: str) -> Any:
        field = {f.name: f for f in dataclasses.fields(cls)}[name]
        if field.default_factory is not dataclasses.MISSING:
            return field.default_factory()
        return field.default

    def merge(self, other: Optional[DatasetConfigType | dict]) -> DatasetConfigType:
        """Return a new config with the explicit fields of `other` on top of this one.

        The result has the type of this config. Its explicit fields are the
        union of the explicit fields of both layers. An explicit `None` in
        `other` overrides like any other value. A field declared with
        `merge_per_key` (`param_defaults`) adds the keys of `other` to its
        dict. `CLEAR` resets a field to the
        class default of this config; the result remembers the reset, so a
        collection passes `CLEAR` on and each child resolves it against its
        own default. A name that this config class does not define is kept
        as is, so a collection can pass it on to its children.

        Args:
            other: Upper layer: `None`, a dict, or a config of any type.

        Returns:
            The merged config.

        Examples:
            >>> base = DatasetConfig(use_cache=False)
            >>> base.merge({'auto_sort_index': False}).use_cache
            False
        """
        merged = copy.copy(self)
        for name, value in self._explicit_items(other):
            if value is DatasetConfig.CLEAR and name in self._field_names():
                setattr(merged, name, self._field_default(name))
                merged._cleared_fields = merged._cleared_fields | {name}
            elif self._merges_per_key(name) and isinstance(value, dict):
                setattr(merged, name, {**(getattr(merged, name) or {}), **value})
            else:
                setattr(merged, name, value)
        return merged

    @staticmethod
    def _explicit_items(other: Optional[DatasetConfig | dict]) -> Iterable[tuple[str, Any]]:
        if other is None:
            return []
        if isinstance(other, dict):
            return other.items()
        if isinstance(other, DatasetConfig):
            return [
                (name, DatasetConfig.CLEAR if name in other._cleared_fields else getattr(other, name))
                for name in sorted(other.explicit_fields)
            ]
        raise TypeError(f'Config must be a dict or a {DatasetConfig.__name__}, got {type(other)}.')

    def __repr__(self) -> str:
        recorder = self._state.get('_read_recorder')
        if recorder is not None:
            self._record_read(recorder, '__dict__', None)
        attrs = {k: v for k, v in self._state.items() if not k.startswith('_')}
        return f'{self.__class__.__name__}({attrs})'

explicit_fields property

explicit_fields: frozenset[str]

Names of the fields that this config sets explicitly.

merge

merge(other: DatasetConfigType | dict | None) -> DatasetConfigType

Return a new config with the explicit fields of other on top of this one.

The result has the type of this config. Its explicit fields are the union of the explicit fields of both layers. An explicit None in other overrides like any other value. A field declared with merge_per_key (param_defaults) adds the keys of other to its dict. CLEAR resets a field to the class default of this config; the result remembers the reset, so a collection passes CLEAR on and each child resolves it against its own default. A name that this config class does not define is kept as is, so a collection can pass it on to its children.

Parameters:

Name Type Description Default
other DatasetConfigType | dict | None

Upper layer: None, a dict, or a config of any type.

required

Returns:

Type Description
DatasetConfigType

The merged config.

Examples:

>>> base = DatasetConfig(use_cache=False)
>>> base.merge({'auto_sort_index': False}).use_cache
False
Source code in src/mesqual/datasets/dataset_config.py
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
def merge(self, other: Optional[DatasetConfigType | dict]) -> DatasetConfigType:
    """Return a new config with the explicit fields of `other` on top of this one.

    The result has the type of this config. Its explicit fields are the
    union of the explicit fields of both layers. An explicit `None` in
    `other` overrides like any other value. A field declared with
    `merge_per_key` (`param_defaults`) adds the keys of `other` to its
    dict. `CLEAR` resets a field to the
    class default of this config; the result remembers the reset, so a
    collection passes `CLEAR` on and each child resolves it against its
    own default. A name that this config class does not define is kept
    as is, so a collection can pass it on to its children.

    Args:
        other: Upper layer: `None`, a dict, or a config of any type.

    Returns:
        The merged config.

    Examples:
        >>> base = DatasetConfig(use_cache=False)
        >>> base.merge({'auto_sort_index': False}).use_cache
        False
    """
    merged = copy.copy(self)
    for name, value in self._explicit_items(other):
        if value is DatasetConfig.CLEAR and name in self._field_names():
            setattr(merged, name, self._field_default(name))
            merged._cleared_fields = merged._cleared_fields | {name}
        elif self._merges_per_key(name) and isinstance(value, dict):
            setattr(merged, name, {**(getattr(merged, name) or {}), **value})
        else:
            setattr(merged, name, value)
    return merged

DatasetConfigManager

Registry of class configs, the class layer of the config hierarchy.

Class configs are global state. A class config applies to all instances of that dataset class and of its subclasses.

Examples:

>>> from mesqual.testing import DummyDataset
>>> DatasetConfigManager.set_class_config(DummyDataset, DatasetConfig(use_cache=False))
>>> DatasetConfigManager.update_class_config_kwargs(DummyDataset, auto_sort_index=False)
>>> DatasetConfigManager.clear_class_configs()
Source code in src/mesqual/datasets/dataset_config.py
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
class DatasetConfigManager:
    """Registry of class configs, the class layer of the config hierarchy.

    Class configs are global state. A class config applies to all instances of
    that dataset class and of its subclasses.

    Examples:
        >>> from mesqual.testing import DummyDataset
        >>> DatasetConfigManager.set_class_config(DummyDataset, DatasetConfig(use_cache=False))
        >>> DatasetConfigManager.update_class_config_kwargs(DummyDataset, auto_sort_index=False)
        >>> DatasetConfigManager.clear_class_configs()
    """

    _class_configs: dict[Type[Dataset], DatasetConfig] = {}

    @classmethod
    def set_class_config(cls, dataset_class: Type[Dataset], config: DatasetConfig) -> None:
        """Set the class config of a dataset class.

        Args:
            dataset_class: Dataset class to configure.
            config: Config whose explicit fields override the config defaults.
        """
        cls._class_configs[dataset_class] = config

    @classmethod
    def update_class_config_kwargs(cls, dataset_class: Type[Dataset], **config_kwargs: Any) -> None:
        """Set fields on the existing class config of a dataset class.

        Args:
            dataset_class: Dataset class whose class config to update.
            **config_kwargs: Field names and values.

        Raises:
            KeyError: The dataset class has no class config.
        """
        if dataset_class not in cls._class_configs:
            raise KeyError(f'{dataset_class.__name__} has no class config. Call set_class_config first.')
        for k, v in config_kwargs.items():
            setattr(cls._class_configs[dataset_class], k, v)

    @classmethod
    def clear_class_configs(cls) -> None:
        """Remove all class configs."""
        cls._class_configs.clear()

    @classmethod
    def get_layers(
        cls,
        dataset_class: Type[Dataset],
        instance_config: Optional[DatasetConfig] = None,
    ) -> list[tuple[str, DatasetConfig]]:
        """Return the config layers of a dataset class, lowest first.

        Args:
            dataset_class: Dataset class.
            instance_config: Optional instance config, the top layer.

        Returns:
            Pairs of layer label and config: `'default'`, `'class <Name>'` for
                each class config along the MRO (base first), and `'instance'`.
        """
        layers = [('default', dataset_class.get_config_type()())]
        for klass in reversed(dataset_class.__mro__):
            class_config = cls._class_configs.get(klass)
            if class_config is not None:
                layers.append((f'class {klass.__name__}', class_config))
        if instance_config is not None:
            layers.append(('instance', instance_config))
        return layers

    @classmethod
    def get_effective_config(
        cls,
        dataset_class: Type[Dataset],
        instance_config: Optional[DatasetConfig] = None,
    ) -> DatasetConfig:
        """Resolve the config of a dataset class without the fetch-time layer.

        Args:
            dataset_class: Dataset class.
            instance_config: Optional instance config.

        Returns:
            The merged config, of the config type of the dataset class.
        """
        layers = cls.get_layers(dataset_class, instance_config)
        effective = layers[0][1]
        for _, config in layers[1:]:
            effective = effective.merge(config)
        return effective

set_class_config classmethod

set_class_config(dataset_class: Type[Dataset], config: DatasetConfig) -> None

Set the class config of a dataset class.

Parameters:

Name Type Description Default
dataset_class Type[Dataset]

Dataset class to configure.

required
config DatasetConfig

Config whose explicit fields override the config defaults.

required
Source code in src/mesqual/datasets/dataset_config.py
286
287
288
289
290
291
292
293
294
@classmethod
def set_class_config(cls, dataset_class: Type[Dataset], config: DatasetConfig) -> None:
    """Set the class config of a dataset class.

    Args:
        dataset_class: Dataset class to configure.
        config: Config whose explicit fields override the config defaults.
    """
    cls._class_configs[dataset_class] = config

update_class_config_kwargs classmethod

update_class_config_kwargs(dataset_class: Type[Dataset], **config_kwargs: Any) -> None

Set fields on the existing class config of a dataset class.

Parameters:

Name Type Description Default
dataset_class Type[Dataset]

Dataset class whose class config to update.

required
**config_kwargs Any

Field names and values.

{}

Raises:

Type Description
KeyError

The dataset class has no class config.

Source code in src/mesqual/datasets/dataset_config.py
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
@classmethod
def update_class_config_kwargs(cls, dataset_class: Type[Dataset], **config_kwargs: Any) -> None:
    """Set fields on the existing class config of a dataset class.

    Args:
        dataset_class: Dataset class whose class config to update.
        **config_kwargs: Field names and values.

    Raises:
        KeyError: The dataset class has no class config.
    """
    if dataset_class not in cls._class_configs:
        raise KeyError(f'{dataset_class.__name__} has no class config. Call set_class_config first.')
    for k, v in config_kwargs.items():
        setattr(cls._class_configs[dataset_class], k, v)

clear_class_configs classmethod

clear_class_configs() -> None

Remove all class configs.

Source code in src/mesqual/datasets/dataset_config.py
312
313
314
315
@classmethod
def clear_class_configs(cls) -> None:
    """Remove all class configs."""
    cls._class_configs.clear()

get_layers classmethod

get_layers(dataset_class: Type[Dataset], instance_config: DatasetConfig | None = None) -> list[tuple[str, DatasetConfig]]

Return the config layers of a dataset class, lowest first.

Parameters:

Name Type Description Default
dataset_class Type[Dataset]

Dataset class.

required
instance_config DatasetConfig | None

Optional instance config, the top layer.

None

Returns:

Type Description
list[tuple[str, DatasetConfig]]

Pairs of layer label and config: 'default', 'class <Name>' for each class config along the MRO (base first), and 'instance'.

Source code in src/mesqual/datasets/dataset_config.py
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
@classmethod
def get_layers(
    cls,
    dataset_class: Type[Dataset],
    instance_config: Optional[DatasetConfig] = None,
) -> list[tuple[str, DatasetConfig]]:
    """Return the config layers of a dataset class, lowest first.

    Args:
        dataset_class: Dataset class.
        instance_config: Optional instance config, the top layer.

    Returns:
        Pairs of layer label and config: `'default'`, `'class <Name>'` for
            each class config along the MRO (base first), and `'instance'`.
    """
    layers = [('default', dataset_class.get_config_type()())]
    for klass in reversed(dataset_class.__mro__):
        class_config = cls._class_configs.get(klass)
        if class_config is not None:
            layers.append((f'class {klass.__name__}', class_config))
    if instance_config is not None:
        layers.append(('instance', instance_config))
    return layers

get_effective_config classmethod

get_effective_config(dataset_class: Type[Dataset], instance_config: DatasetConfig | None = None) -> DatasetConfig

Resolve the config of a dataset class without the fetch-time layer.

Parameters:

Name Type Description Default
dataset_class Type[Dataset]

Dataset class.

required
instance_config DatasetConfig | None

Optional instance config.

None

Returns:

Type Description
DatasetConfig

The merged config, of the config type of the dataset class.

Source code in src/mesqual/datasets/dataset_config.py
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
@classmethod
def get_effective_config(
    cls,
    dataset_class: Type[Dataset],
    instance_config: Optional[DatasetConfig] = None,
) -> DatasetConfig:
    """Resolve the config of a dataset class without the fetch-time layer.

    Args:
        dataset_class: Dataset class.
        instance_config: Optional instance config.

    Returns:
        The merged config, of the config type of the dataset class.
    """
    layers = cls.get_layers(dataset_class, instance_config)
    effective = layers[0][1]
    for _, config in layers[1:]:
        effective = effective.merge(config)
    return effective

TimeIndexFilter dataclass

Keeps the rows whose value on one level of the time index passes the filter.

The filter acts on the time index of time series flags. level names a level of the time index, for example hour of (day, hour). None means the datetime level. A result without that level is returned unchanged, so model flags pass through.

A row passes when its value is in include (if set) and not in exclude (if set). include and exclude can mix plain values and Range objects.

On a datetime level, a datetime.date or an ISO date string matches the local calendar date of the row, and a Timestamp matches that exact timestamp. On any other level, values match directly.

On a MultiIndex, the other levels stay.

Attributes:

Name Type Description
level str | None

Level of the time index to filter. None: the datetime level.

include Collection | None

Values or ranges to keep. Put a tuple value in a list ([(1, 2)]), else its elements count as separate values.

exclude Collection | None

Values or ranges to drop. Same form as include.

Examples:

>>> march_without_29th = TimeIndexFilter.dates(include=[Range('2024-03-01', '2024-03-31')], exclude=['2024-03-29'])
>>> prices = dataset.fetch('Node.Results.price', config={'time_index_filter': march_without_29th})
>>> sorted(set(prices.index.day))
[28, 30, 31]
>>> first_day_periods = TimeIndexFilter(level='period', include=[Range(1, 24)])
Source code in src/mesqual/datasets/time_index_filter.py
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
@dataclass(frozen=True)
class TimeIndexFilter:
    """Keeps the rows whose value on one level of the time index passes the filter.

    The filter acts on the time index of time series flags. `level` names a
    level of the time index, for example `hour` of `(day, hour)`.
    `None` means the datetime level. A result without that level is returned
    unchanged, so model flags pass through.

    A row passes when its value is in `include` (if set) and not in `exclude`
    (if set). `include` and `exclude` can mix plain values and `Range` objects.

    On a datetime level, a `datetime.date` or an ISO date string matches the
    local calendar date of the row, and a `Timestamp` matches that exact
    timestamp. On any other level, values match directly.

    On a `MultiIndex`, the other levels stay.

    Attributes:
        level: Level of the time index to filter. `None`: the datetime level.
        include: Values or ranges to keep. Put a tuple value in a list
            (`[(1, 2)]`), else its elements count as separate values.
        exclude: Values or ranges to drop. Same form as `include`.

    Examples:
        >>> march_without_29th = TimeIndexFilter.dates(include=[Range('2024-03-01', '2024-03-31')], exclude=['2024-03-29'])
        >>> prices = dataset.fetch('Node.Results.price', config={'time_index_filter': march_without_29th})
        >>> sorted(set(prices.index.day))
        [28, 30, 31]
        >>> first_day_periods = TimeIndexFilter(level='period', include=[Range(1, 24)])
    """

    level: str | None = None
    include: Collection | None = None
    exclude: Collection | None = None

    @classmethod
    def dates(cls, include: Collection | None = None, exclude: Collection | None = None) -> TimeIndexFilter:
        """Short form for a filter on calendar dates of the time level.

        Args:
            include: Dates, timestamps or ranges to keep.
            exclude: Dates, timestamps or ranges to drop.

        Returns:
            A filter on the time level of the result.
        """
        return cls(level=None, include=include, exclude=exclude)

    def apply(self, data: pd.Series | pd.DataFrame, time_level: str | int | None) -> pd.Series | pd.DataFrame:
        """Filter the rows of `data`.

        Args:
            data: Result to filter.
            time_level: Name or position of the time level of `data`, used
                when `level` is `None`. `None`: the result has no time level.

        Returns:
            The rows that pass the filter.
        """
        level = self.level if self.level is not None else time_level
        if level is None or not self._has_level(data.index, level):
            return data
        values = data.index.get_level_values(level)
        mask = np.ones(len(values), dtype=bool)
        if self.include is not None:
            mask &= self._matches(values, self.include)
        if self.exclude is not None:
            mask &= ~self._matches(values, self.exclude)
        return data.loc[mask]

    @staticmethod
    def _has_level(index: pd.Index, level: str | int) -> bool:
        if isinstance(level, int):
            return level < index.nlevels
        return level in index.names

    @classmethod
    def _matches(cls, values: pd.Index, items: Collection) -> np.ndarray:
        if isinstance(items, (str, Range, datetime.date)) or not isinstance(items, Collection):
            items = [items]
        if isinstance(values, pd.DatetimeIndex):
            return _DatetimeMatcher(values).matches(items)
        return _ValueMatcher(values).matches(items)

dates classmethod

dates(include: Collection | None = None, exclude: Collection | None = None) -> TimeIndexFilter

Short form for a filter on calendar dates of the time level.

Parameters:

Name Type Description Default
include Collection | None

Dates, timestamps or ranges to keep.

None
exclude Collection | None

Dates, timestamps or ranges to drop.

None

Returns:

Type Description
TimeIndexFilter

A filter on the time level of the result.

Source code in src/mesqual/datasets/time_index_filter.py
68
69
70
71
72
73
74
75
76
77
78
79
@classmethod
def dates(cls, include: Collection | None = None, exclude: Collection | None = None) -> TimeIndexFilter:
    """Short form for a filter on calendar dates of the time level.

    Args:
        include: Dates, timestamps or ranges to keep.
        exclude: Dates, timestamps or ranges to drop.

    Returns:
        A filter on the time level of the result.
    """
    return cls(level=None, include=include, exclude=exclude)

apply

apply(data: Series | DataFrame, time_level: str | int | None) -> Series | DataFrame

Filter the rows of data.

Parameters:

Name Type Description Default
data Series | DataFrame

Result to filter.

required
time_level str | int | None

Name or position of the time level of data, used when level is None. None: the result has no time level.

required

Returns:

Type Description
Series | DataFrame

The rows that pass the filter.

Source code in src/mesqual/datasets/time_index_filter.py
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
def apply(self, data: pd.Series | pd.DataFrame, time_level: str | int | None) -> pd.Series | pd.DataFrame:
    """Filter the rows of `data`.

    Args:
        data: Result to filter.
        time_level: Name or position of the time level of `data`, used
            when `level` is `None`. `None`: the result has no time level.

    Returns:
        The rows that pass the filter.
    """
    level = self.level if self.level is not None else time_level
    if level is None or not self._has_level(data.index, level):
        return data
    values = data.index.get_level_values(level)
    mask = np.ones(len(values), dtype=bool)
    if self.include is not None:
        mask &= self._matches(values, self.include)
    if self.exclude is not None:
        mask &= ~self._matches(values, self.exclude)
    return data.loc[mask]

Range dataclass

Inclusive range of index values.

On a datetime level, start and end are dates (datetime.date or ISO date strings), which match every day from start to end, or Timestamps, which match start <= timestamp <= end. On any other level, the range matches start <= value <= end.

Attributes:

Name Type Description
start Any

First value of the range.

end Any

Last value of the range.

Source code in src/mesqual/datasets/time_index_filter.py
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
@dataclass(frozen=True)
class Range:
    """Inclusive range of index values.

    On a datetime level, `start` and `end` are dates (`datetime.date` or ISO
    date strings), which match every day from start to end, or `Timestamp`s,
    which match `start <= timestamp <= end`. On any other level, the range
    matches `start <= value <= end`.

    Attributes:
        start: First value of the range.
        end: Last value of the range.
    """

    start: Any
    end: Any

AmbiguousOverrideError

Bases: ValueError

Raised when more than one interpreter, override or spec claims the same flag or interpreter.

Two overrides of one interpreter on one class or on sibling classes, two answering interpreters with one spec name, or two specs that give one flag name raise this error.

Source code in src/mesqual/flag/exceptions.py
14
15
16
17
18
19
20
class AmbiguousOverrideError(ValueError):
    """Raised when more than one interpreter, override or spec claims the same flag or interpreter.

    Two overrides of one interpreter on one class or on sibling classes,
    two answering interpreters with one spec name, or two specs that give
    one flag name raise this error.
    """

FlagUnavailableError

Bases: ValueError

A flag is not available on a dataset.

fetch raises it for an accepted flag whose availability is False. It is a subclass of ValueError. Dimension.check_value raises it when the values function of a flag family fetches an unavailable flag. For the values of such a family, see Dimension.values_for.

Parameters:

Name Type Description Default
flag str

The flag.

required
dataset str

The name of the dataset.

required
reason str

Why the flag is not available, as in Dataset.availability(flag).reason.

required

Attributes:

Name Type Description
flag

The flag.

dataset

The name of the dataset.

reason

Why the flag is not available.

Source code in src/mesqual/flag/exceptions.py
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
class FlagUnavailableError(ValueError):
    """A flag is not available on a dataset.

    `fetch` raises it for an accepted flag whose availability is `False`. It
    is a subclass of `ValueError`. `Dimension.check_value` raises it when the
    values function of a flag family fetches an unavailable flag. For the
    values of such a family, see `Dimension.values_for`.

    Args:
        flag: The flag.
        dataset: The name of the dataset.
        reason: Why the flag is not available, as in `Dataset.availability(flag).reason`.

    Attributes:
        flag: The flag.
        dataset: The name of the dataset.
        reason: Why the flag is not available.
    """

    def __init__(self, flag: str, dataset: str, reason: str):
        super().__init__(f'Flag {flag} is not available on dataset {dataset}: {reason}')
        self.flag = flag
        self.dataset = dataset
        self.reason = reason

    def __reduce__(self):
        return type(self), (self.flag, self.dataset, self.reason)

FetchContext

The stack of open fetch frames of the current context.

In _fetch, call FetchContext.record_file_read(path) for each source file, so an edited file makes the cache entries that read it stale. The other methods are for mesqual itself.

The stack lives in context variables. Code in a thread started inside a fetch runs with contextvars.copy_context().run(...), else its fetches count as outermost and are not recorded in the open frames.

Source code in src/mesqual/datasets/fetch_context.py
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
class FetchContext:
    """The stack of open fetch frames of the current context.

    In `_fetch`, call `FetchContext.record_file_read(path)` for each source
    file, so an edited file makes the cache entries that read it stale. The
    other methods are for mesqual itself.

    The stack lives in context variables. Code in a thread started inside a fetch
    runs with `contextvars.copy_context().run(...)`, else its fetches count as
    outermost and are not recorded in the open frames.
    """

    _frames: contextvars.ContextVar[tuple[FetchFrame, ...]] = contextvars.ContextVar('mesqual_fetch_frames', default=())
    _recording: contextvars.ContextVar[bool] = contextvars.ContextVar('mesqual_fetch_recording', default=True)

    @classmethod
    def current(cls) -> FetchFrame | None:
        """The innermost open frame, or `None` outside of a fetch."""
        frames = cls._frames.get()
        return frames[-1] if frames else None

    @classmethod
    def is_outermost(cls) -> bool:
        """`True` if no fetch is open, so the next fetch is the outermost one."""
        return not cls._frames.get()

    @classmethod
    def is_recording(cls) -> bool:
        """`False` while mesqual prepares a fetch, so its own config reads are not recorded."""
        return cls._recording.get()

    @classmethod
    @contextlib.contextmanager
    def frame(cls, record: FetchRecord, dataset: Any = None) -> Iterator[FetchFrame]:
        """Open a frame for one fetch call of `dataset`. On exit, its record goes to the enclosing frame.

        If the call raises, the record has `raised=True`.
        """
        frame = FetchFrame(record, dataset)
        parent = cls.current()
        frames_token = cls._frames.set(cls._frames.get() + (frame,))
        recording_token = cls._recording.set(True)
        try:
            yield frame
        except BaseException:
            frame.raised = True
            raise
        finally:
            cls._recording.reset(recording_token)
            cls._frames.reset(frames_token)
            frame._close_into(parent)

    @classmethod
    @contextlib.contextmanager
    def internal(cls) -> Iterator[None]:
        """Suspend the recording of config reads, for mesqual code that prepares a fetch."""
        token = cls._recording.set(False)
        try:
            yield
        finally:
            cls._recording.reset(token)

    @classmethod
    def record_file_read(cls, path: str | os.PathLike) -> None:
        """Record that the open fetch read the file at `path`: path, size, modification time, content hash.

        Call it in `_fetch` for each source file that the result depends on,
        before the file is read, so a change during the read makes the
        entry stale.
        Call it in the thread of the fetch, or in a thread that runs with
        `contextvars.copy_context().run(...)`.
        A cache entry is up to date only if each recorded file is unchanged,
        so an edited file makes stale exactly the entries whose fetches read
        it. Outside of a fetch, it does nothing. The content hash is computed
        only if a fetch on the stack can store its result in a cache.

        Raises:
            ValueError: `path` is a directory.
            OSError: The file does not exist or cannot be read.

        Examples:
            ```python
            def _fetch(self, flag, effective_config, **params):
                FetchContext.record_file_read(self.path)
                return pd.read_csv(self.path)
            ```
        """
        frame = cls.current()
        if frame is not None and not frame.closed and cls.is_recording():
            file_read = FileRead.of(path, with_hash=any(f.stores for f in cls._frames.get()))
            frame.file_reads[file_read.path] = file_read

    @staticmethod
    def watch(config: DatasetConfig, frame: FetchFrame) -> DatasetConfig:
        """Return a copy of `config` whose field reads go to `frame`, and store it as `frame.watched_config`."""
        frame.watched_config = config._watched(frame.record_config_read)
        return frame.watched_config

current classmethod

current() -> FetchFrame | None

The innermost open frame, or None outside of a fetch.

Source code in src/mesqual/datasets/fetch_context.py
426
427
428
429
430
@classmethod
def current(cls) -> FetchFrame | None:
    """The innermost open frame, or `None` outside of a fetch."""
    frames = cls._frames.get()
    return frames[-1] if frames else None

is_outermost classmethod

is_outermost() -> bool

True if no fetch is open, so the next fetch is the outermost one.

Source code in src/mesqual/datasets/fetch_context.py
432
433
434
435
@classmethod
def is_outermost(cls) -> bool:
    """`True` if no fetch is open, so the next fetch is the outermost one."""
    return not cls._frames.get()

is_recording classmethod

is_recording() -> bool

False while mesqual prepares a fetch, so its own config reads are not recorded.

Source code in src/mesqual/datasets/fetch_context.py
437
438
439
440
@classmethod
def is_recording(cls) -> bool:
    """`False` while mesqual prepares a fetch, so its own config reads are not recorded."""
    return cls._recording.get()

frame classmethod

frame(record: FetchRecord, dataset: Any = None) -> Iterator[FetchFrame]

Open a frame for one fetch call of dataset. On exit, its record goes to the enclosing frame.

If the call raises, the record has raised=True.

Source code in src/mesqual/datasets/fetch_context.py
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
@classmethod
@contextlib.contextmanager
def frame(cls, record: FetchRecord, dataset: Any = None) -> Iterator[FetchFrame]:
    """Open a frame for one fetch call of `dataset`. On exit, its record goes to the enclosing frame.

    If the call raises, the record has `raised=True`.
    """
    frame = FetchFrame(record, dataset)
    parent = cls.current()
    frames_token = cls._frames.set(cls._frames.get() + (frame,))
    recording_token = cls._recording.set(True)
    try:
        yield frame
    except BaseException:
        frame.raised = True
        raise
    finally:
        cls._recording.reset(recording_token)
        cls._frames.reset(frames_token)
        frame._close_into(parent)

internal classmethod

internal() -> Iterator[None]

Suspend the recording of config reads, for mesqual code that prepares a fetch.

Source code in src/mesqual/datasets/fetch_context.py
463
464
465
466
467
468
469
470
471
@classmethod
@contextlib.contextmanager
def internal(cls) -> Iterator[None]:
    """Suspend the recording of config reads, for mesqual code that prepares a fetch."""
    token = cls._recording.set(False)
    try:
        yield
    finally:
        cls._recording.reset(token)

record_file_read classmethod

record_file_read(path: str | PathLike) -> None

Record that the open fetch read the file at path: path, size, modification time, content hash.

Call it in _fetch for each source file that the result depends on, before the file is read, so a change during the read makes the entry stale. Call it in the thread of the fetch, or in a thread that runs with contextvars.copy_context().run(...). A cache entry is up to date only if each recorded file is unchanged, so an edited file makes stale exactly the entries whose fetches read it. Outside of a fetch, it does nothing. The content hash is computed only if a fetch on the stack can store its result in a cache.

Raises:

Type Description
ValueError

path is a directory.

OSError

The file does not exist or cannot be read.

Examples:

def _fetch(self, flag, effective_config, **params):
    FetchContext.record_file_read(self.path)
    return pd.read_csv(self.path)
Source code in src/mesqual/datasets/fetch_context.py
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
@classmethod
def record_file_read(cls, path: str | os.PathLike) -> None:
    """Record that the open fetch read the file at `path`: path, size, modification time, content hash.

    Call it in `_fetch` for each source file that the result depends on,
    before the file is read, so a change during the read makes the
    entry stale.
    Call it in the thread of the fetch, or in a thread that runs with
    `contextvars.copy_context().run(...)`.
    A cache entry is up to date only if each recorded file is unchanged,
    so an edited file makes stale exactly the entries whose fetches read
    it. Outside of a fetch, it does nothing. The content hash is computed
    only if a fetch on the stack can store its result in a cache.

    Raises:
        ValueError: `path` is a directory.
        OSError: The file does not exist or cannot be read.

    Examples:
        ```python
        def _fetch(self, flag, effective_config, **params):
            FetchContext.record_file_read(self.path)
            return pd.read_csv(self.path)
        ```
    """
    frame = cls.current()
    if frame is not None and not frame.closed and cls.is_recording():
        file_read = FileRead.of(path, with_hash=any(f.stores for f in cls._frames.get()))
        frame.file_reads[file_read.path] = file_read

watch staticmethod

watch(config: DatasetConfig, frame: FetchFrame) -> DatasetConfig

Return a copy of config whose field reads go to frame, and store it as frame.watched_config.

Source code in src/mesqual/datasets/fetch_context.py
503
504
505
506
507
@staticmethod
def watch(config: DatasetConfig, frame: FetchFrame) -> DatasetConfig:
    """Return a copy of `config` whose field reads go to `frame`, and store it as `frame.watched_config`."""
    frame.watched_config = config._watched(frame.record_config_read)
    return frame.watched_config

FetchRecord dataclass

What one fetch call used: the call as given, the resolved parameters, the config reads and the dependencies.

A record has two views of a call:

  • The call as given: call_kwargs and call_config. They identify the call.
  • The parameters after resolution: params. _fetch got these values.

Every keyword argument of fetch is a parameter. A flag spec declares flag parameters (FlagSpec.params). A dataset class declares dataset parameters (Dataset.fetch_params). Both resolve the same way: the keyword argument, then param_defaults of the effective config, then the default of the dimension.

Examples:

The spec of 'Node.Results.price' declares the flag parameter solution with the default 'final'. The instance config sets param_defaults={'solution': 'first'}.

>>> high = study.scen.get_dataset('high_demand')
>>> high.set_instance_config_kwargs(param_defaults={'solution': 'first'})
>>> prices = high.fetch('Node.Results.price', config={'use_cache': False})
>>> record = high.get_fetch_record('Node.Results.price', config={'use_cache': False})
>>> record.call_kwargs, record.call_config
({}, {'use_cache': False})
>>> record.params, record.param_default_reads
({'solution': 'first'}, {'solution': 'first'})

solution is not in the call, so it comes from param_defaults. Without that entry, it is 'final' and param_default_reads is empty.

The records form a tree. The record of a fetch lists the fetches that it calls directly as its dependencies, and each of them lists its own. Every nested fetch has its own record, also a call that a platform dataset routes to an interpreter, a link collection to a child, or an override to its base. iter_dependencies walks the tree.

Attributes:

Name Type Description
dataset_class str

f'{cls.__module__}.{cls.__qualname__}' of the dataset.

dataset_name str

name of the dataset.

flag str

The fetched flag, as string.

call_kwargs Mapping[str, Any]

The keyword arguments of the call, as given. Defaults are not filled in.

call_config Mapping[str, Any]

The config override of the call: its explicit fields with their values. DatasetConfig.CLEAR stays as is. INHERIT if the call got the effective config of the enclosing fetch, as a collection passes it to its children. The call then follows the current config of its parent record.

params Mapping[str, Any]

All parameters after resolution: the flag parameters and the dataset parameters, with defaults filled in. For a collection: its own dataset parameters (fetch_params), resolved, and the flag parameters and the dataset parameters of its children that the call gave. A collection resolves neither of the last two.

version tuple

Version identity of the flag (Dataset.version_identity).

param_default_reads Mapping[str, Any]

The entries of param_defaults that parameter resolution used: one per parameter that the call did not give and that param_defaults names.

config_reads Mapping[str, Any]

Each config field that _fetch or the mesqual post-processing read from effective_config, with the value read. Reading the whole config (dataclasses.asdict, vars, copy, pickle, repr) records all fields. param_defaults is here only if _fetch read the field itself.

dependencies tuple[FetchRecord, ...]

The records of the fetches that this fetch called directly, in the order of the calls, each call key once.

raised bool

True if the call raised.

file_reads tuple[FileRead, ...]

The state of each file that _fetch recorded with FetchContext.record_file_read, one FileRead per path.

reads_given_params bool

True if the fetch passed its given parameters (given_params) on to its children, as a collection does.

The values in call_kwargs, call_config, params, param_default_reads and config_reads are deep copies, taken when the call recorded them. A record hashes by its key, so records work in sets. DependencyCheck (mesqual.datasets.dependency_check) decides if a stored record is up to date.

Source code in src/mesqual/datasets/fetch_context.py
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
@dataclass(frozen=True)
class FetchRecord:
    """What one `fetch` call used: the call as given, the resolved parameters, the config reads and the dependencies.

    A record has two views of a call:

    - The call as given: `call_kwargs` and `call_config`. They identify the call.
    - The parameters after resolution: `params`. `_fetch` got these values.

    Every keyword argument of `fetch` is a parameter. A flag spec declares
    flag parameters (`FlagSpec.params`). A dataset class declares dataset
    parameters (`Dataset.fetch_params`). Both resolve the same way: the
    keyword argument, then `param_defaults` of the effective config, then the
    `default` of the dimension.

    Examples:
        The spec of `'Node.Results.price'` declares the flag parameter
        `solution` with the default `'final'`. The instance config sets
        `param_defaults={'solution': 'first'}`.

        >>> high = study.scen.get_dataset('high_demand')
        >>> high.set_instance_config_kwargs(param_defaults={'solution': 'first'})
        >>> prices = high.fetch('Node.Results.price', config={'use_cache': False})
        >>> record = high.get_fetch_record('Node.Results.price', config={'use_cache': False})
        >>> record.call_kwargs, record.call_config
        ({}, {'use_cache': False})
        >>> record.params, record.param_default_reads
        ({'solution': 'first'}, {'solution': 'first'})

        `solution` is not in the call, so it comes from `param_defaults`.
        Without that entry, it is `'final'` and `param_default_reads` is empty.

    The records form a tree. The record of a fetch lists the fetches that it
    calls directly as its dependencies, and each of them lists its own. Every
    nested fetch has its own record, also a call that a platform dataset
    routes to an interpreter, a link collection to a child, or an override to
    its base. `iter_dependencies` walks the tree.

    Attributes:
        dataset_class: `f'{cls.__module__}.{cls.__qualname__}'` of the dataset.
        dataset_name: `name` of the dataset.
        flag: The fetched flag, as string.
        call_kwargs: The keyword arguments of the call, as given. Defaults
            are not filled in.
        call_config: The config override of the call: its explicit fields
            with their values. `DatasetConfig.CLEAR` stays as is. `INHERIT`
            if the call got the effective config of the enclosing fetch, as a
            collection passes it to its children. The call then follows the
            current config of its parent record.
        params: All parameters after resolution: the flag parameters and the
            dataset parameters, with defaults filled in. For a collection:
            its own dataset parameters (`fetch_params`), resolved, and the
            flag parameters and the dataset parameters of its children that
            the call gave. A collection resolves neither of the last two.
        version: Version identity of the flag (`Dataset.version_identity`).
        param_default_reads: The entries of `param_defaults` that parameter
            resolution used: one per parameter that the call did not give
            and that `param_defaults` names.
        config_reads: Each config field that `_fetch` or the mesqual
            post-processing read from `effective_config`, with the value read.
            Reading the whole config (`dataclasses.asdict`, `vars`, `copy`,
            `pickle`, `repr`) records all fields. `param_defaults` is here
            only if `_fetch` read the field itself.
        dependencies: The records of the fetches that this fetch called
            directly, in the order of the calls, each call key once.
        raised: `True` if the call raised.
        file_reads: The state of each file that `_fetch` recorded with
            `FetchContext.record_file_read`, one `FileRead` per path.
        reads_given_params: `True` if the fetch passed its given parameters
            (`given_params`) on to its children, as a collection does.

    The values in `call_kwargs`, `call_config`, `params`,
    `param_default_reads` and `config_reads` are deep copies, taken when the
    call recorded them. A record hashes by its `key`, so records work in
    sets. `DependencyCheck` (`mesqual.datasets.dependency_check`) decides if a
    stored record is up to date.
    """

    dataset_class: str
    dataset_name: str
    flag: str
    call_kwargs: Mapping[str, Any]
    call_config: Mapping[str, Any]
    params: Mapping[str, Any]
    version: tuple
    param_default_reads: Mapping[str, Any] = field(default_factory=dict)
    config_reads: Mapping[str, Any] = field(default_factory=dict)
    dependencies: tuple[FetchRecord, ...] = ()
    raised: bool = False
    file_reads: tuple[FileRead, ...] = ()
    reads_given_params: bool = False

    def __hash__(self) -> int:
        return hash(self.key)

    @property
    def key(self) -> tuple:
        """Identity of the call: dataset class, dataset name, flag, resolved parameters, call kwargs and call config.

        Two fetches of one flag with different keyword arguments or config
        overrides are two keys.
        """
        call_config = repr(INHERIT) if self.call_config is INHERIT else self.params_key(self.call_config)
        return (
            self.dataset_class,
            self.dataset_name,
            self.flag,
            self.params_key(self.params),
            self.params_key(self.call_kwargs),
            call_config,
        )

    @property
    def given_params(self) -> dict[str, Any]:
        """The resolved parameters that the call or `param_defaults` gave: names in `call_kwargs` or `param_default_reads`.

        A parameter that resolved to the default of its dimension is not here.
        """
        return {
            name: value
            for name, value in self.params.items()
            if name in self.call_kwargs or name in self.param_default_reads
        }

    def iter_dependencies(self) -> Iterator[FetchRecord]:
        """Walk the dependency tree depth first, parents before children.

        A record that two parents call appears once under each parent.
        """
        for dependency in self.dependencies:
            yield dependency
            yield from dependency.iter_dependencies()

    @classmethod
    def params_key(cls, params: Mapping[str, Any]) -> tuple:
        """Canonical form of parameters: pairs of name and canonical text of the value, sorted by name.

        The text is the `repr` of the value, with the items of a dict and of
        a set sorted, so the insertion order does not change it.
        """
        return tuple(sorted((name, cls.canonical(value)) for name, value in params.items()))

    @classmethod
    def canonical(cls, value: Any) -> str:
        """Canonical text of a value: its `repr`, with the items of dicts and sets sorted, also nested."""
        if isinstance(value, Mapping):
            items = sorted(f'{cls.canonical(k)}: {cls.canonical(v)}' for k, v in value.items())
            return f'{type(value).__name__}({{{", ".join(items)}}})'
        if isinstance(value, (set, frozenset)):
            return f'{type(value).__name__}({{{", ".join(sorted(map(cls.canonical, value)))}}})'
        if isinstance(value, (list, tuple)):
            return f'{type(value).__name__}([{", ".join(map(cls.canonical, value))}])'
        return repr(value)

key property

key: tuple

Identity of the call: dataset class, dataset name, flag, resolved parameters, call kwargs and call config.

Two fetches of one flag with different keyword arguments or config overrides are two keys.

given_params property

given_params: dict[str, Any]

The resolved parameters that the call or param_defaults gave: names in call_kwargs or param_default_reads.

A parameter that resolved to the default of its dimension is not here.

iter_dependencies

iter_dependencies() -> Iterator[FetchRecord]

Walk the dependency tree depth first, parents before children.

A record that two parents call appears once under each parent.

Source code in src/mesqual/datasets/fetch_context.py
295
296
297
298
299
300
301
302
def iter_dependencies(self) -> Iterator[FetchRecord]:
    """Walk the dependency tree depth first, parents before children.

    A record that two parents call appears once under each parent.
    """
    for dependency in self.dependencies:
        yield dependency
        yield from dependency.iter_dependencies()

params_key classmethod

params_key(params: Mapping[str, Any]) -> tuple

Canonical form of parameters: pairs of name and canonical text of the value, sorted by name.

The text is the repr of the value, with the items of a dict and of a set sorted, so the insertion order does not change it.

Source code in src/mesqual/datasets/fetch_context.py
304
305
306
307
308
309
310
311
@classmethod
def params_key(cls, params: Mapping[str, Any]) -> tuple:
    """Canonical form of parameters: pairs of name and canonical text of the value, sorted by name.

    The text is the `repr` of the value, with the items of a dict and of
    a set sorted, so the insertion order does not change it.
    """
    return tuple(sorted((name, cls.canonical(value)) for name, value in params.items()))

canonical classmethod

canonical(value: Any) -> str

Canonical text of a value: its repr, with the items of dicts and sets sorted, also nested.

Source code in src/mesqual/datasets/fetch_context.py
313
314
315
316
317
318
319
320
321
322
323
@classmethod
def canonical(cls, value: Any) -> str:
    """Canonical text of a value: its `repr`, with the items of dicts and sets sorted, also nested."""
    if isinstance(value, Mapping):
        items = sorted(f'{cls.canonical(k)}: {cls.canonical(v)}' for k, v in value.items())
        return f'{type(value).__name__}({{{", ".join(items)}}})'
    if isinstance(value, (set, frozenset)):
        return f'{type(value).__name__}({{{", ".join(sorted(map(cls.canonical, value)))}}})'
    if isinstance(value, (list, tuple)):
        return f'{type(value).__name__}([{", ".join(map(cls.canonical, value))}])'
    return repr(value)

FileRead dataclass

The state of a file: path, size, modification time and, for a small file, a content hash.

The cache records one FileRead for each file that a fetch read. A cache entry is up to date only if each of these files is unchanged (is_unchanged).

Attributes:

Name Type Description
path str

Absolute path of the file: the path as given, made absolute with the working directory at record time. Symbolic links and .. are kept, so a link that points to another file is changed.

size int

Size in bytes.

mtime_ns int

Modification time in nanoseconds.

sha256 str | None

Hex digest of the content, for a file of at most HASH_MAX_BYTES bytes. None for a larger file, or if read with with_hash=False.

Source code in src/mesqual/datasets/fetch_context.py
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
@dataclass(frozen=True)
class FileRead:
    """The state of a file: path, size, modification time and, for a small file, a content hash.

    The cache records one `FileRead` for each file that a fetch read. A cache entry is up to
    date only if each of these files is unchanged (`is_unchanged`).

    Attributes:
        path: Absolute path of the file: the path as given, made absolute
            with the working directory at record time. Symbolic links and
            `..` are kept, so a link that points to another file is changed.
        size: Size in bytes.
        mtime_ns: Modification time in nanoseconds.
        sha256: Hex digest of the content, for a file of at most
            `HASH_MAX_BYTES` bytes. `None` for a larger file, or if read
            with `with_hash=False`.
    """

    HASH_MAX_BYTES: ClassVar[int] = 16 * 1024**2
    """Files up to this size in bytes get a content hash. A larger file is checked by size and modification time."""

    path: str
    size: int
    mtime_ns: int
    sha256: str | None = None

    @classmethod
    def of(cls, path: str | os.PathLike, with_hash: bool = True) -> FileRead:
        """Return the current state of the file at `path`.

        Args:
            path: The path of the file.
            with_hash: `False` gives no content hash, also for a small file.

        Raises:
            ValueError: `path` is a directory.
            OSError: The file does not exist or cannot be read.
        """
        absolute = Path(path).absolute()
        if absolute.is_dir():
            raise ValueError(f'{absolute} is a directory. Give the files in it one by one.')
        stat = absolute.stat()
        digest = cls._sha256(absolute) if with_hash and stat.st_size <= cls.HASH_MAX_BYTES else None
        return cls(str(absolute), stat.st_size, stat.st_mtime_ns, digest)

    def is_unchanged(self) -> bool:
        """`True` if the file still exists with the same size and content.

        A file with a recorded content hash is unchanged if the size and
        the hash are the same; the modification time does not count. A file
        without content hash is unchanged if the size and the modification
        time are the same.
        """
        try:
            stat = os.stat(self.path)
            if stat.st_size != self.size:
                return False
            if self.sha256 is None:
                return stat.st_mtime_ns == self.mtime_ns
            return self._sha256(Path(self.path)) == self.sha256
        except OSError:
            return False

    @property
    def key(self) -> str | tuple[str, int, int]:
        """Identity of the file state without the directory.

        A file with a content hash (at most `HASH_MAX_BYTES`, read with
        `with_hash=True`) has the hash as key, without the file name. Any
        other file has the key `(file name, size, mtime_ns)`, so two copies of
        it have the same key only if they have the same modification time.
        A fingerprint pairs each key with the name of the file relative to
        the source folder, so a rename or a swap of two files changes it.
        """
        return self.sha256 or (Path(self.path).name, self.size, self.mtime_ns)

    @staticmethod
    def _sha256(path: Path) -> str:
        digest = hashlib.sha256()
        with open(path, 'rb') as file:
            for chunk in iter(lambda: file.read(1024**2), b''):
                digest.update(chunk)
        return digest.hexdigest()

HASH_MAX_BYTES class-attribute

HASH_MAX_BYTES: int = 16 * 1024 ** 2

Files up to this size in bytes get a content hash. A larger file is checked by size and modification time.

key property

key: str | tuple[str, int, int]

Identity of the file state without the directory.

A file with a content hash (at most HASH_MAX_BYTES, read with with_hash=True) has the hash as key, without the file name. Any other file has the key (file name, size, mtime_ns), so two copies of it have the same key only if they have the same modification time. A fingerprint pairs each key with the name of the file relative to the source folder, so a rename or a swap of two files changes it.

of classmethod

of(path: str | PathLike, with_hash: bool = True) -> FileRead

Return the current state of the file at path.

Parameters:

Name Type Description Default
path str | PathLike

The path of the file.

required
with_hash bool

False gives no content hash, also for a small file.

True

Raises:

Type Description
ValueError

path is a directory.

OSError

The file does not exist or cannot be read.

Source code in src/mesqual/datasets/fetch_context.py
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
@classmethod
def of(cls, path: str | os.PathLike, with_hash: bool = True) -> FileRead:
    """Return the current state of the file at `path`.

    Args:
        path: The path of the file.
        with_hash: `False` gives no content hash, also for a small file.

    Raises:
        ValueError: `path` is a directory.
        OSError: The file does not exist or cannot be read.
    """
    absolute = Path(path).absolute()
    if absolute.is_dir():
        raise ValueError(f'{absolute} is a directory. Give the files in it one by one.')
    stat = absolute.stat()
    digest = cls._sha256(absolute) if with_hash and stat.st_size <= cls.HASH_MAX_BYTES else None
    return cls(str(absolute), stat.st_size, stat.st_mtime_ns, digest)

is_unchanged

is_unchanged() -> bool

True if the file still exists with the same size and content.

A file with a recorded content hash is unchanged if the size and the hash are the same; the modification time does not count. A file without content hash is unchanged if the size and the modification time are the same.

Source code in src/mesqual/datasets/fetch_context.py
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
def is_unchanged(self) -> bool:
    """`True` if the file still exists with the same size and content.

    A file with a recorded content hash is unchanged if the size and
    the hash are the same; the modification time does not count. A file
    without content hash is unchanged if the size and the modification
    time are the same.
    """
    try:
        stat = os.stat(self.path)
        if stat.st_size != self.size:
            return False
        if self.sha256 is None:
            return stat.st_mtime_ns == self.mtime_ns
        return self._sha256(Path(self.path)) == self.sha256
    except OSError:
        return False

Unset

Type of the UNSET marker, for the type hints of typed fetch overrides. There is one instance.

Source code in src/mesqual/datasets/fetch_context.py
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
class Unset:
    """Type of the `UNSET` marker, for the type hints of typed `fetch` overrides. There is one instance."""

    _instance = None

    def __new__(cls):
        if cls._instance is None:
            cls._instance = super().__new__(cls)
        return cls._instance

    def __reduce__(self):
        return Unset, ()

    def __repr__(self) -> str:
        return '<from param_defaults or default>'

config_field

config_field(default: Any = MISSING, description: str = '', merge_per_key: bool = False, **kwargs: Any) -> Any

Declare a config field with a description.

The description goes into the field metadata. describe_config() reads it from there.

Parameters:

Name Type Description Default
default Any

Default value of the field.

MISSING
description str

One-line description of the field.

''
merge_per_key bool

For a dict field: a layer adds its keys to the dict of the lower layers instead of replacing it.

False
**kwargs Any

Further arguments for dataclasses.field, for example default_factory.

{}

Returns:

Type Description
Any

A dataclasses.field with the description in its metadata.

Examples:

>>> from dataclasses import dataclass
>>> @dataclass
... class MyPlatformConfig(DatasetConfig):
...     convert_timestamps: bool = config_field(True, 'Convert period numbers to timestamps.')
Source code in src/mesqual/datasets/dataset_config.py
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
def config_field(
    default: Any = dataclasses.MISSING, description: str = '', merge_per_key: bool = False, **kwargs: Any
) -> Any:
    """Declare a config field with a description.

    The description goes into the field metadata. `describe_config()` reads
    it from there.

    Args:
        default: Default value of the field.
        description: One-line description of the field.
        merge_per_key: For a dict field: a layer adds its keys to the dict of
            the lower layers instead of replacing it.
        **kwargs: Further arguments for `dataclasses.field`, for example
            `default_factory`.

    Returns:
        A `dataclasses.field` with the description in its metadata.

    Examples:
        >>> from dataclasses import dataclass
        >>> @dataclass
        ... class MyPlatformConfig(DatasetConfig):
        ...     convert_timestamps: bool = config_field(True, 'Convert period numbers to timestamps.')
    """
    metadata = {**kwargs.pop('metadata', {}), 'description': description, 'merge_per_key': merge_per_key}
    return dataclasses.field(default=default, metadata=metadata, **kwargs)