Dataset References¶
Describe a dataset¶
The describe methods show what a dataset offers. They load no time series data.
Describe methods work on an instance. A method with class_ in its name works on the class.
describe_class_flags(): the flag specs that a dataset, interpreter or platform dataset class declares.describe_flags(): the flags that a dataset can deliver, with resolved values, parameter defaults and availability.flag_info(flag): all details of one flag, as aFlagInfo.describe_class_area_layers(): the area layers registered on a dataset class.describe_class_border_layers(): the border layers registered on a dataset class.describe_datasets(max_depth=None): the tree of a dataset and its children, down tomax_depthlevels.describe_config(): the config fields of a dataset, with their effective values.StudyManager.describe_study(): the scenarios and comparisons of a study, and the flags of each.
Use describe_class_flags to see what a platform offers before you build a dataset. Use describe_flags to see what you can fetch from a dataset you have.
Both give one row per spec, with the same index. The columns of describe_class_flags are the first columns of describe_flags, in the same order. describe_flags only adds columns.
A layer spec on area or border layers (AreaFlagSpec, BorderFlagSpec) has one row. layers names its layers. object names the model flags of these layers, in the same order, for example Country.Model.
describe_class_flags() |
describe_flags() |
|
|---|---|---|
| Question | Which flags does this kind of dataset declare? | Which flags can this dataset deliver, and with which settings? |
| Called on | a dataset, interpreter or platform dataset class. A collection class raises TypeError. |
an instance |
| Reads | spec declarations only | model flags and the source inventory; never time series |
| Flag families | template, for example Node.Results.generation_{fuel}, with the declared values or resolved per dataset |
resolved from the dataset, for example one entry per fuel |
| Layers | the layers registered on the class | the registered layers of the dataset |
| Parameter defaults | spec default | effective default and its source: spec, dataset, config, required or per dataset |
| Availability | none | available, unknown, partial or unavailable, and the number of flags |
| Collections | not supported: a collection has no flags of its own | union of the children, plus which children have each flag |
All describe methods except flag_info and describe_config take as_records=True and then return plain dicts, ready for JSON.
datasets
¶
Datasets, collections, comparisons, platform datasets and their configs.
INHERIT
module-attribute
¶
INHERIT = _Inherit()
FetchRecord.call_config of a call that got the effective config of the enclosing fetch as its config.
UNSET
module-attribute
¶
UNSET = Unset()
Default of the typed dataset parameters of a fetch override: the call does not give the parameter.
The parameter then resolves from param_defaults of the effective config,
else from the default of its dimension (Dataset.fetch_params).
Dataset
¶
Bases: Generic[DatasetConfigType, FlagType, FlagIndexType], ABC
Base class of all datasets: one fetch(flag) interface for all data.
A scenario, a scenario merged from several runs or sources, a collection of scenarios and a scenario comparison are all datasets.
A dataset declares its flags with flag specs in flag_specs and
implements _fetch(flag, effective_config, **params). accepted_flags
comes from the specs. A dataset can have attributes (scenario metadata), a
KPI collection and a cache of fetch results.
Attributes:
| Name | Type | Description |
|---|---|---|
name |
str
|
Name of the dataset. |
kpi_collection |
KPICollection
|
The KPIs of the dataset. |
Examples:
A dataset class:
class NodeResults(Dataset):
flag_specs = (NODE_MODEL_SPEC, NODE_PRICE_SPEC)
def _fetch(self, flag, effective_config, **params):
...
A fetch:
>>> prices = dataset.fetch('Node.Results.price')
Source code in src/mesqual/datasets/dataset.py
136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061 1062 1063 1064 1065 1066 1067 1068 1069 1070 1071 1072 1073 1074 1075 1076 1077 1078 1079 1080 1081 1082 1083 1084 1085 1086 1087 1088 1089 1090 1091 1092 1093 1094 1095 1096 1097 1098 1099 1100 1101 1102 1103 1104 1105 1106 1107 1108 1109 1110 1111 1112 1113 1114 1115 1116 1117 1118 1119 1120 1121 1122 1123 1124 1125 1126 1127 1128 1129 1130 1131 1132 1133 1134 1135 1136 1137 1138 1139 1140 1141 1142 1143 1144 1145 1146 1147 1148 1149 1150 1151 1152 1153 1154 1155 1156 1157 1158 1159 1160 1161 1162 1163 1164 1165 1166 1167 1168 1169 1170 1171 1172 1173 1174 1175 1176 1177 1178 1179 1180 1181 1182 1183 1184 1185 1186 1187 1188 1189 1190 1191 1192 1193 1194 1195 1196 1197 1198 1199 1200 1201 1202 1203 1204 1205 1206 1207 1208 1209 1210 1211 1212 1213 1214 1215 1216 1217 1218 1219 1220 1221 1222 1223 1224 1225 1226 1227 1228 1229 1230 1231 1232 1233 1234 1235 1236 1237 1238 1239 1240 1241 1242 1243 1244 1245 1246 1247 1248 1249 1250 1251 1252 1253 1254 1255 1256 1257 1258 1259 1260 1261 1262 1263 1264 1265 1266 1267 1268 1269 1270 1271 1272 1273 1274 1275 1276 1277 1278 1279 1280 1281 1282 1283 1284 1285 1286 1287 1288 1289 1290 1291 1292 1293 1294 1295 1296 1297 1298 1299 1300 1301 1302 1303 1304 1305 1306 1307 1308 1309 1310 1311 1312 1313 1314 1315 1316 1317 1318 1319 1320 1321 1322 1323 1324 1325 1326 1327 1328 1329 1330 1331 1332 1333 1334 1335 1336 1337 1338 1339 1340 1341 1342 1343 1344 1345 1346 1347 1348 1349 1350 1351 1352 1353 1354 1355 1356 1357 1358 1359 1360 1361 1362 1363 1364 1365 1366 1367 1368 1369 1370 1371 1372 1373 1374 1375 1376 1377 1378 1379 1380 1381 1382 1383 1384 1385 1386 1387 1388 1389 1390 1391 1392 1393 1394 1395 1396 1397 1398 1399 1400 1401 1402 1403 1404 1405 1406 1407 1408 1409 1410 1411 1412 1413 1414 1415 1416 1417 1418 1419 1420 1421 1422 1423 1424 1425 1426 1427 1428 1429 1430 1431 1432 1433 1434 1435 1436 1437 1438 1439 1440 1441 1442 1443 1444 1445 1446 1447 1448 1449 1450 1451 1452 1453 1454 1455 1456 1457 1458 1459 1460 1461 1462 1463 1464 1465 1466 1467 1468 1469 1470 1471 1472 1473 1474 1475 1476 1477 1478 1479 1480 1481 1482 1483 1484 1485 1486 1487 1488 1489 1490 1491 1492 1493 1494 1495 1496 1497 1498 1499 1500 1501 1502 1503 1504 1505 1506 1507 1508 1509 1510 1511 1512 1513 1514 1515 1516 1517 1518 1519 1520 1521 1522 1523 1524 1525 1526 1527 1528 1529 1530 1531 1532 1533 1534 1535 1536 1537 1538 1539 1540 1541 1542 1543 1544 1545 1546 1547 1548 1549 1550 1551 1552 1553 1554 1555 1556 1557 1558 1559 1560 1561 1562 1563 1564 1565 1566 1567 1568 1569 1570 1571 1572 1573 1574 1575 1576 1577 1578 1579 1580 1581 1582 1583 1584 1585 1586 1587 1588 1589 1590 1591 1592 1593 1594 1595 1596 1597 1598 1599 1600 1601 1602 1603 1604 1605 1606 1607 1608 1609 1610 1611 1612 1613 1614 1615 1616 1617 1618 1619 1620 1621 1622 1623 1624 1625 1626 1627 1628 1629 1630 1631 1632 1633 1634 1635 1636 1637 1638 1639 1640 1641 1642 1643 1644 1645 1646 1647 1648 1649 1650 1651 1652 1653 1654 1655 1656 1657 1658 1659 1660 1661 1662 1663 1664 1665 1666 1667 1668 1669 1670 1671 1672 1673 1674 1675 1676 1677 1678 1679 1680 1681 1682 1683 1684 1685 1686 1687 1688 1689 1690 1691 1692 1693 1694 1695 1696 1697 1698 1699 1700 1701 1702 1703 1704 1705 1706 1707 1708 1709 1710 1711 1712 1713 1714 1715 1716 1717 1718 1719 1720 1721 1722 1723 1724 1725 1726 1727 1728 1729 1730 1731 1732 1733 1734 1735 1736 1737 1738 1739 1740 1741 1742 1743 1744 1745 1746 1747 1748 1749 1750 1751 1752 1753 1754 1755 1756 1757 1758 1759 1760 1761 1762 1763 1764 1765 1766 1767 1768 1769 1770 1771 1772 1773 1774 1775 1776 1777 1778 1779 1780 1781 1782 1783 1784 1785 1786 1787 1788 1789 1790 1791 1792 1793 1794 1795 1796 1797 1798 1799 1800 1801 1802 1803 1804 1805 1806 1807 1808 1809 1810 1811 1812 1813 1814 1815 1816 1817 1818 1819 1820 1821 1822 1823 1824 1825 1826 1827 1828 1829 1830 1831 1832 1833 1834 1835 1836 1837 1838 1839 1840 1841 1842 1843 1844 1845 1846 1847 1848 1849 1850 1851 1852 1853 1854 1855 1856 1857 1858 1859 1860 1861 1862 1863 1864 1865 1866 1867 1868 1869 1870 1871 1872 1873 1874 1875 1876 1877 1878 1879 1880 1881 1882 1883 1884 1885 1886 1887 1888 1889 1890 1891 1892 1893 1894 1895 1896 1897 1898 1899 1900 1901 1902 1903 1904 1905 1906 1907 1908 1909 1910 1911 1912 1913 1914 1915 1916 1917 1918 1919 1920 1921 1922 1923 1924 1925 1926 1927 1928 1929 1930 1931 1932 1933 1934 1935 1936 1937 1938 1939 1940 1941 1942 1943 1944 1945 1946 1947 1948 1949 1950 1951 1952 1953 1954 1955 1956 1957 1958 1959 1960 1961 1962 1963 1964 1965 1966 1967 1968 1969 1970 1971 1972 1973 1974 1975 1976 1977 1978 1979 1980 1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 2023 2024 2025 2026 2027 2028 2029 2030 2031 2032 2033 2034 2035 2036 2037 2038 2039 2040 2041 2042 2043 2044 2045 2046 2047 2048 2049 2050 2051 | |
flag_specs
class-attribute
¶
flag_specs: tuple[FlagSpec, ...] = ()
Flag specs of this dataset. accepted_flags and the flag index come from them.
description
class-attribute
instance-attribute
¶
description: str | None = None
Optional free text about the dataset. A platform class or a study sets it, on the class or the instance.
describe_datasets() shows it.
fetch_params
class-attribute
¶
fetch_params: tuple[Dimension, ...] = ()
Dataset parameters: keyword arguments of fetch that the dataset class uses itself.
A class has the dataset parameters of its base classes and adds its own:
fetch_params of a subclass lists only the new ones, and the class
gets the merged tuple.
A flag spec declares the parameters of one flag (FlagSpec.params).
fetch_params declares the parameters of every fetch of this dataset
class, for example comparison_type of DatasetComparison. Both kinds
resolve the same way: the keyword argument, then param_defaults of the
effective config, then the default of the dimension. The resolved value
goes to _fetch as a keyword argument.
A name may not be both a flag parameter and a dataset parameter of one
dataset. A collection also accepts the dataset parameters of its
children. It passes each child the flag parameters of the call and the
dataset parameters that the child accepts and that the call or the
param_defaults of the collection gave, its own ones included (for
example comparison_type to the comparisons of a comparison of
comparisons). A value in the call and the same value in param_defaults
reach the child alike. The collection never passes the default of a
dimension.
Examples:
class NodePrices(Dataset):
flag_specs = (NODE_PRICE_SPEC,)
fetch_params = (Dimension('currency', 'Currency of the prices.', {
'EUR': 'euro', 'USD': 'US dollar',
}, default='EUR'),)
def _fetch(self, flag, effective_config, currency, **params):
...
NodePrices().fetch('Node.Results.price', currency='USD')
A subclass may override fetch to show its dataset parameters in the
IDE: one typed keyword-only argument per dataset parameter, with the
default UNSET. The override passes on to super().fetch only the
arguments that are not UNSET (_given). So a parameter that the call
leaves out still resolves from param_defaults, and call_kwargs holds
only what the caller gave.
accepted_flags
property
¶
accepted_flags: set[FlagType]
Set of the flags of the specs of this dataset.
It holds concrete flag names, computed from the declarations in
flag_specs without opening source data. Flags of a flag family are
not listed, because their values come from the data of the dataset.
flag_is_accepted accepts them by their name template. Whether a
source has the data of a flag is a separate question (availability).
A subclass may narrow the spec flags using declarations of siblings or attributes, never source data.
Returns:
| Type | Description |
|---|---|
set[FlagType]
|
Set of flags that can be used with the fetch() method |
Examples:
>>> 'Line.Results.flow_sent_net' in dataset.accepted_flags
True
flag_index
property
¶
flag_index: FlagIndex
The flag index: lookup from a flag to its spec.
Built from flag_specs at construction. A dataset constructed with a
parent dataset, for example an interpreter, uses the index of its
parent. Collections and platform datasets document their own rule.
cache
property
¶
cache: Cache | None
The cache of fetch results of this dataset, or None.
fetch uses it when use_cache is set (see
Cache).
attributes
property
¶
attributes: dict
Access the metadata attributes dictionary for this dataset.
Attributes store scenario-level metadata such as configuration parameters, simulation settings, or descriptive labels. These are useful for filtering, grouping, and annotating datasets in collections.
Returns:
| Name | Type | Description |
|---|---|---|
dict |
dict
|
Dictionary of attribute key-value pairs. Keys are strings, values are primitive types (bool, int, float, str). |
Examples:
>>> dataset.attributes
{'first_date': '2024-03-28', 'last_date': '2024-04-01', 'days': 5, 'demand_factor': 1.0, 'periods': 2}
See Also
set_attributes()- Set attribute valuesget_attributes_series()- Convert to pandas Series
parent_dataset
property
writable
¶
parent_dataset: 'DatasetLinkCollection'
The parent dataset of an interpreter: the platform dataset that holds it and its sibling interpreters.
An interpreter answers a part of the flags of a platform dataset. It
fetches the flags of its siblings through the parent, so it needs no
reference to them. A fetch through the parent reaches an override of the
sibling interpreter. The parent is a DatasetLinkCollection, usually a
PlatformDataset.
Returns:
| Name | Type | Description |
|---|---|---|
DatasetLinkCollection |
'DatasetLinkCollection'
|
The parent collection. |
Raises:
| Type | Description |
|---|---|
RuntimeError
|
The parent is not set, for example for an interpreter used without a parent dataset. |
Examples:
An interpreter that sums the generation per node from two flags of its siblings:
class NodeGeneration(MyInterpreter):
flag_specs = (NODE_GENERATION_SPEC,)
def _fetch(self, flag, effective_config, **params):
generation = self.parent_dataset.fetch('Generator.Results.generation', effective_config)
node_of_generator = self.parent_dataset.fetch('Generator.Model', effective_config)['node']
return generation.T.groupby(node_of_generator).sum().T
See Also
DatasetLinkCollection.get_dataset_by_type(): a sibling interpreter by its class.
instance_config
property
¶
instance_config: DatasetConfigType
Get the effective configuration for this dataset instance.
Computes the merged configuration by combining:
- Base config defaults
- Class-level config (set via set_class_config)
- Instance-level config (passed to init or set via set_instance_config)
Later settings override earlier ones. This is the configuration used by fetch() unless overridden by a fetch-time config parameter.
Returns:
| Name | Type | Description |
|---|---|---|
DatasetConfigType |
DatasetConfigType
|
The fully resolved configuration for this instance |
Examples:
>>> dataset.instance_config.use_cache
True
See Also
set_instance_config()- Replace instance configurationset_class_config()- Set class-level defaultsDatasetConfigManager- Configuration management system
__init__
¶
__init__(name: str = None, parent_dataset: Dataset = None, attributes: dict = None, cache: Cache = None, config: DatasetConfigType = None)
Initialize a new Dataset instance.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
name
|
str
|
Human-readable identifier. If None, auto-generates from class name |
None
|
parent_dataset
|
Dataset
|
Optional parent dataset for hierarchical relationships |
None
|
attributes
|
dict
|
Dictionary of metadata attributes for the dataset. Keys must be of type str, values of type bool, int, float or str. |
None
|
cache
|
Cache
|
Optional cache of fetch results |
None
|
config
|
DatasetConfigType
|
Configuration object controlling dataset behavior |
None
|
Raises:
| Type | Description |
|---|---|
TypeError
|
If an attribute key or value has an invalid type. |
ValueError
|
A time series spec names a model flag without model spec. |
AmbiguousOverrideError
|
Two specs give the same flag name. |
ValueError
|
A name is a flag parameter and a dataset parameter
( |
Source code in src/mesqual/datasets/dataset.py
259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 | |
fetch
¶
fetch(flag: FlagType, config: dict | DatasetConfigType = None, *, granularity: str | Timedelta | None | Unset = UNSET, **kwargs: Any) -> Series | DataFrame
Fetch the data of one flag.
fetch takes two kinds of input:
config: overrides of config fields for this call.**kwargs: parameters. Every keyword argument is a parameter. A flag spec declares flag parameters (FlagSpec.params), for examplesolution. The dataset class declares dataset parameters (fetch_params), for examplecomparison_typeof a comparison.
Each parameter resolves the same way, first match wins:
- The keyword argument of this call.
param_defaultsof the effective config.- The
defaultof the dimension.
_fetch gets the resolved parameters as keyword arguments. A
collection resolves only its own dataset parameters. It passes the
flag parameters of the call to its children. It gives each child
the dataset parameters that the child accepts and that the call or
the param_defaults of the collection gave, its own ones included,
never the default of a dimension (_kwargs_for_child).
Config resolution, later wins: config class defaults, the class
config of each class in the MRO (base first), the instance config,
and the config of this call. Only the explicit fields of a layer
override lower layers (the keys of a dict, or the explicit fields of
a config object), also with a None value. DatasetConfig.CLEAR
resets a field to its default.
Steps:
- Raise
UnknownFlagErrorif no flag spec declares the flag, andValueErrorif this dataset does not accept it or if a dataset that reads the data itself (not a collection) has the flag unavailable (_is_availableisFalse). Collections fetch only from children on which the flag is not unavailable. - Resolve the effective config.
- Resolve the parameters.
- Open a frame in the fetch context. Nested fetches and the
config fields that
_fetchreads are recorded in it (get_fetch_record). - Call
_fetchwith the resolved parameters, or read the cache, and post-process. - In the outermost call, apply the time index filter to a time
series flag, then convert it to the
granularityof the effective config.
Settings applied after the cache (time_index_filter,
granularity): they apply only in the outermost fetch call, from
its effective config, and only to time series flags, first the
filter, then the conversion. _fetch gets them as None, so a
cached value is never filtered or converted. Nested fetches
(collection children, interpreters, layer flags) return unfiltered
native data: such a setting in the config of a child dataset has no
effect when the child is fetched through a collection. The fetch
depth lives in a context variable: run code in a thread started
inside a fetch with contextvars.copy_context().run(...), else its
fetches count as outermost.
The config field granularity converts a time series flag to
another time resolution with TimeSeriesGranularityConverter and
the quantity type of the flag: intensive values are averaged or
repeated, extensive values summed or split. None means the native
resolution. On a row MultiIndex it converts the datetime level per
combination of the other levels. Model and other flags are returned
unchanged. The argument granularity is a short form of
config={'granularity': ...} in the outermost call. In a nested
fetch it has no effect, so the call keeps its config as given.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
flag
|
FlagType
|
The flag. It must be in |
required |
config
|
dict | DatasetConfigType
|
Config overrides for this call: a dict with config field names as keys, or a config object of the config type of the dataset. A platform config may add its own fields. |
None
|
granularity
|
str | Timedelta | None | Unset
|
Target time resolution of a time series flag, a
pandas frequency such as |
UNSET
|
**kwargs
|
Any
|
Parameters: flag parameters of the spec and dataset parameters of the dataset class (for a collection, also of its children). |
{}
|
Returns:
| Type | Description |
|---|---|
Series | DataFrame
|
DataFrame or Series with the data of the flag. |
Raises:
| Type | Description |
|---|---|
UnknownFlagError
|
No flag spec declares the flag. |
FlagUnavailableError
|
The flag is not available
( |
ValueError
|
The dataset does not accept the flag; a keyword
argument is no declared parameter; a parameter is missing or
has a value that its dimension does not allow;
|
TypeError
|
A |
Examples:
A time series in another resolution. The base scenario has hourly rows, then quarter-hourly rows. The result has one row per hour:
>>> hourly = dataset.fetch('Node.Results.price', granularity='1h')
>>> len(dataset.fetch('Node.Results.price')), len(hourly)
(332, 119)
Source code in src/mesqual/datasets/dataset.py
301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 | |
get_fetch_record
¶
get_fetch_record(flag: FlagType, config: dict | DatasetConfigType = None, *, granularity: str | Timedelta | None | Unset = UNSET, **kwargs: Any) -> FetchRecord | None
Return the record of the last fetch with the same flag, keyword arguments, config and resolved parameters.
A fetch that raised also leaves a record, with raised=True.
The record lists the fetches that the call made directly, each with
its own record, and the config fields that _fetch read. The lookup
uses the record key (FetchRecord.key): the keyword arguments and
the call config as given, and the parameters, resolved as in fetch.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
flag
|
FlagType
|
The flag. |
required |
config
|
dict | DatasetConfigType
|
Fetch-time config, as in |
None
|
granularity
|
str | Timedelta | None | Unset
|
Applies as in an outermost fetch. To look up a
nested call, leave it out. No effect with |
UNSET
|
**kwargs
|
Any
|
Parameters, as in |
{}
|
Returns:
| Type | Description |
|---|---|
FetchRecord | None
|
The record, or |
Examples:
>>> high = study.scen.get_dataset('high_demand')
>>> prices = high.fetch('Node.Results.price', solution='first')
>>> high.get_fetch_record('Node.Results.price', solution='first').call_kwargs
{'solution': 'first'}
>>> high.get_fetch_record('Node.Results.price') is None
True
Source code in src/mesqual/datasets/dataset.py
460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 | |
version_identity
¶
version_identity(flag: FlagType) -> tuple
Return the version identity of a flag: one entry per link.
A dataset gives ((type(self).__qualname__, spec version),). A
platform dataset gives one pair per link of the interpreter chain of
the flag. The answering interpreter of a platform dataset gives the
identity of its platform dataset, so a version change of an override
changes the records of the override and of everything that calls it.
A collection gives one (class path, name) pair per child that it can
read the flag from; its children record their own identities. A concat collection
adds its layout first (DatasetConcatCollection.version_identity).
Source code in src/mesqual/datasets/dataset.py
538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 | |
get_accepted_flags_containing_x
¶
get_accepted_flags_containing_x(x: str, match_case: bool = False) -> set[FlagType]
Find all accepted flags containing a specific substring.
Useful for discovering related data flags or filtering flags by category.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
x
|
str
|
Substring to search for in flag names |
required |
match_case
|
bool
|
If True, performs case-sensitive search. Default is False. |
False
|
Returns:
| Type | Description |
|---|---|
set[FlagType]
|
Set of accepted flags containing the substring |
Examples:
>>> sorted(dataset.get_accepted_flags_containing_x('line.results.flow_sent'))
['Line.Results.flow_sent_down', 'Line.Results.flow_sent_net', 'Line.Results.flow_sent_up']
>>> dataset.get_accepted_flags_containing_x('line.results', match_case=True)
set()
Source code in src/mesqual/datasets/dataset.py
968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 | |
flag_is_accepted
¶
flag_is_accepted(flag: FlagType) -> bool
Boolean check whether a flag is accepted by the Dataset.
True for a flag in accepted_flags and for a flag of a flag family
spec of this dataset. Uses only declarations, never source data.
Source code in src/mesqual/datasets/dataset.py
989 990 991 992 993 994 995 996 997 998 | |
add_kpis_from_definitions
¶
add_kpis_from_definitions(kpi_definitions: KPIDefinition | list[KPIDefinition])
Generate and add KPIs from one or more KPI definitions.
KPI definitions are templates that generate concrete KPI instances based on the dataset's structure. This method processes definitions and adds the resulting KPIs to the dataset's KPI collection.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
kpi_definitions
|
KPIDefinition | list[KPIDefinition]
|
Single KPIDefinition or list of definitions. Each definition's generate_kpis() method is called with this dataset to produce KPI instances. |
required |
Examples:
>>> from mesqual.kpis import Aggregations, FlagAggKPIBuilder
>>> definitions = FlagAggKPIBuilder().for_flag('Country.Results.price').with_aggregation(Aggregations.Mean).build()
>>> dataset.add_kpis_from_definitions(definitions)
>>> len(dataset.kpi_collection)
2
See Also
add_kpi(): add one KPI.add_kpis(): add several KPIs.
Source code in src/mesqual/datasets/dataset.py
1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061 1062 1063 1064 | |
add_kpis
¶
add_kpis(kpis: Iterable[KPI])
Add multiple KPIs to this dataset's KPI collection.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
kpis
|
Iterable[KPI]
|
Iterable of KPI instances, factories, or classes to add |
required |
Source code in src/mesqual/datasets/dataset.py
1066 1067 1068 1069 1070 1071 1072 1073 1074 1075 1076 1077 1078 1079 1080 1081 1082 1083 | |
add_kpi
¶
add_kpi(kpi: KPI)
Add a single KPI to this dataset's KPI collection.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
kpi
|
KPI
|
KPI instance, factory, or class to add |
required |
Source code in src/mesqual/datasets/dataset.py
1085 1086 1087 1088 1089 1090 1091 1092 | |
clear_kpi_collection
¶
clear_kpi_collection()
Clear the KPI collection.
Source code in src/mesqual/datasets/dataset.py
1094 1095 1096 1097 1098 | |
get_attributes_series
¶
get_attributes_series() -> Series
Convert dataset attributes to a pandas Series.
Creates a Series with attribute names as the index and attribute values as data. The Series name is set to the dataset name, making it suitable for concatenation with other datasets' attribute series.
Returns:
| Type | Description |
|---|---|
Series
|
pd.Series: Series containing attribute values, indexed by attribute names, with the dataset name as the Series name |
Examples:
>>> series = dataset.get_attributes_series()
>>> series.name, series['days']
('base', 5)
See Also
attributes- Access raw attributes dictionaryset_attributes()- Set attribute values
Source code in src/mesqual/datasets/dataset.py
1123 1124 1125 1126 1127 1128 1129 1130 1131 1132 1133 1134 1135 1136 1137 1138 1139 1140 1141 1142 1143 1144 1145 | |
set_attributes
¶
set_attributes(**kwargs: bool | int | float | str) -> None
Set one or more metadata attributes on this dataset.
Attributes are key-value pairs that store scenario metadata. They must use string keys and primitive values (bool, int, float, str) to ensure serializability and consistent comparison behavior.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
**kwargs
|
bool | int | float | str
|
Attribute key-value pairs to set. Keys must be strings, values must be bool, int, float, or str. |
{}
|
Raises:
| Type | Description |
|---|---|
TypeError
|
If any key is not a string |
TypeError
|
If any value is not bool, int, float, or str |
Examples:
>>> dataset.set_attributes(year=2030, includes_nuclear=True)
>>> dataset.attributes['year']
2030
See Also
attributes- Access attributes dictionaryget_attributes_series()- Convert to pandas Series
Source code in src/mesqual/datasets/dataset.py
1147 1148 1149 1150 1151 1152 1153 1154 1155 1156 1157 1158 1159 1160 1161 1162 1163 1164 1165 1166 1167 1168 1169 1170 1171 1172 | |
fetch_multiple_flags_and_concat
¶
fetch_multiple_flags_and_concat(flags: Iterable[FlagType], flag_concat_axis: int = 1, concat_level_name: str = 'variable', concat_level_at_top: bool = True, config: dict | DatasetConfigType = None, *, granularity: str | Timedelta | None | Unset = UNSET, **kwargs: Any) -> Series | DataFrame
Fetch multiple flags and concatenate results into a single DataFrame.
Convenience method for retrieving data from multiple flags and combining them into a single DataFrame with a MultiIndex. Useful for comparative analysis of multiple variables or creating wide-format data structures.
All results must have the same shape of axes. Before the concat, each
result is checked against the first one: both are a Series or both
are a DataFrame, and each axis has the same number of levels and the
same index type per level (for example DatetimeIndex). A Series
has only its index to check.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
flags
|
Iterable[FlagType]
|
Iterable of flags to fetch and concatenate |
required |
flag_concat_axis
|
int
|
Axis along which to concatenate (0=rows,
1=columns). Default is 1 (columns). It is not the
|
1
|
concat_level_name
|
str
|
Name for the new MultiIndex level identifying the source flag. Default is 'variable'. |
'variable'
|
concat_level_at_top
|
bool
|
If True, the flag level is the outermost level in the MultiIndex. If False, it's moved to the innermost level. Default is True. |
True
|
config
|
dict | DatasetConfigType
|
Optional configuration override (see fetch() for details) |
None
|
granularity
|
str | Timedelta | None | Unset
|
Target time resolution, as in |
UNSET
|
**kwargs
|
Any
|
Parameters. Each fetch gets the flag parameters that its spec declares, and the dataset parameters of this dataset. |
{}
|
Returns:
| Type | Description |
|---|---|
Series | DataFrame
|
DataFrame with concatenated data and a MultiIndex identifying the source flag for each section |
Raises:
| Type | Description |
|---|---|
ValueError
|
If two results differ in kind, number of levels or index type on an axis. The message names both flags. |
ValueError
|
A keyword argument is no flag parameter of any of the flags and no dataset parameter of this dataset. |
Examples:
>>> demand_and_gas = dataset.fetch_multiple_flags_and_concat(
... ['Node.Results.demand_total', 'Node.Results.generation_gas'], concat_level_name='variable',
... )
>>> list(demand_and_gas.columns.names)
['variable', 'Node']
See Also
fetch()- Single flag data retrieval
Source code in src/mesqual/datasets/dataset.py
1356 1357 1358 1359 1360 1361 1362 1363 1364 1365 1366 1367 1368 1369 1370 1371 1372 1373 1374 1375 1376 1377 1378 1379 1380 1381 1382 1383 1384 1385 1386 1387 1388 1389 1390 1391 1392 1393 1394 1395 1396 1397 1398 1399 1400 1401 1402 1403 1404 1405 1406 1407 1408 1409 1410 1411 1412 1413 1414 1415 1416 1417 1418 1419 1420 1421 1422 1423 1424 1425 1426 1427 1428 1429 1430 | |
get_flag_type
classmethod
¶
get_flag_type() -> Type[FlagType]
Get the flag type class for this dataset type.
Returns the type used for data flags in this dataset class. Subclasses can override to specify a custom flag type for type checking and validation.
Returns:
| Type | Description |
|---|---|
Type[FlagType]
|
Type[FlagType]: The flag type class (default: FlagTypeProtocol) |
Note
Override in subclasses that use custom flag types.
Source code in src/mesqual/datasets/dataset.py
1470 1471 1472 1473 1474 1475 1476 1477 1478 1479 1480 1481 1482 1483 1484 1485 1486 1487 | |
get_config_type
classmethod
¶
get_config_type() -> Type[DatasetConfigType]
Get the configuration type class for this dataset type.
Returns the DatasetConfig subclass used by this dataset. Platform interfaces typically override this to return their extended config class with platform-specific options.
Returns:
| Type | Description |
|---|---|
Type[DatasetConfigType]
|
Type[DatasetConfigType]: The config type class (default: DatasetConfig) |
Examples:
>>> from mesqual.testing import DummyDataset
>>> DummyDataset.get_config_type()
<class 'mesqual.testing.dataset.DummyConfig'>
Note
Override in platform dataset subclasses to return platform-specific config types with additional options.
Source code in src/mesqual/datasets/dataset.py
1489 1490 1491 1492 1493 1494 1495 1496 1497 1498 1499 1500 1501 1502 1503 1504 1505 1506 1507 1508 1509 1510 1511 1512 | |
describe_config
¶
describe_config() -> DataFrame
Describe the config fields of this dataset, one row per field.
Returns:
| Type | Description |
|---|---|
DataFrame
|
DataFrame indexed by field name with the columns |
Examples:
>>> dataset.describe_config().loc['use_cache', ['value', 'layer']].tolist()
[True, 'default']
Source code in src/mesqual/datasets/dataset.py
1544 1545 1546 1547 1548 1549 1550 1551 1552 1553 1554 1555 1556 1557 1558 1559 1560 1561 1562 1563 1564 1565 1566 1567 1568 1569 1570 1571 1572 1573 1574 | |
set_instance_config
¶
set_instance_config(config: DatasetConfigType) -> None
Replace the instance-level configuration for this dataset.
Sets the configuration that will be merged with class-level defaults to produce the effective configuration used by fetch().
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
config
|
DatasetConfigType
|
New configuration object to use for this instance |
required |
Examples:
>>> from mesqual.datasets import DatasetConfig
>>> dataset.set_instance_config(DatasetConfig(use_cache=False, auto_sort_index=False))
See Also
instance_config- Get the effective configurationset_instance_config_kwargs()- Update settingsset_class_config()- Set class-level defaults
Source code in src/mesqual/datasets/dataset.py
1576 1577 1578 1579 1580 1581 1582 1583 1584 1585 1586 1587 1588 1589 1590 1591 1592 1593 1594 1595 | |
merge_instance_config
¶
merge_instance_config(config: DatasetConfigType) -> None
Merge config on top of the instance config of this dataset.
Uses DatasetConfig.merge on copies: a field that config sets
explicitly wins, and the other explicit fields of the instance
config stay. Without an instance config, a copy of config becomes
the instance config. Later changes to config do not reach the
dataset.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
config
|
DatasetConfigType
|
Upper layer for the instance config. |
required |
Examples:
>>> from mesqual.datasets import DatasetConfig
>>> dataset.set_instance_config(DatasetConfig(use_cache=False))
>>> dataset.merge_instance_config(DatasetConfig(granularity='1h'))
>>> dataset.instance_config.use_cache, dataset.instance_config.granularity
(False, '1h')
See Also
set_instance_config()- Replace the config
Source code in src/mesqual/datasets/dataset.py
1597 1598 1599 1600 1601 1602 1603 1604 1605 1606 1607 1608 1609 1610 1611 1612 1613 1614 1615 1616 1617 1618 1619 1620 | |
set_instance_config_kwargs
¶
set_instance_config_kwargs(**kwargs: Any) -> None
Update individual configuration settings on this instance.
Modifies specific attributes of the existing instance configuration without replacing the entire config object. Useful for tweaking individual settings.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
**kwargs
|
Any
|
Configuration attribute names and values to set |
{}
|
Examples:
>>> dataset.set_instance_config_kwargs(use_cache=False)
>>> dataset.instance_config.use_cache
False
Creates an instance config of get_config_type() first if the
dataset has none.
See Also
set_instance_config()- Replace entire configurationinstance_config- Get the effective configuration
Source code in src/mesqual/datasets/dataset.py
1622 1623 1624 1625 1626 1627 1628 1629 1630 1631 1632 1633 1634 1635 1636 1637 1638 1639 1640 1641 1642 1643 1644 1645 1646 1647 1648 | |
set_class_config
classmethod
¶
set_class_config(config: DatasetConfigType) -> None
Set the class-level configuration for all instances of this dataset type.
Class-level configuration serves as the default for all instances of this class. Instance-level configuration (set via set_instance_config) can override these defaults.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
config
|
DatasetConfigType
|
Configuration object to use as class-level defaults |
required |
Examples:
>>> from mesqual.datasets import DatasetConfig
>>> from mesqual.testing import DummyDataset
>>> DummyDataset.set_class_config(DatasetConfig(use_cache=False))
>>> study.scen.get_dataset('high_demand').instance_config.use_cache
False
Note
This affects all instances of the class, including existing ones that haven't overridden the setting at instance level.
See Also
set_instance_config()- Override for specific instancesDatasetConfigManager- Configuration management system
Source code in src/mesqual/datasets/dataset.py
1650 1651 1652 1653 1654 1655 1656 1657 1658 1659 1660 1661 1662 1663 1664 1665 1666 1667 1668 1669 1670 1671 1672 1673 1674 1675 1676 1677 1678 1679 | |
is_flag_available
¶
is_flag_available(flag: FlagType) -> bool | None
Return whether this dataset can answer a flag, without reading time series data.
A flag that the dataset does not accept, or a flag of a family with
a variant value that is not a value on the dataset, is not available
(False). Else:
- Platform dataset: the
_is_availableof the answering interpreter decides. If it returnsNone, the flag is unknown (None) until its first successful fetch from the platform dataset. - Collection: the union of the children (available in one child,
else unknown in one child, else
False). Comparison: the intersection of both children (Falseif one child isFalse, else unknown if one child is unknown, else available). - Any other dataset: its own
_is_available.Nonecounts as available.
The interpreter of a layer flag implements _is_available with
the availability of the flags that its fetch reads: the base flag,
the base model or border model, and the input_flags of the
aggregation. availability and flag_info give the reason.
Returns:
| Type | Description |
|---|---|
bool | None
|
|
Source code in src/mesqual/datasets/dataset.py
1681 1682 1683 1684 1685 1686 1687 1688 1689 1690 1691 1692 1693 1694 1695 1696 1697 1698 1699 1700 1701 1702 1703 1704 1705 1706 1707 | |
availability
¶
availability(flag: FlagType) -> Availability
Return whether this dataset can answer a flag, with the reason. Reads no time series data.
The same rules as is_flag_available, which returns the
available field of the result. An interpreter uses it on its
parent dataset to combine the availability of the flags that it
fetches (Availability.of_inputs).
Examples:
>>> from mesqual.testing import DummyDataset, DummyScenario
>>> without_prices = DummyDataset(DummyScenario(missing_tables=('node_price',)))
>>> without_prices.availability('Country.Results.price').reason
"DummyAreaFlags._is_available: unavailable input Node.Results.price (DummyTimeSeriesInterpreter._is_available: table 'node_price' not in the source)"
Source code in src/mesqual/datasets/dataset.py
1709 1710 1711 1712 1713 1714 1715 1716 1717 1718 1719 1720 1721 1722 1723 | |
describe_class_flags
classmethod
¶
describe_class_flags(contains: str | None = None, layer: str | None = None, item_type: ItemTypeEnum | str | None = None, as_records: bool = False) -> DataFrame | list[dict]
Describe the flag specs that this class declares, one row per spec. Needs no source.
On a platform dataset class: the specs of its answering
interpreters. On a dataset or interpreter class: its specs.
Flag families show their template, and dimensions with values per
dataset show 'resolved per dataset'.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
contains
|
str | None
|
Keep specs whose name or one of whose flags contains this substring, case-insensitive. |
None
|
layer
|
str | None
|
Keep specs on this area or border layer. |
None
|
item_type
|
ItemTypeEnum | str | None
|
Keep specs of this item type. |
None
|
as_records
|
bool
|
Return a list of plain dicts, ready for JSON. |
False
|
Returns:
| Type | Description |
|---|---|
DataFrame | list[dict]
|
One row per spec, indexed by name, with the columns |
Raises:
| Type | Description |
|---|---|
TypeError
|
Called on a collection class other than a platform
dataset class, or through an instance of one. A collection
has no flags of its own: use |
See Also
describe_flags: the flags of one dataset instance, with resolved flag families and layers, effective parameter defaults and availability. Same index, and the same columns plus some more.
Examples:
>>> from mesqual.testing import DummyDataset
>>> DummyDataset.describe_class_flags(contains='loading')[['object', 'unit']]
object unit
name
Line.Results.loading Line.Model per_unit
Source code in src/mesqual/datasets/dataset.py
1846 1847 1848 1849 1850 1851 1852 1853 1854 1855 1856 1857 1858 1859 1860 1861 1862 1863 1864 1865 1866 1867 1868 1869 1870 1871 1872 1873 1874 1875 1876 1877 1878 1879 1880 1881 1882 1883 1884 1885 1886 1887 1888 1889 1890 1891 1892 1893 1894 1895 | |
describe_flags
¶
describe_flags(contains: str | None = None, layer: str | None = None, item_type: ItemTypeEnum | str | None = None, available_only: bool = True, as_records: bool = False) -> DataFrame | list[dict]
Describe the flags of this dataset, one row per spec. Loads no time series data.
As describe_class_flags, with the values of flag families and layers
resolved for this dataset (from model flags), the effective parameter
defaults with their source (spec, dataset, config, required,
or per dataset on a collection whose children resolve different
defaults), and the availability. A collection describes the union of
its children.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
contains
|
str | None
|
Keep specs whose name or one of whose flags contains this substring, case-insensitive. |
None
|
layer
|
str | None
|
Keep specs on this area or border layer. |
None
|
item_type
|
ItemTypeEnum | str | None
|
Keep specs of this item type. |
None
|
available_only
|
bool
|
Drop the specs whose flags are all unavailable. Specs with unknown availability stay. |
True
|
as_records
|
bool
|
Return a list of plain dicts, ready for JSON. |
False
|
Returns:
| Type | Description |
|---|---|
DataFrame | list[dict]
|
One row per spec, indexed by name, with the columns of
|
See Also
describe_class_flags: the specs as declared on the class, without a source; no resolved values and no availability.
Examples:
>>> dataset.describe_flags(contains='loading')[['object', 'availability']]
object availability
name
Line.Results.loading Line.Model available
Source code in src/mesqual/datasets/dataset.py
1897 1898 1899 1900 1901 1902 1903 1904 1905 1906 1907 1908 1909 1910 1911 1912 1913 1914 1915 1916 1917 1918 1919 1920 1921 1922 1923 1924 1925 1926 1927 1928 1929 1930 1931 1932 1933 1934 1935 1936 1937 1938 1939 1940 1941 1942 1943 1944 1945 | |
flag_info
¶
Return everything mesqual knows about one flag of this dataset, without its data.
Returns:
| Type | Description |
|---|---|
FlagInfo
|
|
Raises:
| Type | Description |
|---|---|
UnknownFlagError
|
No spec declares the flag. |
Examples:
>>> dataset.flag_info('Line.Results.flow_received_net').description
'Power flow on the line. Side: where the flow leaves the line, after losses. Direction: up minus down.'
Source code in src/mesqual/datasets/dataset.py
1947 1948 1949 1950 1951 1952 1953 1954 1955 1956 1957 1958 1959 1960 1961 1962 1963 1964 1965 | |
describe_class_area_layers
classmethod
¶
describe_class_area_layers(as_records: bool = False) -> DataFrame | list[dict]
Describe the area layers registered on this class, one row per layer.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
as_records
|
bool
|
Return a list of plain dicts, ready for JSON. |
False
|
Returns:
| Type | Description |
|---|---|
DataFrame | list[dict]
|
One row per area layer, indexed by name, with the columns
|
Raises:
| Type | Description |
|---|---|
TypeError
|
Called on a collection class other than a platform dataset class, or through an instance of one. |
See Also
describe_class_border_layers: the border layers.
Source code in src/mesqual/datasets/dataset.py
1967 1968 1969 1970 1971 1972 1973 1974 1975 1976 1977 1978 1979 1980 1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 | |
describe_class_border_layers
classmethod
¶
describe_class_border_layers(as_records: bool = False) -> DataFrame | list[dict]
Describe the border layers registered on this class, one row per layer.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
as_records
|
bool
|
Return a list of plain dicts, ready for JSON. |
False
|
Returns:
| Type | Description |
|---|---|
DataFrame | list[dict]
|
One row per border layer, indexed by name, with the columns
|
Raises:
| Type | Description |
|---|---|
TypeError
|
Called on a collection class other than a platform dataset class, or through an instance of one. |
See Also
describe_class_area_layers: the area layers.
Source code in src/mesqual/datasets/dataset.py
1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 | |
describe_datasets
¶
describe_datasets(max_depth: int | None = None, as_records: bool = False) -> DataFrame | list[dict]
Describe the tree of this dataset and its children, one row per dataset.
A platform dataset is a leaf: its interpreters are not listed.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
max_depth
|
int | None
|
Number of levels below this dataset. |
None
|
as_records
|
bool
|
Return a list of plain dicts, ready for JSON. |
False
|
Returns:
| Type | Description |
|---|---|
DataFrame | list[dict]
|
One row per dataset, depth first, indexed by |
Raises:
| Type | Description |
|---|---|
ValueError
|
|
Examples:
>>> study.scen.describe_datasets(max_depth=1)['class'].tolist()
['DatasetConcatCollection', 'DatasetMergeCollection', 'DummyDataset', 'DummyDataset', 'DummyDataset']
Source code in src/mesqual/datasets/dataset.py
2017 2018 2019 2020 2021 2022 2023 2024 2025 2026 2027 2028 2029 2030 2031 2032 2033 2034 2035 2036 2037 2038 2039 2040 2041 2042 2043 2044 2045 2046 2047 2048 | |
DatasetCollection
¶
Bases: Generic[DatasetType, DatasetConfigType, FlagType, FlagIndexType], Dataset[DatasetConfigType, FlagType, FlagIndexType], ABC
Abstract base class for collections of datasets.
A DatasetCollection is a Dataset with child datasets. It has the Dataset API, so a collection can be a child of another collection. It gives access to its children, accepts the flags of all children and runs KPI operations on all children. A collection can be used wherever a Dataset is expected.
Each collection class gives the schema of its results with
transform_schema, so flag_info(flag).schema describes the fetched
result.
Attributes:
| Name | Type | Description |
|---|---|---|
datasets |
list[DatasetType]
|
List of child datasets in this collection |
Source code in src/mesqual/datasets/dataset_collection.py
36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 | |
flag_index
property
¶
flag_index: FlagIndex
The flag index: the union of the flag indexes of the children.
Raises:
| Type | Description |
|---|---|
ValueError
|
Two children have different specs with the same name. |
__init__
¶
__init__(datasets: list[DatasetType] = None, name: str = None, parent_dataset: Dataset = None, attributes: dict = None, cache: Cache = None, config: DatasetConfigType = None)
Create a collection of child datasets.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
datasets
|
list[DatasetType]
|
The child datasets. Their names must be unique. |
None
|
name
|
str
|
Name of the collection. |
None
|
parent_dataset
|
Dataset
|
Optional parent dataset. |
None
|
attributes
|
dict
|
Attributes of the collection. |
None
|
cache
|
Cache
|
Optional cache of fetch results. |
None
|
config
|
DatasetConfigType
|
Instance config. |
None
|
Raises:
| Type | Description |
|---|---|
ValueError
|
Two children have the same name, or two children have different specs with the same name. |
Source code in src/mesqual/datasets/dataset_collection.py
63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 | |
version_identity
¶
version_identity(flag: FlagType) -> tuple
Return one entry per child that a fetch of flag can read from, in order.
Each entry is a (class path, name) pair. A collection gives no
version of its own data; its children record their own identities.
The pairs make an entry of the collection stale for an added or
removed child, a child of another class or name, or a flag that
becomes available or unavailable on a child. A child replaced by one
of the same class and name keeps the entry: the name identifies the
source (see Cache).
DatasetConcatCollection.version_identity adds the concat layout
first.
Source code in src/mesqual/datasets/dataset_collection.py
131 132 133 134 135 136 137 138 139 140 141 142 143 144 | |
transform_schema
¶
transform_schema(schema: FrameSchema) -> FrameSchema
Return the schema of a result of this collection, from the schema of the result of a child.
Link, merge, sum and comparison collections keep the schema. A concat collection adds its level.
Source code in src/mesqual/datasets/dataset_collection.py
146 147 148 149 150 151 152 | |
get_merged_kpi_collection
¶
get_merged_kpi_collection(deep: bool = True) -> KPICollection
Merge KPI collections from all child datasets.
This method collects KPIs from all child datasets' kpi_collection properties and returns a unified collection. Optionally recurses into nested DatasetCollections.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
deep
|
bool
|
If True, recursively merge from nested DatasetCollections |
True
|
Returns:
| Type | Description |
|---|---|
KPICollection
|
KPICollection containing all KPIs from all child datasets |
Examples:
>>> from mesqual.kpis import Aggregations, FlagAggKPIBuilder
>>> definitions = FlagAggKPIBuilder().for_flag('Country.Results.price').with_aggregation(Aggregations.Mean).build()
>>> study.scen.add_kpis_from_definitions_to_all_child_datasets(definitions)
>>> len(study.scen.get_merged_kpi_collection()) # 4 scenarios, 2 countries
8
Source code in src/mesqual/datasets/dataset_collection.py
255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 | |
add_dataset
¶
add_dataset(dataset: DatasetType)
Add a child dataset. A child with the same name replaces the existing one and logs a warning.
The collection runs the checks of the constructor on the children with the new child. If a check fails, the collection does not change.
Raises:
| Type | Description |
|---|---|
TypeError
|
The dataset is not of |
ValueError
|
A name is a flag parameter of one dataset and a
dataset parameter ( |
Source code in src/mesqual/datasets/dataset_collection.py
394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 | |
fetch_merged
¶
fetch_merged(flag: FlagType, config: dict | DatasetConfigType = None, precedence: Literal['first', 'last'] | Sequence[str] = 'first', **kwargs: Any) -> Series | DataFrame
Fetch a flag from all child datasets and merge the results, as DatasetMergeCollection does.
The fetch goes through a temporary DatasetMergeCollection of the
children. It has no cache, so it caches nothing, and its record is
not kept. The records of the children live on the child datasets.
Dataset parameters of the children (for example comparison_type)
pass through to them.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
flag
|
FlagType
|
The flag. |
required |
config
|
dict | DatasetConfigType
|
Fetch-time config, as in |
None
|
precedence
|
Literal['first', 'last'] | Sequence[str]
|
Which child wins where two children overlap, as in
|
'first'
|
**kwargs
|
Any
|
Flag parameters and dataset parameters of the children. |
{}
|
Source code in src/mesqual/datasets/dataset_collection.py
426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 | |
get_merged_dataset_collection
¶
get_merged_dataset_collection(precedence: Literal['first', 'last'] | Sequence[str] = 'first') -> 'DatasetMergeCollection'
Return a DatasetMergeCollection of the children of this collection.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
precedence
|
Literal['first', 'last'] | Sequence[str]
|
Which child wins where two children overlap, as in
|
'first'
|
Source code in src/mesqual/datasets/dataset_collection.py
451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 | |
DatasetLinkCollection
¶
Bases: Generic[DatasetType, DatasetConfigType, FlagType, FlagIndexType], DatasetCollection[DatasetType, DatasetConfigType, FlagType, FlagIndexType]
Routes each flag to the one child dataset that accepts it.
A link collection combines child datasets that each answer a different
subset of flags into one dataset. It does not combine scenarios; use
DatasetConcatCollection or DatasetMergeCollection for that.
Routing
fetch(flag) goes to the one child that accepts the flag. If no
child accepts the flag, or the flag is unavailable on that child,
it raises ValueError. Two children that accept the same flag
raise AmbiguousOverrideError at construction.
Children reach their siblings through parent_dataset:
self.parent_dataset.fetch('other_flag', effective_config)self.parent_dataset.get_dataset_by_type(SomeDataset)
PlatformDataset builds on this class with an interpreter registry. To
replace a platform interpreter, register an override with
PlatformDataset.register_interpreter(Override, override_interpreter=Original, version=...).
The constructor checks the model_flag references of the union of the
child specs. So a time series child may refer to the model spec of a
sibling. Fetch such a child through the link collection: a child fetched
directly checks its own flag index.
Examples:
class ModelData(Dataset):
flag_specs = (GENERATOR_MODEL_SPEC,) # 'Generator.Model'
...
class TimeSeriesData(Dataset):
flag_specs = (GENERATION_SPEC,) # 'Generator.Results.generation', model_flag='Generator.Model'
...
linked = DatasetLinkCollection([ModelData(), TimeSeriesData()], name='base')
linked.fetch('Generator.Results.generation') # answered by TimeSeriesData
Source code in src/mesqual/datasets/dataset_collection.py
468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 | |
__init__
¶
__init__(datasets: list[DatasetType], name: str = None, parent_dataset: Dataset = None, attributes: dict = None, cache: Cache = None, config: DatasetConfigType = None)
Create a link collection.
The arguments are the same as for DatasetCollection.
Raises:
| Type | Description |
|---|---|
AmbiguousOverrideError
|
Two children accept the same flag. |
ValueError
|
Two children have the same name or different specs
with the same name, or a |
Source code in src/mesqual/datasets/dataset_collection.py
519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 | |
get_dataset_by_type
¶
get_dataset_by_type(ds_type: type[Dataset]) -> DatasetType
Returns instance of child dataset that matches the ds_type.
Source code in src/mesqual/datasets/dataset_collection.py
563 564 565 566 567 568 | |
DatasetMergeCollection
¶
Bases: Generic[DatasetType, DatasetConfigType, FlagType, FlagIndexType], DatasetCollection[DatasetType, DatasetConfigType, FlagType, FlagIndexType]
Merges the results of all children that have a flag into one result.
Use it for fragmented data of one scenario, for example simulation runs per month, or a model table from two sources.
The results combine pairwise (combine_dfs): disjoint columns with
overlapping index concat on columns; shared columns with disjoint index
concat on rows; else combine_first. Where two children overlap,
precedence decides which values stay. With a name list as
precedence, the columns of disjoint results also follow the order of
the list, and so do their rows if auto_sort_index is False.
Examples:
Two periods of one scenario:
>>> from mesqual.testing import DummyDataset, DummyScenario
>>> first, second = DummyScenario(days=4).split('2024-01-03')
>>> merged = DatasetMergeCollection([DummyDataset(first), DummyDataset(second)], name='base')
>>> len(merged.fetch('Node.Results.price')) # 4 days of hours
96
A rerun of the first day replaces the values of the base run there:
>>> base = DummyDataset(DummyScenario(days=4))
>>> rerun = DummyDataset(DummyScenario(name='rerun', days=1, demand_factor=1.2))
>>> patched = DatasetMergeCollection([base, rerun], precedence=['rerun', 'base'])
>>> patched.fetch('Node.Results.demand_total').iloc[0].equals(rerun.fetch('Node.Results.demand_total').iloc[0])
True
Source code in src/mesqual/datasets/dataset_collection.py
571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 | |
__init__
¶
__init__(datasets: list[DatasetType], name: str = None, parent_dataset: Dataset = None, attributes: dict = None, cache: Cache = None, config: DatasetConfigType = None, precedence: Literal['first', 'last'] | Sequence[str] = 'first')
Create a merge collection.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
datasets
|
list[DatasetType]
|
The child datasets. |
required |
name
|
str
|
Name of the collection. |
None
|
parent_dataset
|
Dataset
|
Optional parent dataset. |
None
|
attributes
|
dict
|
Attributes of the collection. |
None
|
cache
|
Cache
|
Optional cache of fetch results. |
None
|
config
|
DatasetConfigType
|
Instance config. |
None
|
precedence
|
Literal['first', 'last'] | Sequence[str]
|
Which child wins where two children overlap:
|
'first'
|
Raises:
| Type | Description |
|---|---|
ValueError
|
|
Source code in src/mesqual/datasets/dataset_collection.py
605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 | |
add_dataset
¶
add_dataset(dataset: DatasetType)
Add a child dataset. A child with the same name replaces the existing one and logs a warning.
With a name list as precedence, first set precedence to a list
that also names the new child, then add the child.
Raises:
| Type | Description |
|---|---|
TypeError
|
The dataset is not of |
ValueError
|
|
Source code in src/mesqual/datasets/dataset_collection.py
645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 | |
DatasetSumCollection
¶
Bases: Generic[DatasetType, DatasetConfigType, FlagType, FlagIndexType], DatasetCollection[DatasetType, DatasetConfigType, FlagType, FlagIndexType]
Sums the time series flags of its children and merges their model flags.
Time series of numeric data add element-wise. An object or row that one
child lacks gets the fill_value of the spec before the addition; a
spec without fill_value gives NaN there. Model flags merge as in
DatasetMergeCollection, the first child wins.
Source code in src/mesqual/datasets/dataset_collection.py
917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 | |
DatasetConcatCollection
¶
Bases: Generic[DatasetType, DatasetConfigType, FlagType, FlagIndexType], DatasetCollection[DatasetType, DatasetConfigType, FlagType, FlagIndexType]
Concatenates the results of its child datasets, with a level that names the child.
It fetches the same flag from each child dataset and concatenates the results into one DataFrame or Series. The scenarios and the comparisons of a study are concat collections.
MultiIndex Structure
The result has a new level named concat_level_name (default
dataset) with the child names, at the top of the axis
concat_axis: 1 (default) concats the child results as columns,
0 stacks them as rows. The axis is fixed per collection. Each
nested concat collection uses its own axis.
concat_top=False puts it last. The join is outer, so children with
different horizons give NaN outside their horizon.
The level names of the children must match on each axis. A child
whose result lacks levels of the longest child result, with the
other levels in the same order, gets these levels with the value
''. So a comparison with comparison_type=BOTH concats with
scenarios. If the child result and the longest child result have
unique level names, a child result with only level names of the
longest child result, in another order, raises ValueError that
names both children and both orders. Other differences raise
NotImplementedError.
With auto_sort_index, a concat on the columns sorts the rows. A
concat on the rows keeps the order of the children; each child
result is sorted already.
Examples:
>>> prices = study.scen.fetch('Node.Results.price')
>>> list(prices.columns.names)
['dataset', 'Node']
>>> base_prices = prices['base']
Stack the scenarios as rows:
>>> stacked = DatasetConcatCollection(study.scen.datasets, name='stacked', concat_axis=0)
>>> list(stacked.fetch('Node.Results.price').index.names)
['dataset', 'time']
Child parameters
It passes each child the dataset parameters that the child accepts
and that the call or param_defaults gave, for example
comparison_type of comparisons.
Source code in src/mesqual/datasets/dataset_collection.py
694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 | |
concat_axis
property
¶
concat_axis: int
Axis of the concat, fixed at construction: 1 columns, 0 rows.
concat_top
property
¶
concat_top: bool
True if the new level is at the top of the concat axis, False if at the bottom. Fixed at construction.
concat_level_name
property
¶
concat_level_name: str
Name of the new level with the child names. Fixed at construction.
__init__
¶
__init__(datasets: list[DatasetType], name: str = None, parent_dataset: Dataset = None, attributes: dict = None, cache: Cache = None, config: DatasetConfigType = None, concat_axis: int = 1, concat_top: bool = True, concat_level_name: str = None)
Create a concat collection.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
datasets
|
list[DatasetType]
|
The child datasets. Their names must be unique. |
required |
name
|
str
|
Name of the collection. |
None
|
parent_dataset
|
Dataset
|
Optional parent dataset. |
None
|
attributes
|
dict
|
Attributes of the collection. |
None
|
cache
|
Cache
|
Optional cache of fetch results. |
None
|
config
|
DatasetConfigType
|
Instance config. |
None
|
concat_axis
|
int
|
Axis of the concat: |
1
|
concat_top
|
bool
|
|
True
|
concat_level_name
|
str
|
Name of the new level. Default |
None
|
Raises:
| Type | Description |
|---|---|
ValueError
|
|
Source code in src/mesqual/datasets/dataset_collection.py
749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 | |
transform_schema
¶
transform_schema(schema: FrameSchema) -> FrameSchema
Add Level(concat_level_name, 'dataset') to the axis of the concat.
The axis is concat_axis of the collection. The level is at the
top, or at the bottom with concat_top=False.
Source code in src/mesqual/datasets/dataset_collection.py
812 813 814 815 816 817 818 819 820 821 822 | |
version_identity
¶
version_identity(flag: FlagType) -> tuple
As DatasetCollection.version_identity, with ('concat', concat_axis, concat_top, concat_level_name) first.
So a collection with the same name and another layout of its result does not reuse a cache entry.
Source code in src/mesqual/datasets/dataset_collection.py
830 831 832 833 834 835 836 837 | |
DatasetComparison
¶
Bases: Generic[DatasetType, DatasetConfigType, FlagType, FlagIndexType], DatasetCollection[DatasetType, DatasetConfigType, FlagType, FlagIndexType]
The difference between a variation dataset and a reference dataset, with the Dataset interface.
fetch fetches the flag from both datasets and compares the results.
Numeric and non-numeric data, frames and series are handled.
Comparison Types
- DELTA: Variation - Reference (default). For non-numeric data, a
changed cell becomes
"<var> (was <ref>)". An entry (row or column) that only the variation has becomes"<var> (new)"; one that only the reference has becomes"DELETED: <ref>". A NaN value in an entry of one side only stays NaN and gets no marker. - VARIATION: Returns variation data with optional NaN for unchanged values
- BOTH: Side-by-side variation and reference data, with a new last
column level
compared(BOTH_LEVEL_NAME) that holds the names of the two datasets.
All comparison types work for frames and series. For two series,
DELTA and VARIATION return a series and BOTH a frame with the
compared columns.
Attributes:
| Name | Type | Description |
|---|---|---|
variation_dataset |
The dataset representing the scenario being compared |
|
reference_dataset |
The dataset representing the baseline for comparison |
Dataset parameters (fetch_params):
comparison_type:DELTA(default),VARIATIONorBOTH.replace_unchanged_values_by_nan:Truereplaces values that are equal in both datasets by NaN. DefaultFalse. Only for flags that are not time series, for example model flags, so the comparison is empty where nothing changed. For a time series flag,TrueraisesValueError, also inside a collection. Mask the final result instead: for a numericDELTA, for exampledelta.mask(delta == 0)after any conversion; forVARIATIONorBOTH, compare with the reference.fill_value: Value for missing entries before the subtraction ofDELTA. DefaultNone: thefill_valueof the flag spec.float('nan'): no fill, so an entry missing on one side gives NaN, also if the spec has afill_value.
Each parameter resolves like a flag parameter (Dataset.fetch).
Flag parameters of the call go to both children. A dataset parameter that the call or
param_defaults gave also goes to a child that accepts it, for
example comparison_type to the children of a comparison of
comparisons.
Examples:
The model change of a scenario without node N4:
>>> base = study.scen.get_dataset('base')
>>> without_n4 = DatasetComparison(study.scen.get_dataset('without_N4'), base)
>>> without_n4.fetch('Node.Model', replace_unchanged_values_by_nan=True)['country']
Node
N1 NaN
N2 NaN
N3 NaN
N4 DELETED: C2
Name: country, dtype: object
comparison_type from the call, fill_value from param_defaults
and replace_unchanged_values_by_nan from its default:
>>> from mesqual.datasets import DatasetConfig
>>> from mesqual.enums import ComparisonTypeEnum
>>> high_demand = DatasetComparison(
... study.scen.get_dataset('high_demand'), base, config=DatasetConfig(param_defaults={'fill_value': 0}),
... )
>>> both = high_demand.fetch('Node.Results.price', comparison_type=ComparisonTypeEnum.BOTH)
>>> list(both.columns.names)
['Node', 'compared']
>>> high_demand.get_fetch_record('Node.Results.price', comparison_type=ComparisonTypeEnum.BOTH).params
{'comparison_type': <ComparisonTypeEnum.BOTH: 'both'>, 'replace_unchanged_values_by_nan': False, 'fill_value': 0}
Source code in src/mesqual/datasets/dataset_comparison.py
42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 | |
BOTH_LEVEL_NAME
class-attribute
instance-attribute
¶
BOTH_LEVEL_NAME = 'compared'
Name of the last column level that BOTH adds. Its values are the names of the variation and the reference.
fetch
¶
fetch(flag: FlagType, config: dict | DatasetConfigType = None, *, comparison_type: ComparisonTypeEnum | Unset = UNSET, replace_unchanged_values_by_nan: bool | Unset = UNSET, fill_value: float | None | Unset = UNSET, **kwargs: Any) -> Series | DataFrame
Fetch a flag from both datasets and compare them, as Dataset.fetch.
A dataset parameter left out of the call resolves from
param_defaults of the effective config, else from the default
given below.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
flag
|
FlagType
|
The flag. |
required |
config
|
dict | DatasetConfigType
|
Config overrides for this call, as in |
None
|
comparison_type
|
ComparisonTypeEnum | Unset
|
|
UNSET
|
replace_unchanged_values_by_nan
|
bool | Unset
|
|
UNSET
|
fill_value
|
float | None | Unset
|
Value for missing entries before the subtraction of
|
UNSET
|
**kwargs
|
Any
|
Flag parameters, which go to both datasets, and the |
{}
|
Raises:
| Type | Description |
|---|---|
ValueError
|
As |
Source code in src/mesqual/datasets/dataset_comparison.py
188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 | |
transform_schema
¶
transform_schema(schema: FrameSchema) -> FrameSchema
Add Level(BOTH_LEVEL_NAME, 'dataset') as the last column level if the default comparison_type is BOTH.
The default comes from the param_defaults of the instance config,
else from the dimension.
Source code in src/mesqual/datasets/dataset_comparison.py
231 232 233 234 235 236 237 238 239 240 241 242 | |
DatasetConcatCollectionOfComparisons
¶
Bases: Generic[DatasetConfigType, FlagType, FlagIndexType], DatasetConcatCollection[DatasetComparison, DatasetConfigType, FlagType, FlagIndexType]
Concat collection of comparisons.
Constructor as DatasetConcatCollection, including concat_axis.
Child parameters
It passes each child the dataset parameters that the child accepts
and that the call or param_defaults gave, for example
comparison_type of comparisons.
A comparison parameter that the call or the param_defaults of the
collection does not give reaches no comparison. Each comparison then resolves it with its own config, so a
comparison gives the same numbers inside the collection as alone.
Examples:
Each comparison gets comparison_type=BOTH as keyword argument and
resolves replace_unchanged_values_by_nan and fill_value itself:
>>> from mesqual.enums import ComparisonTypeEnum
>>> both = study.comp.fetch('Node.Results.price', comparison_type=ComparisonTypeEnum.BOTH)
>>> study.comp.get_fetch_record('Node.Results.price', comparison_type=ComparisonTypeEnum.BOTH).params
{'comparison_type': <ComparisonTypeEnum.BOTH: 'both'>}
Source code in src/mesqual/datasets/dataset_comparison.py
400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 | |
fetch
¶
fetch(flag: FlagType, config: dict | DatasetConfigType = None, *, comparison_type: ComparisonTypeEnum | Unset = UNSET, replace_unchanged_values_by_nan: bool | Unset = UNSET, fill_value: float | None | Unset = UNSET, **kwargs: Any) -> Series | DataFrame
Fetch a flag from each comparison and concat the results, as Dataset.fetch.
A dataset parameter left out of the call resolves from
param_defaults of the effective config, else from the default
given below.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
flag
|
FlagType
|
The flag. |
required |
config
|
dict | DatasetConfigType
|
Config overrides for this call, as in |
None
|
comparison_type
|
ComparisonTypeEnum | Unset
|
|
UNSET
|
replace_unchanged_values_by_nan
|
bool | Unset
|
|
UNSET
|
fill_value
|
float | None | Unset
|
Value for missing entries before the subtraction of
|
UNSET
|
**kwargs
|
Any
|
Flag parameters, which go to all comparisons, and the |
{}
|
Source code in src/mesqual/datasets/dataset_comparison.py
427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 | |
PlatformDataset
¶
Bases: Generic[DatasetType, DatasetConfigType, FlagType, FlagIndexType], DatasetLinkCollection[DatasetType, DatasetConfigType, FlagType, FlagIndexType], ABC
Base class for platform-specific datasets with an interpreter registry.
A platform dataset holds one instance of each interpreter that answers (per original interpreter, the original or the override that replaces it) and routes each flag to the interpreter that declares its spec.
Registry rules
- Each subclass has its own registry.
register_interpreteron a class does not change its parent or sibling classes. get_registered_interpreters()resolves the registry at call time along the MRO, so a parent registration made after the subclass definition reaches the subclass.- An instance resolves the registry once, in
__init__, and builds the interpreter chains and the flag index there.register_interpreterraises for invalid arguments and for conflicts between overrides in the own registry of the class. All other spec and override errors are raised at instance creation and in the class lookups. It asks the interpreters for theiraccepted_flagsat call time (fetch,flag_is_accepted,accepted_flags,get_flags_by_interpreter), soaccepted_flagsof an interpreter can read its siblings. datasetsandget_registered_interpreters()list the answering interpreters of base classes first, then in registration order.- Any
Datasetsubclass can be registered as interpreter, for example anAreaFlagInterpreter.get_child_dataset_type()gives the constructor arguments that the platform passes to them. - Each spec has exactly one answering interpreter. Two answering
interpreters that declare the same spec name raise
AmbiguousOverrideErrorat instance creation; the message names both interpreters. - A platform dataset class has no specs of its own. A subclass that
sets
flag_specsraisesTypeErrorat class creation. - Each interpreter is named
f'{platform.name}.{Interpreter.__qualname__}'. Two registered interpreters with the same__qualname__raiseValueErrorat instance creation; the message names their modules.
Source inventory and default attributes
A subclass implements build_source_inventory with cheap reads of
its source. source_inventory builds it once per instance. The
interpreters read it in their _is_available. _default_attributes
adds attributes from the inventory or model flags; explicit
attributes win. _default_attributes runs in __init__: set the
fields that build_source_inventory reads before calling
super().__init__.
Overrides
register_interpreter(Override, override_interpreter=Original, version=1)
replaces a whole interpreter. The override answers its own
flag_specs and reuses code by subclassing and super()._fetch.
Layer flags that fetch a flag of the override through
self.parent_dataset.fetch(...) get the override result.
get_answering_interpreter(flag) returns the interpreter that
answers now, so a package can subclass it to build on it.
Processing after the interpreter chain
_fetch is the hook to process the result of each flag after its
interpreter chain: a subclass overrides it and changes the result
of super()._fetch(flag, effective_config, **params). The rules
are in Interpreter overrides,
item "After the chain".
Examples:
class MyPlatformDataset(PlatformDataset[MyInterpreter, ...]):
def build_source_inventory(self):
return MyInventory(fingerprint=hash_of_file_keys(self.folder), files=...)
@classmethod
def get_child_dataset_type(cls):
return MyInterpreter
@MyPlatformDataset.register_interpreter
class NodeInterpreter(MyInterpreter):
flag_specs = (NODE_MODEL_SPEC, NODE_PRICE_SPEC)
...
A study dataset that rounds the node price after the interpreter chain:
>>> from mesqual.testing import DummyDataset, DummyScenario
>>> class RoundedDataset(DummyDataset):
... def _fetch(self, flag, effective_config, **params):
... result = super()._fetch(flag, effective_config, **params)
... return result.round(0) if flag == 'Node.Results.price' else result
>>> RoundedDataset(DummyScenario()).fetch('Node.Results.price').iloc[0].tolist()
[38.0, 44.0, 46.0, 55.0]
Source code in src/mesqual/datasets/platform_dataset.py
275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 | |
flag_index
property
¶
flag_index: FlagIndex
The flag index: built at construction from the specs of the answering interpreters and the layers of the class.
source_inventory
property
¶
source_inventory: SourceInventory
The source inventory, built once per instance with build_source_inventory, then kept in memory.
accepted_flags
property
¶
accepted_flags: set[FlagType]
The flags in the accepted_flags of each interpreter.
__init__
¶
__init__(name: str = None, attributes: dict = None, cache: Cache = None, config: DatasetConfigType = None, **kwargs: Any)
Create the interpreters of this class and build the flag index.
The keyword arguments are the own arguments below and the constructor
arguments of the interpreter base get_child_dataset_type(), without
parent_dataset. The platform dataset passes
these to every interpreter it creates: the given value, else the
default. A base argument with the name of an own argument (name,
attributes, cache, config) binds to the own argument, so the
interpreters get its default. Any other keyword argument raises
TypeError.
A subclass with arguments of its own takes them in its own __init__
and passes only the others to super().__init__, or makes them
constructor arguments of its interpreter base. For example,
DummyDataset takes a scenario and passes the interpreter argument
source.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
name
|
str
|
Dataset name. Default: the class name and the object id. |
None
|
attributes
|
dict
|
Attributes of the dataset. Default attributes from
|
None
|
cache
|
Cache
|
Optional cache of fetch results. |
None
|
config
|
DatasetConfigType
|
Optional instance config. |
None
|
**kwargs
|
Any
|
The constructor arguments of the interpreter base. |
{}
|
Raises:
| Type | Description |
|---|---|
TypeError
|
A keyword argument is neither an own argument nor a constructor argument of the interpreter base. The message names the unknown and the accepted arguments. |
Source code in src/mesqual/datasets/platform_dataset.py
381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 | |
build_source_inventory
abstractmethod
¶
build_source_inventory() -> SourceInventory
Build the source inventory of this dataset. Read file names and small tables only.
Returns:
| Type | Description |
|---|---|
SourceInventory
|
A |
Source code in src/mesqual/datasets/platform_dataset.py
449 450 451 452 453 454 455 456 457 | |
register_interpreter
classmethod
¶
register_interpreter(interpreter: Type[DatasetType] | None = None, *, override_interpreter: Type[DatasetType] | None = None, version: int | None = None) -> Type[DatasetType] | Callable[[Type[DatasetType]], Type[DatasetType]]
Register an interpreter, or an override that replaces a whole interpreter.
Usable as a decorator, with or without keyword arguments. The registration applies to this class and its subclasses only.
An override replaces override_interpreter on the dataset: the
replaced interpreter is not created, and the override answers its
own flag_specs. A subclass of the replaced interpreter inherits
its specs and can declare a spec again by name with other content.
Per original interpreter, the registration on the most derived
class answers. Reuse code by subclassing and super()._fetch; to
build on what answers now, subclass get_answering_interpreter(flag).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
interpreter
|
Type[DatasetType] | None
|
The interpreter class. |
None
|
override_interpreter
|
Type[DatasetType] | None
|
The interpreter that the override replaces: an original, or the override that answers on this class or a parent class. |
None
|
version
|
int | None
|
Version of the override logic. Required with
|
None
|
Returns:
| Type | Description |
|---|---|
Type[DatasetType] | Callable[[Type[DatasetType]], Type[DatasetType]]
|
The interpreter class, or a decorator if |
Raises:
| Type | Description |
|---|---|
TypeError
|
The interpreter or |
ValueError
|
|
ValueError
|
The interpreter names itself as |
ValueError
|
The interpreter is already registered on this class with other arguments. |
ValueError
|
The override overrides an override on this class, or an override on this class overrides it. |
AmbiguousOverrideError
|
Another override on this class names the
same |
A failed registration does not change the registry. At instance
creation and in the class lookups, ValueError if
override_interpreter does not answer on the class or a parent
class (the message names the interpreter to override instead), and
AmbiguousOverrideError if two overrides on one class replace one
original through different interpreters, or two overrides on
sibling classes replace one original without an override on a
common subclass.
Examples:
The override declares the price spec of DummyTimeSeriesInterpreter
again, with the extra parameter cap, and keeps its other specs:
>>> import dataclasses
>>> from mesqual.flag import Dimension
>>> from mesqual.testing import DummyDataset, DummyScenario
>>> from mesqual.testing.dataset import DummyTimeSeriesInterpreter as Original
>>> class StudyDataset(DummyDataset):
... pass
>>> CAP = Dimension('cap', 'Price cap in EUR/MWh.', default=1000)
>>> CAPPED_PRICE_SPEC = dataclasses.replace(Original.PRICE_SPEC, params=Original.PRICE_SPEC.params + (CAP,))
>>> @StudyDataset.register_interpreter(override_interpreter=Original, version=1)
... class CappedPrices(Original):
... flag_specs = (CAPPED_PRICE_SPEC,) + Original.flag_specs[1:]
... def _fetch(self, flag, effective_config, cap=1000, **params):
... result = super()._fetch(flag, effective_config, **params)
... return result.clip(upper=cap) if flag == 'Node.Results.price' else result
>>> float(StudyDataset(DummyScenario()).fetch('Node.Results.price', cap=40).max().max())
40.0
Source code in src/mesqual/datasets/platform_dataset.py
535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 | |
register_area_layer
classmethod
¶
register_area_layer(layer: AreaLayer) -> None
Register an area layer on this class and its subclasses.
The layer applies to every layer spec whose base is on
layer.base_model. An instance checks the layer against the model
specs at creation: the spec of the base model needs exactly one
membership (ModelFlagSpec.memberships) to the layer model, else
ValueError.
Raises:
| Type | Description |
|---|---|
TypeError
|
|
Examples:
A layer of price zones, here one zone per country through the
membership country of 'Node.Model':
>>> from mesqual.flag import AreaLayer
>>> from mesqual.testing import DummyDataset, DummyScenario
>>> class StudyDataset(DummyDataset):
... pass
>>> StudyDataset.register_area_layer(AreaLayer(
... name='Zone', index='Zone', base_model='Node.Model', layer_model='Country.Model',
... description='Price zones, one per country.',
... ))
>>> StudyDataset(DummyScenario()).fetch('Zone.Results.price').columns.tolist()
['C1', 'C2']
Source code in src/mesqual/datasets/platform_dataset.py
633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 | |
register_border_layer
classmethod
¶
register_border_layer(layer: BorderLayer) -> None
Register a border layer on this class and its subclasses.
Raises:
| Type | Description |
|---|---|
TypeError
|
|
Examples:
>>> from mesqual.flag import BorderLayer
>>> from mesqual.testing import DummyDataset, DummyScenario
>>> class StudyDataset(DummyDataset):
... pass
>>> StudyDataset.register_border_layer(BorderLayer(
... name='CountryPair', index='CountryPair', area_layer='Country', edge_model='Line.Model',
... description='Borders between two countries, crossed by lines.',
... ))
>>> StudyDataset(DummyScenario()).fetch('CountryPair.Model').index.tolist()
['C1 - C2', 'C2 - C1']
Source code in src/mesqual/datasets/platform_dataset.py
663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 | |
get_area_layers
classmethod
¶
get_area_layers() -> list[AreaLayer]
Return the area layers of this class and its parents, base classes first.
Source code in src/mesqual/datasets/platform_dataset.py
684 685 686 687 | |
get_border_layers
classmethod
¶
get_border_layers() -> list[BorderLayer]
Return the border layers of this class and its parents, base classes first.
Source code in src/mesqual/datasets/platform_dataset.py
689 690 691 692 | |
get_registered_interpreters
classmethod
¶
get_registered_interpreters() -> list[Type[DatasetType]]
Return the interpreters that a dataset of this class creates: per original, the one that answers.
Resolved at call time along the MRO, base classes first. A replaced interpreter is not in the list. Raises as at instance creation.
Source code in src/mesqual/datasets/platform_dataset.py
715 716 717 718 719 720 721 722 | |
get_answering_interpreter
classmethod
¶
get_answering_interpreter(flag: FlagType) -> Type[DatasetType]
Return the interpreter class that answers a flag on this class, after all registered overrides.
The lookup reads the specs and the registry, resolved along the MRO
at call time. It does not check if a source provides the flag: an
interpreter whose accepted_flags leaves out the flag on an
instance still answers it here. Subclass the result to build an
override on what answers now.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
flag
|
FlagType
|
A flag of a spec of a registered interpreter. |
required |
Returns:
| Type | Description |
|---|---|
Type[DatasetType]
|
The interpreter that answers the spec of the flag. |
Raises:
| Type | Description |
|---|---|
UnknownFlagError
|
No registered interpreter declares the flag. |
AmbiguousOverrideError
|
Two interpreters declare the spec, or overrides conflict, as at instance creation. |
ValueError
|
An override is not valid, as at instance creation. |
Examples:
>>> from mesqual.testing import DummyDataset
>>> class StudyDataset(DummyDataset):
... pass
>>> answering = StudyDataset.get_answering_interpreter('Node.Results.price')
>>> @StudyDataset.register_interpreter(override_interpreter=answering, version=1)
... class PriceFloor(answering):
... def _fetch(self, flag, effective_config, **params):
... return super()._fetch(flag, effective_config, **params).clip(lower=0)
>>> StudyDataset.get_answering_interpreter('Node.Results.price').__name__
'PriceFloor'
Source code in src/mesqual/datasets/platform_dataset.py
724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 | |
get_interpreter_instance
¶
get_interpreter_instance(interpreter_type: Type[DatasetType]) -> DatasetType
Return the interpreter instance of exactly this class (type(instance) is interpreter_type).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
interpreter_type
|
Type[DatasetType]
|
Interpreter class. |
required |
Returns:
| Type | Description |
|---|---|
DatasetType
|
The interpreter instance of this dataset. |
Raises:
| Type | Description |
|---|---|
ValueError
|
No interpreter of this class is registered. |
Source code in src/mesqual/datasets/platform_dataset.py
764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 | |
get_flags_by_interpreter
¶
get_flags_by_interpreter() -> dict[Type[DatasetType], set[FlagType]]
Return the flags each interpreter answers. A replaced interpreter is not on the dataset.
Source code in src/mesqual/datasets/platform_dataset.py
781 782 783 784 785 | |
flag_is_accepted
¶
flag_is_accepted(flag: FlagType) -> bool
True if the interpreter that declares the spec of the flag accepts it.
Source code in src/mesqual/datasets/platform_dataset.py
792 793 794 795 | |
version_identity
¶
version_identity(flag: FlagType) -> tuple
Return one (class name, version) pair per link of the interpreter chain of the flag.
The first link is the original interpreter with the version of the
spec in the flag index. Each override along the dataset class
hierarchy adds its register_interpreter(version=). A layer spec
adds the identity of its aggregation after the first link.
Examples:
>>> high_demand = study.scen.get_dataset('high_demand')
>>> high_demand.version_identity('Node.Results.price')
(('DummyTimeSeriesInterpreter', 1),)
>>> high_demand.version_identity('Country.Results.price')
(('DummyAreaFlags', 1), ('WeightedMean', {'weight': 'Node.Results.demand_total', 'version': 2}))
Source code in src/mesqual/datasets/platform_dataset.py
803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 | |
SourceInventory
dataclass
¶
What the source of a platform dataset holds, read with cheap reads only.
A platform dataset builds its inventory once (PlatformDataset.build_source_inventory),
for example from the file list and small tables of the source. mesqual
reads no field of the inventory. A platform subclass adds the fields
that its _is_available implementations and its default attributes
need.
Attributes:
| Name | Type | Description |
|---|---|---|
fingerprint |
str
|
Changes when the source changes, for example a hash of
the relative name and the |
Examples:
>>> from dataclasses import dataclass
>>> @dataclass(frozen=True)
... class MyInventory(SourceInventory):
... files: frozenset[str] = frozenset()
>>> MyInventory(fingerprint='3f2a', files=frozenset({'prices.csv'}))
MyInventory(fingerprint='3f2a', files=frozenset({'prices.csv'}))
Source code in src/mesqual/datasets/source_inventory.py
8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 | |
Availability
dataclass
¶
Whether a dataset can answer a flag.
Attributes:
| Name | Type | Description |
|---|---|---|
available |
bool | None
|
|
reason |
str
|
Where the answer comes from. |
datasets |
tuple[str, ...] | None
|
For a collection: names of the children on which the flag
is available or unknown. |
Source code in src/mesqual/datasets/describe.py
29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 | |
union
classmethod
¶
union(by_child: Mapping[str, Availability]) -> Availability
Combine the availability of the children of a collection.
Available if one child has the flag available, else unknown if one child does not know, else unavailable.
Source code in src/mesqual/datasets/describe.py
49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 | |
intersection
classmethod
¶
intersection(by_child: Mapping[str, Availability]) -> Availability
Combine the availability of children that are all needed, for example of a comparison.
Unavailable if one child has the flag unavailable, else unknown if one child does not know, else available.
Source code in src/mesqual/datasets/describe.py
66 67 68 69 70 71 72 73 74 75 76 | |
of_inputs
classmethod
¶
of_inputs(by_flag: Mapping[str, Availability]) -> Availability
Combine the availability of the flags that a flag needs.
Unavailable if one input is unavailable, else unknown if one input is unknown, else available. The reason names the unavailable or the unknown inputs with their reasons.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
by_flag
|
Mapping[str, Availability]
|
The availability of each input flag. |
required |
Examples:
>>> Availability.of_inputs({
... 'Node.Results.price': Availability(True, 'prices in the source'),
... 'Node.Results.demand': Availability(False, 'no demand table'),
... }).reason
'unavailable input Node.Results.demand (no demand table)'
Source code in src/mesqual/datasets/describe.py
78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 | |
FlagInfo
dataclass
¶
Everything mesqual knows about one flag of a dataset, without its data.
Attributes:
| Name | Type | Description |
|---|---|---|
flag |
str
|
The flag. |
spec |
FlagSpec
|
The flag spec. |
base |
FlagSpec | None
|
The base spec of a layer spec, else |
aggregation |
Any
|
The aggregation of a layer spec, else |
item_type |
ItemTypeEnum
|
Item type of the flag. |
variant |
dict[str, Any]
|
The variant values of the flag. |
description |
str
|
Spec description and one line per variant value. |
unit |
Unit | None
|
Unit of the values, |
unit_source |
str
|
|
quantity_type |
QuantityTypeEnum
|
Quantity type of the values. |
quantity_type_source |
str
|
|
schema |
FrameSchema | None
|
Index and column levels of the result. |
params |
tuple[ParamInfo, ...]
|
The flag parameters and the dataset parameters, with their effective defaults. |
fill_value |
float | None
|
Value for objects or rows that one side lacks, |
model_flag |
str | None
|
The linked model flag, or |
available |
bool | None
|
|
availability_reason |
str
|
Where the availability comes from. |
chain |
tuple[str, ...]
|
For a platform dataset: the interpreter chain of the flag, from the original interpreter to the one that answers. |
dependencies |
tuple[FetchRecord, ...]
|
The recorded dependencies of the last fetch of the flag without arguments. Empty before the first fetch. |
Source code in src/mesqual/datasets/describe.py
155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 | |
as_record
¶
as_record() -> dict
Return the info as a plain dict, ready for JSON.
Spec, base and aggregation appear by name, the schema as
{'index': [level, ...], 'columns': [level, ...], 'dtype': ...}
with one dict per level (name, role).
Source code in src/mesqual/datasets/describe.py
203 204 205 206 207 208 209 210 211 212 213 214 215 216 | |
ParamInfo
dataclass
¶
One parameter of a flag: a flag parameter of the spec or a dataset parameter of the dataset class.
Attributes:
| Name | Type | Description |
|---|---|---|
name |
str
|
Keyword argument of |
owner |
str
|
|
description |
str
|
What the parameter selects. |
values |
Mapping[Any, str] | str | None
|
Allowed values with their descriptions, |
default |
Any
|
Effective default. |
default_source |
str
|
|
Source code in src/mesqual/datasets/describe.py
114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 | |
as_record
¶
as_record() -> dict
Return the parameter as a plain dict, ready for JSON.
values is a list of {'value': ..., 'description': ...}, so the
default is equal to the value of one entry.
Source code in src/mesqual/datasets/describe.py
138 139 140 141 142 143 144 145 146 147 148 149 | |
DatasetConfig
dataclass
¶
Base config for the fetch behavior of a dataset.
A platform or a study subclasses it as a dataclass and adds fields with
config_field. A dataset declares its config class with
Dataset.get_config_type().
Layers resolve in this order, later wins: config class defaults, class
config of each class in the MRO (base first), instance config, fetch-time
config. A layer overrides only its explicit fields: the constructor
arguments, fields assigned after construction, or the keys of a dict.
An explicit None overrides like any other value. Set a field to
DatasetConfig.CLEAR to reset it to its class default and override the
lower layers.
Examples:
>>> prices = dataset.fetch('Node.Results.price', config=dict(auto_sort_index=False))
Source code in src/mesqual/datasets/dataset_config.py
88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 | |
explicit_fields
property
¶
explicit_fields: frozenset[str]
Names of the fields that this config sets explicitly.
merge
¶
merge(other: DatasetConfigType | dict | None) -> DatasetConfigType
Return a new config with the explicit fields of other on top of this one.
The result has the type of this config. Its explicit fields are the
union of the explicit fields of both layers. An explicit None in
other overrides like any other value. A field declared with
merge_per_key (param_defaults) adds the keys of other to its
dict. CLEAR resets a field to the
class default of this config; the result remembers the reset, so a
collection passes CLEAR on and each child resolves it against its
own default. A name that this config class does not define is kept
as is, so a collection can pass it on to its children.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
other
|
DatasetConfigType | dict | None
|
Upper layer: |
required |
Returns:
| Type | Description |
|---|---|
DatasetConfigType
|
The merged config. |
Examples:
>>> base = DatasetConfig(use_cache=False)
>>> base.merge({'auto_sort_index': False}).use_cache
False
Source code in src/mesqual/datasets/dataset_config.py
215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 | |
DatasetConfigManager
¶
Registry of class configs, the class layer of the config hierarchy.
Class configs are global state. A class config applies to all instances of that dataset class and of its subclasses.
Examples:
>>> from mesqual.testing import DummyDataset
>>> DatasetConfigManager.set_class_config(DummyDataset, DatasetConfig(use_cache=False))
>>> DatasetConfigManager.update_class_config_kwargs(DummyDataset, auto_sort_index=False)
>>> DatasetConfigManager.clear_class_configs()
Source code in src/mesqual/datasets/dataset_config.py
271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 | |
set_class_config
classmethod
¶
set_class_config(dataset_class: Type[Dataset], config: DatasetConfig) -> None
Set the class config of a dataset class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dataset_class
|
Type[Dataset]
|
Dataset class to configure. |
required |
config
|
DatasetConfig
|
Config whose explicit fields override the config defaults. |
required |
Source code in src/mesqual/datasets/dataset_config.py
286 287 288 289 290 291 292 293 294 | |
update_class_config_kwargs
classmethod
¶
update_class_config_kwargs(dataset_class: Type[Dataset], **config_kwargs: Any) -> None
Set fields on the existing class config of a dataset class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dataset_class
|
Type[Dataset]
|
Dataset class whose class config to update. |
required |
**config_kwargs
|
Any
|
Field names and values. |
{}
|
Raises:
| Type | Description |
|---|---|
KeyError
|
The dataset class has no class config. |
Source code in src/mesqual/datasets/dataset_config.py
296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 | |
clear_class_configs
classmethod
¶
clear_class_configs() -> None
Remove all class configs.
Source code in src/mesqual/datasets/dataset_config.py
312 313 314 315 | |
get_layers
classmethod
¶
get_layers(dataset_class: Type[Dataset], instance_config: DatasetConfig | None = None) -> list[tuple[str, DatasetConfig]]
Return the config layers of a dataset class, lowest first.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dataset_class
|
Type[Dataset]
|
Dataset class. |
required |
instance_config
|
DatasetConfig | None
|
Optional instance config, the top layer. |
None
|
Returns:
| Type | Description |
|---|---|
list[tuple[str, DatasetConfig]]
|
Pairs of layer label and config: |
Source code in src/mesqual/datasets/dataset_config.py
317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 | |
get_effective_config
classmethod
¶
get_effective_config(dataset_class: Type[Dataset], instance_config: DatasetConfig | None = None) -> DatasetConfig
Resolve the config of a dataset class without the fetch-time layer.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dataset_class
|
Type[Dataset]
|
Dataset class. |
required |
instance_config
|
DatasetConfig | None
|
Optional instance config. |
None
|
Returns:
| Type | Description |
|---|---|
DatasetConfig
|
The merged config, of the config type of the dataset class. |
Source code in src/mesqual/datasets/dataset_config.py
342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 | |
TimeIndexFilter
dataclass
¶
Keeps the rows whose value on one level of the time index passes the filter.
The filter acts on the time index of time series flags. level names a
level of the time index, for example hour of (day, hour).
None means the datetime level. A result without that level is returned
unchanged, so model flags pass through.
A row passes when its value is in include (if set) and not in exclude
(if set). include and exclude can mix plain values and Range objects.
On a datetime level, a datetime.date or an ISO date string matches the
local calendar date of the row, and a Timestamp matches that exact
timestamp. On any other level, values match directly.
On a MultiIndex, the other levels stay.
Attributes:
| Name | Type | Description |
|---|---|---|
level |
str | None
|
Level of the time index to filter. |
include |
Collection | None
|
Values or ranges to keep. Put a tuple value in a list
( |
exclude |
Collection | None
|
Values or ranges to drop. Same form as |
Examples:
>>> march_without_29th = TimeIndexFilter.dates(include=[Range('2024-03-01', '2024-03-31')], exclude=['2024-03-29'])
>>> prices = dataset.fetch('Node.Results.price', config={'time_index_filter': march_without_29th})
>>> sorted(set(prices.index.day))
[28, 30, 31]
>>> first_day_periods = TimeIndexFilter(level='period', include=[Range(1, 24)])
Source code in src/mesqual/datasets/time_index_filter.py
32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 | |
dates
classmethod
¶
dates(include: Collection | None = None, exclude: Collection | None = None) -> TimeIndexFilter
Short form for a filter on calendar dates of the time level.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
include
|
Collection | None
|
Dates, timestamps or ranges to keep. |
None
|
exclude
|
Collection | None
|
Dates, timestamps or ranges to drop. |
None
|
Returns:
| Type | Description |
|---|---|
TimeIndexFilter
|
A filter on the time level of the result. |
Source code in src/mesqual/datasets/time_index_filter.py
68 69 70 71 72 73 74 75 76 77 78 79 | |
apply
¶
apply(data: Series | DataFrame, time_level: str | int | None) -> Series | DataFrame
Filter the rows of data.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data
|
Series | DataFrame
|
Result to filter. |
required |
time_level
|
str | int | None
|
Name or position of the time level of |
required |
Returns:
| Type | Description |
|---|---|
Series | DataFrame
|
The rows that pass the filter. |
Source code in src/mesqual/datasets/time_index_filter.py
81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 | |
Range
dataclass
¶
Inclusive range of index values.
On a datetime level, start and end are dates (datetime.date or ISO
date strings), which match every day from start to end, or Timestamps,
which match start <= timestamp <= end. On any other level, the range
matches start <= value <= end.
Attributes:
| Name | Type | Description |
|---|---|---|
start |
Any
|
First value of the range. |
end |
Any
|
Last value of the range. |
Source code in src/mesqual/datasets/time_index_filter.py
14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 | |
AmbiguousOverrideError
¶
Bases: ValueError
Raised when more than one interpreter, override or spec claims the same flag or interpreter.
Two overrides of one interpreter on one class or on sibling classes, two answering interpreters with one spec name, or two specs that give one flag name raise this error.
Source code in src/mesqual/flag/exceptions.py
14 15 16 17 18 19 20 | |
FlagUnavailableError
¶
Bases: ValueError
A flag is not available on a dataset.
fetch raises it for an accepted flag whose availability is False. It
is a subclass of ValueError. Dimension.check_value raises it when the
values function of a flag family fetches an unavailable flag. For the
values of such a family, see Dimension.values_for.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
flag
|
str
|
The flag. |
required |
dataset
|
str
|
The name of the dataset. |
required |
reason
|
str
|
Why the flag is not available, as in |
required |
Attributes:
| Name | Type | Description |
|---|---|---|
flag |
The flag. |
|
dataset |
The name of the dataset. |
|
reason |
Why the flag is not available. |
Source code in src/mesqual/flag/exceptions.py
23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 | |
FetchContext
¶
The stack of open fetch frames of the current context.
In _fetch, call FetchContext.record_file_read(path) for each source
file, so an edited file makes the cache entries that read it stale. The
other methods are for mesqual itself.
The stack lives in context variables. Code in a thread started inside a fetch
runs with contextvars.copy_context().run(...), else its fetches count as
outermost and are not recorded in the open frames.
Source code in src/mesqual/datasets/fetch_context.py
411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 | |
current
classmethod
¶
current() -> FetchFrame | None
The innermost open frame, or None outside of a fetch.
Source code in src/mesqual/datasets/fetch_context.py
426 427 428 429 430 | |
is_outermost
classmethod
¶
is_outermost() -> bool
True if no fetch is open, so the next fetch is the outermost one.
Source code in src/mesqual/datasets/fetch_context.py
432 433 434 435 | |
is_recording
classmethod
¶
is_recording() -> bool
False while mesqual prepares a fetch, so its own config reads are not recorded.
Source code in src/mesqual/datasets/fetch_context.py
437 438 439 440 | |
frame
classmethod
¶
frame(record: FetchRecord, dataset: Any = None) -> Iterator[FetchFrame]
Open a frame for one fetch call of dataset. On exit, its record goes to the enclosing frame.
If the call raises, the record has raised=True.
Source code in src/mesqual/datasets/fetch_context.py
442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 | |
internal
classmethod
¶
internal() -> Iterator[None]
Suspend the recording of config reads, for mesqual code that prepares a fetch.
Source code in src/mesqual/datasets/fetch_context.py
463 464 465 466 467 468 469 470 471 | |
record_file_read
classmethod
¶
record_file_read(path: str | PathLike) -> None
Record that the open fetch read the file at path: path, size, modification time, content hash.
Call it in _fetch for each source file that the result depends on,
before the file is read, so a change during the read makes the
entry stale.
Call it in the thread of the fetch, or in a thread that runs with
contextvars.copy_context().run(...).
A cache entry is up to date only if each recorded file is unchanged,
so an edited file makes stale exactly the entries whose fetches read
it. Outside of a fetch, it does nothing. The content hash is computed
only if a fetch on the stack can store its result in a cache.
Raises:
| Type | Description |
|---|---|
ValueError
|
|
OSError
|
The file does not exist or cannot be read. |
Examples:
def _fetch(self, flag, effective_config, **params):
FetchContext.record_file_read(self.path)
return pd.read_csv(self.path)
Source code in src/mesqual/datasets/fetch_context.py
473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 | |
watch
staticmethod
¶
watch(config: DatasetConfig, frame: FetchFrame) -> DatasetConfig
Return a copy of config whose field reads go to frame, and store it as frame.watched_config.
Source code in src/mesqual/datasets/fetch_context.py
503 504 505 506 507 | |
FetchRecord
dataclass
¶
What one fetch call used: the call as given, the resolved parameters, the config reads and the dependencies.
A record has two views of a call:
- The call as given:
call_kwargsandcall_config. They identify the call. - The parameters after resolution:
params._fetchgot these values.
Every keyword argument of fetch is a parameter. A flag spec declares
flag parameters (FlagSpec.params). A dataset class declares dataset
parameters (Dataset.fetch_params). Both resolve the same way: the
keyword argument, then param_defaults of the effective config, then the
default of the dimension.
Examples:
The spec of 'Node.Results.price' declares the flag parameter
solution with the default 'final'. The instance config sets
param_defaults={'solution': 'first'}.
>>> high = study.scen.get_dataset('high_demand')
>>> high.set_instance_config_kwargs(param_defaults={'solution': 'first'})
>>> prices = high.fetch('Node.Results.price', config={'use_cache': False})
>>> record = high.get_fetch_record('Node.Results.price', config={'use_cache': False})
>>> record.call_kwargs, record.call_config
({}, {'use_cache': False})
>>> record.params, record.param_default_reads
({'solution': 'first'}, {'solution': 'first'})
solution is not in the call, so it comes from param_defaults.
Without that entry, it is 'final' and param_default_reads is empty.
The records form a tree. The record of a fetch lists the fetches that it
calls directly as its dependencies, and each of them lists its own. Every
nested fetch has its own record, also a call that a platform dataset
routes to an interpreter, a link collection to a child, or an override to
its base. iter_dependencies walks the tree.
Attributes:
| Name | Type | Description |
|---|---|---|
dataset_class |
str
|
|
dataset_name |
str
|
|
flag |
str
|
The fetched flag, as string. |
call_kwargs |
Mapping[str, Any]
|
The keyword arguments of the call, as given. Defaults are not filled in. |
call_config |
Mapping[str, Any]
|
The config override of the call: its explicit fields
with their values. |
params |
Mapping[str, Any]
|
All parameters after resolution: the flag parameters and the
dataset parameters, with defaults filled in. For a collection:
its own dataset parameters ( |
version |
tuple
|
Version identity of the flag ( |
param_default_reads |
Mapping[str, Any]
|
The entries of |
config_reads |
Mapping[str, Any]
|
Each config field that |
dependencies |
tuple[FetchRecord, ...]
|
The records of the fetches that this fetch called directly, in the order of the calls, each call key once. |
raised |
bool
|
|
file_reads |
tuple[FileRead, ...]
|
The state of each file that |
reads_given_params |
bool
|
|
The values in call_kwargs, call_config, params,
param_default_reads and config_reads are deep copies, taken when the
call recorded them. A record hashes by its key, so records work in
sets. DependencyCheck (mesqual.datasets.dependency_check) decides if a
stored record is up to date.
Source code in src/mesqual/datasets/fetch_context.py
171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 | |
key
property
¶
key: tuple
Identity of the call: dataset class, dataset name, flag, resolved parameters, call kwargs and call config.
Two fetches of one flag with different keyword arguments or config overrides are two keys.
given_params
property
¶
given_params: dict[str, Any]
The resolved parameters that the call or param_defaults gave: names in call_kwargs or param_default_reads.
A parameter that resolved to the default of its dimension is not here.
iter_dependencies
¶
iter_dependencies() -> Iterator[FetchRecord]
Walk the dependency tree depth first, parents before children.
A record that two parents call appears once under each parent.
Source code in src/mesqual/datasets/fetch_context.py
295 296 297 298 299 300 301 302 | |
params_key
classmethod
¶
params_key(params: Mapping[str, Any]) -> tuple
Canonical form of parameters: pairs of name and canonical text of the value, sorted by name.
The text is the repr of the value, with the items of a dict and of
a set sorted, so the insertion order does not change it.
Source code in src/mesqual/datasets/fetch_context.py
304 305 306 307 308 309 310 311 | |
canonical
classmethod
¶
canonical(value: Any) -> str
Canonical text of a value: its repr, with the items of dicts and sets sorted, also nested.
Source code in src/mesqual/datasets/fetch_context.py
313 314 315 316 317 318 319 320 321 322 323 | |
FileRead
dataclass
¶
The state of a file: path, size, modification time and, for a small file, a content hash.
The cache records one FileRead for each file that a fetch read. A cache entry is up to
date only if each of these files is unchanged (is_unchanged).
Attributes:
| Name | Type | Description |
|---|---|---|
path |
str
|
Absolute path of the file: the path as given, made absolute
with the working directory at record time. Symbolic links and
|
size |
int
|
Size in bytes. |
mtime_ns |
int
|
Modification time in nanoseconds. |
sha256 |
str | None
|
Hex digest of the content, for a file of at most
|
Source code in src/mesqual/datasets/fetch_context.py
86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 | |
HASH_MAX_BYTES
class-attribute
¶
HASH_MAX_BYTES: int = 16 * 1024 ** 2
Files up to this size in bytes get a content hash. A larger file is checked by size and modification time.
key
property
¶
key: str | tuple[str, int, int]
Identity of the file state without the directory.
A file with a content hash (at most HASH_MAX_BYTES, read with
with_hash=True) has the hash as key, without the file name. Any
other file has the key (file name, size, mtime_ns), so two copies of
it have the same key only if they have the same modification time.
A fingerprint pairs each key with the name of the file relative to
the source folder, so a rename or a swap of two files changes it.
of
classmethod
¶
of(path: str | PathLike, with_hash: bool = True) -> FileRead
Return the current state of the file at path.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
str | PathLike
|
The path of the file. |
required |
with_hash
|
bool
|
|
True
|
Raises:
| Type | Description |
|---|---|
ValueError
|
|
OSError
|
The file does not exist or cannot be read. |
Source code in src/mesqual/datasets/fetch_context.py
112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 | |
is_unchanged
¶
is_unchanged() -> bool
True if the file still exists with the same size and content.
A file with a recorded content hash is unchanged if the size and the hash are the same; the modification time does not count. A file without content hash is unchanged if the size and the modification time are the same.
Source code in src/mesqual/datasets/fetch_context.py
131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 | |
Unset
¶
Type of the UNSET marker, for the type hints of typed fetch overrides. There is one instance.
Source code in src/mesqual/datasets/fetch_context.py
50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 | |
config_field
¶
config_field(default: Any = MISSING, description: str = '', merge_per_key: bool = False, **kwargs: Any) -> Any
Declare a config field with a description.
The description goes into the field metadata. describe_config() reads
it from there.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
default
|
Any
|
Default value of the field. |
MISSING
|
description
|
str
|
One-line description of the field. |
''
|
merge_per_key
|
bool
|
For a dict field: a layer adds its keys to the dict of the lower layers instead of replacing it. |
False
|
**kwargs
|
Any
|
Further arguments for |
{}
|
Returns:
| Type | Description |
|---|---|
Any
|
A |
Examples:
>>> from dataclasses import dataclass
>>> @dataclass
... class MyPlatformConfig(DatasetConfig):
... convert_timestamps: bool = config_field(True, 'Convert period numbers to timestamps.')
Source code in src/mesqual/datasets/dataset_config.py
27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 | |