Cache References¶
A cache stores fetch results. Pass it to a dataset: MyDataset(name='base', cache=PickleCache('cache')).
- The key (
CacheKey) holds the dataset class, the dataset name, the flag and the resolved parameters. The version identity and the config are not in the key. - A key holds one entry: the value and the record of its fetch, with the version identity, the config fields and files that the fetch read, and its nested fetches. A fetch uses the entry only if the version identity is the same, each recorded read gives the same value now and each nested fetch is up to date. Else the fetch computes the value and replaces the entry.
- In
_fetch, callFetchContext.record_file_read(path)for each source file, so an edited file makes its entries stale.
cache
¶
Caches of fetch results: the cache key, the abstract Cache and the file-based PickleCache.
Entry
module-attribute
¶
Entry = tuple[FetchRecord, pd.Series | pd.DataFrame]
One cache entry: (record, value), the fetch record of the fetch that computed the value, and the value.
Cache
¶
Bases: ABC
Abstract storage of fetch results: one entry per cache key.
An entry is (record, value): the fetch record of the fetch that
computed the value, and the value. A subclass implements entry, set,
delete and list_keys. The cache does not check entries:
Dataset.fetch checks if the record of the entry is still up to date
(see Cache).
Source code in src/mesqual/cache/cache.py
62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 | |
entry
abstractmethod
¶
The entry of key as (record, value), or None if the key has no entry.
An exception does not fail the fetch: Dataset.fetch logs a warning
and computes the value as on a miss.
Source code in src/mesqual/cache/cache.py
72 73 74 75 76 77 78 | |
set
abstractmethod
¶
set(key: CacheKey, value: Series | DataFrame, record: FetchRecord) -> None
Store value with the fetch record record of its fetch as the entry of key.
A key holds one entry, so the new entry replaces the old one. An
exception does not fail the fetch: Dataset.fetch returns the value
and logs the reason.
Source code in src/mesqual/cache/cache.py
80 81 82 83 84 85 86 87 | |
delete
abstractmethod
¶
delete(dataset: Dataset | None = None, flag: Any = None) -> None
Delete the entries of the keys of dataset (class and name) and flag. None matches all.
Source code in src/mesqual/cache/cache.py
89 90 91 | |
list_keys
abstractmethod
¶
The keys of dataset (class and name) and flag that have an entry. None matches all.
Source code in src/mesqual/cache/cache.py
93 94 95 | |
CacheKey
dataclass
¶
The key of a cache entry: dataset class, dataset name, flag and parameters.
The version identity and the config reads are not in the key; the
entry's record holds them. Dataset.fetch uses the entry only if the
record is still up to date: same version identity and parameters, same
config reads and files, and each dependency up to date (see
Cache).
Attributes:
| Name | Type | Description |
|---|---|---|
dataset_class |
str
|
|
dataset_name |
str
|
|
flag |
str
|
The flag, as string. |
params |
tuple
|
The resolved parameters in canonical form
( |
Source code in src/mesqual/cache/cache.py
21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 | |
of
classmethod
¶
of(record: FetchRecord) -> CacheKey
The key of the fetch with the record record.
Source code in src/mesqual/cache/cache.py
45 46 47 48 | |
matches
¶
matches(dataset: Dataset | None = None, flag: Any = None) -> bool
True if the key belongs to dataset (class and name) and to flag. None matches all.
Source code in src/mesqual/cache/cache.py
55 56 57 58 59 | |
PickleCache
¶
Bases: Cache
Cache that stores each cache key in one pickle file in a folder.
The file name is the digest of the key (CacheKey.digest) with the
suffix SUFFIX, so any flag name works, and other files in the folder
are ignored. The file holds the key and its entry: the fetch record and
the value. A write goes to a temporary file first and then replaces the
file.
A file that cannot be read counts as a key without entry, with a
warning, and the next set of that key replaces it. A set that cannot
write, for example for a value that cannot be pickled, raises and leaves
no temporary file; Dataset.fetch returns the value and logs the
reason.
The file stores the path of each file read (FileRead.path) relative to
the folder, and entry joins it back onto the current folder. So the
entries stay up to date when the data and the cache folder move together.
After a move that changes the relative location of a file, the entry is
stale and is computed again. A path that holds .. or is on another
drive stays absolute.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
folder_path
|
str | PathLike
|
The folder of the files. It is created if it does not exist. |
required |
Examples:
>>> from mesqual.testing import DummyDataset, DummyScenario
>>> cache = PickleCache('cache')
>>> cached = DummyDataset(DummyScenario(), cache=cache)
>>> prices = cached.fetch('Node.Results.price')
>>> cache.list_keys(cached, 'Node.Results.price')
[CacheKey(dataset_class='mesqual.testing.dataset.DummyDataset', dataset_name='base', flag='Node.Results.price', ...)]
>>> cache.delete(cached)
Source code in src/mesqual/cache/pickle_cache.py
24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 | |
SUFFIX
class-attribute
instance-attribute
¶
SUFFIX = '.entry.pickle'
The suffix of the files of the cache. list_keys and delete read only files with this suffix.
entry
¶
The entry of key, read from the file of the key. None if the file does not exist or cannot be read.
Source code in src/mesqual/cache/pickle_cache.py
72 73 74 75 76 77 78 | |
set
¶
set(key: CacheKey, value: Series | DataFrame, record: FetchRecord) -> None
Replace the file of key with the new entry. If the entry cannot be written, raise and keep the file.
Source code in src/mesqual/cache/pickle_cache.py
80 81 82 | |
delete
¶
delete(dataset: Dataset | None = None, flag: Any = None) -> None
Delete the files of the keys of dataset and flag.
Source code in src/mesqual/cache/pickle_cache.py
84 85 86 87 | |
list_keys
¶
The keys of dataset and flag, read from the files with SUFFIX. Files that cannot be read are skipped.
Source code in src/mesqual/cache/pickle_cache.py
89 90 91 | |