Skip to content

Cache References

A cache stores fetch results. Pass it to a dataset: MyDataset(name='base', cache=PickleCache('cache')).

  • The key (CacheKey) holds the dataset class, the dataset name, the flag and the resolved parameters. The version identity and the config are not in the key.
  • A key holds one entry: the value and the record of its fetch, with the version identity, the config fields and files that the fetch read, and its nested fetches. A fetch uses the entry only if the version identity is the same, each recorded read gives the same value now and each nested fetch is up to date. Else the fetch computes the value and replaces the entry.
  • In _fetch, call FetchContext.record_file_read(path) for each source file, so an edited file makes its entries stale.

cache

Caches of fetch results: the cache key, the abstract Cache and the file-based PickleCache.

Entry module-attribute

Entry = tuple[FetchRecord, pd.Series | pd.DataFrame]

One cache entry: (record, value), the fetch record of the fetch that computed the value, and the value.

Cache

Bases: ABC

Abstract storage of fetch results: one entry per cache key.

An entry is (record, value): the fetch record of the fetch that computed the value, and the value. A subclass implements entry, set, delete and list_keys. The cache does not check entries: Dataset.fetch checks if the record of the entry is still up to date (see Cache).

Source code in src/mesqual/cache/cache.py
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
class Cache(ABC):
    """Abstract storage of fetch results: one entry per cache key.

    An entry is `(record, value)`: the fetch record of the fetch that
    computed the value, and the value. A subclass implements `entry`, `set`,
    `delete` and `list_keys`. The cache does not check entries:
    `Dataset.fetch` checks if the record of the entry is still up to date
    (see [Cache](../architecture/other-contracts.md#cache)).
    """

    @abstractmethod
    def entry(self, key: CacheKey) -> Entry | None:
        """The entry of `key` as `(record, value)`, or `None` if the key has no entry.

        An exception does not fail the fetch: `Dataset.fetch` logs a warning
        and computes the value as on a miss.
        """

    @abstractmethod
    def set(self, key: CacheKey, value: pd.Series | pd.DataFrame, record: FetchRecord) -> None:
        """Store `value` with the fetch record `record` of its fetch as the entry of `key`.

        A key holds one entry, so the new entry replaces the old one. An
        exception does not fail the fetch: `Dataset.fetch` returns the value
        and logs the reason.
        """

    @abstractmethod
    def delete(self, dataset: Dataset | None = None, flag: Any = None) -> None:
        """Delete the entries of the keys of `dataset` (class and name) and `flag`. `None` matches all."""

    @abstractmethod
    def list_keys(self, dataset: Dataset | None = None, flag: Any = None) -> list[CacheKey]:
        """The keys of `dataset` (class and name) and `flag` that have an entry. `None` matches all."""

entry abstractmethod

entry(key: CacheKey) -> Entry | None

The entry of key as (record, value), or None if the key has no entry.

An exception does not fail the fetch: Dataset.fetch logs a warning and computes the value as on a miss.

Source code in src/mesqual/cache/cache.py
72
73
74
75
76
77
78
@abstractmethod
def entry(self, key: CacheKey) -> Entry | None:
    """The entry of `key` as `(record, value)`, or `None` if the key has no entry.

    An exception does not fail the fetch: `Dataset.fetch` logs a warning
    and computes the value as on a miss.
    """

set abstractmethod

set(key: CacheKey, value: Series | DataFrame, record: FetchRecord) -> None

Store value with the fetch record record of its fetch as the entry of key.

A key holds one entry, so the new entry replaces the old one. An exception does not fail the fetch: Dataset.fetch returns the value and logs the reason.

Source code in src/mesqual/cache/cache.py
80
81
82
83
84
85
86
87
@abstractmethod
def set(self, key: CacheKey, value: pd.Series | pd.DataFrame, record: FetchRecord) -> None:
    """Store `value` with the fetch record `record` of its fetch as the entry of `key`.

    A key holds one entry, so the new entry replaces the old one. An
    exception does not fail the fetch: `Dataset.fetch` returns the value
    and logs the reason.
    """

delete abstractmethod

delete(dataset: Dataset | None = None, flag: Any = None) -> None

Delete the entries of the keys of dataset (class and name) and flag. None matches all.

Source code in src/mesqual/cache/cache.py
89
90
91
@abstractmethod
def delete(self, dataset: Dataset | None = None, flag: Any = None) -> None:
    """Delete the entries of the keys of `dataset` (class and name) and `flag`. `None` matches all."""

list_keys abstractmethod

list_keys(dataset: Dataset | None = None, flag: Any = None) -> list[CacheKey]

The keys of dataset (class and name) and flag that have an entry. None matches all.

Source code in src/mesqual/cache/cache.py
93
94
95
@abstractmethod
def list_keys(self, dataset: Dataset | None = None, flag: Any = None) -> list[CacheKey]:
    """The keys of `dataset` (class and name) and `flag` that have an entry. `None` matches all."""

CacheKey dataclass

The key of a cache entry: dataset class, dataset name, flag and parameters.

The version identity and the config reads are not in the key; the entry's record holds them. Dataset.fetch uses the entry only if the record is still up to date: same version identity and parameters, same config reads and files, and each dependency up to date (see Cache).

Attributes:

Name Type Description
dataset_class str

f'{cls.__module__}.{cls.__qualname__}' of the dataset.

dataset_name str

name of the dataset.

flag str

The flag, as string.

params tuple

The resolved parameters in canonical form (FetchRecord.params_key). A param_defaults entry reaches the key only through them.

Source code in src/mesqual/cache/cache.py
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
@dataclass(frozen=True)
class CacheKey:
    """The key of a cache entry: dataset class, dataset name, flag and parameters.

    The version identity and the config reads are not in the key; the
    entry's record holds them. `Dataset.fetch` uses the entry only if the
    record is still up to date: same version identity and parameters, same
    config reads and files, and each dependency up to date (see
    [Cache](../architecture/other-contracts.md#cache)).

    Attributes:
        dataset_class: `f'{cls.__module__}.{cls.__qualname__}'` of the dataset.
        dataset_name: `name` of the dataset.
        flag: The flag, as string.
        params: The resolved parameters in canonical form
            (`FetchRecord.params_key`). A `param_defaults` entry reaches the
            key only through them.
    """

    dataset_class: str
    dataset_name: str
    flag: str
    params: tuple

    @classmethod
    def of(cls, record: FetchRecord) -> CacheKey:
        """The key of the fetch with the record `record`."""
        return cls(record.dataset_class, record.dataset_name, record.flag, record.params_key(record.params))

    @property
    def digest(self) -> str:
        """Hex digest of the key parts, for example as a file name."""
        return hashlib.sha256(repr(self).encode()).hexdigest()

    def matches(self, dataset: Dataset | None = None, flag: Any = None) -> bool:
        """`True` if the key belongs to `dataset` (class and name) and to `flag`. `None` matches all."""
        if dataset is not None and (self.dataset_class, self.dataset_name) != (dataset._class_path(), dataset.name):
            return False
        return flag is None or self.flag == str(flag)

digest property

digest: str

Hex digest of the key parts, for example as a file name.

of classmethod

of(record: FetchRecord) -> CacheKey

The key of the fetch with the record record.

Source code in src/mesqual/cache/cache.py
45
46
47
48
@classmethod
def of(cls, record: FetchRecord) -> CacheKey:
    """The key of the fetch with the record `record`."""
    return cls(record.dataset_class, record.dataset_name, record.flag, record.params_key(record.params))

matches

matches(dataset: Dataset | None = None, flag: Any = None) -> bool

True if the key belongs to dataset (class and name) and to flag. None matches all.

Source code in src/mesqual/cache/cache.py
55
56
57
58
59
def matches(self, dataset: Dataset | None = None, flag: Any = None) -> bool:
    """`True` if the key belongs to `dataset` (class and name) and to `flag`. `None` matches all."""
    if dataset is not None and (self.dataset_class, self.dataset_name) != (dataset._class_path(), dataset.name):
        return False
    return flag is None or self.flag == str(flag)

PickleCache

Bases: Cache

Cache that stores each cache key in one pickle file in a folder.

The file name is the digest of the key (CacheKey.digest) with the suffix SUFFIX, so any flag name works, and other files in the folder are ignored. The file holds the key and its entry: the fetch record and the value. A write goes to a temporary file first and then replaces the file.

A file that cannot be read counts as a key without entry, with a warning, and the next set of that key replaces it. A set that cannot write, for example for a value that cannot be pickled, raises and leaves no temporary file; Dataset.fetch returns the value and logs the reason.

The file stores the path of each file read (FileRead.path) relative to the folder, and entry joins it back onto the current folder. So the entries stay up to date when the data and the cache folder move together. After a move that changes the relative location of a file, the entry is stale and is computed again. A path that holds .. or is on another drive stays absolute.

Parameters:

Name Type Description Default
folder_path str | PathLike

The folder of the files. It is created if it does not exist.

required

Examples:

>>> from mesqual.testing import DummyDataset, DummyScenario
>>> cache = PickleCache('cache')
>>> cached = DummyDataset(DummyScenario(), cache=cache)
>>> prices = cached.fetch('Node.Results.price')
>>> cache.list_keys(cached, 'Node.Results.price')
[CacheKey(dataset_class='mesqual.testing.dataset.DummyDataset', dataset_name='base', flag='Node.Results.price', ...)]
>>> cache.delete(cached)
Source code in src/mesqual/cache/pickle_cache.py
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
class PickleCache(Cache):
    """Cache that stores each cache key in one pickle file in a folder.

    The file name is the digest of the key (`CacheKey.digest`) with the
    suffix `SUFFIX`, so any flag name works, and other files in the folder
    are ignored. The file holds the key and its entry: the fetch record and
    the value. A write goes to a temporary file first and then replaces the
    file.

    A file that cannot be read counts as a key without entry, with a
    warning, and the next `set` of that key replaces it. A `set` that cannot
    write, for example for a value that cannot be pickled, raises and leaves
    no temporary file; `Dataset.fetch` returns the value and logs the
    reason.

    The file stores the path of each file read (`FileRead.path`) relative to
    the folder, and `entry` joins it back onto the current folder. So the
    entries stay up to date when the data and the cache folder move together.
    After a move that changes the relative location of a file, the entry is
    stale and is computed again. A path that holds `..` or is on another
    drive stays absolute.

    Args:
        folder_path: The folder of the files. It is created if it does not exist.

    Examples:
        >>> from mesqual.testing import DummyDataset, DummyScenario
        >>> cache = PickleCache('cache')
        >>> cached = DummyDataset(DummyScenario(), cache=cache)
        >>> prices = cached.fetch('Node.Results.price')
        >>> cache.list_keys(cached, 'Node.Results.price')
        [CacheKey(dataset_class='mesqual.testing.dataset.DummyDataset', dataset_name='base', flag='Node.Results.price', ...)]
        >>> cache.delete(cached)
    """

    SUFFIX = '.entry.pickle'
    """The suffix of the files of the cache. `list_keys` and `delete` read only files with this suffix."""

    def __init__(self, folder_path: str | os.PathLike):
        self._folder_path = Path(folder_path)
        self._folder_path.mkdir(parents=True, exist_ok=True)
        self._absolute_folder = self._folder_path.absolute()

    @property
    def folder_path(self) -> Path:
        """The folder of the files."""
        return self._folder_path

    def entry(self, key: CacheKey) -> Entry | None:
        """The entry of `key`, read from the file of the key. `None` if the file does not exist or cannot be read."""
        content = self._read(self._path(key))
        if content is None:
            return None
        record, value = content['entry']
        return self._with_paths(record, self._current_path), value

    def set(self, key: CacheKey, value: pd.Series | pd.DataFrame, record: FetchRecord) -> None:
        """Replace the file of `key` with the new entry. If the entry cannot be written, raise and keep the file."""
        self._write(key, (self._with_paths(record, self._stored_path), value))

    def delete(self, dataset: Dataset | None = None, flag: Any = None) -> None:
        """Delete the files of the keys of `dataset` and `flag`."""
        for key in self.list_keys(dataset, flag):
            self._path(key).unlink(missing_ok=True)

    def list_keys(self, dataset: Dataset | None = None, flag: Any = None) -> list[CacheKey]:
        """The keys of `dataset` and `flag`, read from the files with `SUFFIX`. Files that cannot be read are skipped."""
        return [key for key in self._keys() if key.matches(dataset, flag)]

    def _path(self, key: CacheKey) -> Path:
        return self._absolute_folder / f'{key.digest}{self.SUFFIX}'

    def _stored_path(self, path: str) -> str:
        """`path` relative to the folder, with `/`. Absolute if it holds `..`, which can follow a link, or has no relative form."""
        if os.path.normpath(path) != path:
            return path
        try:
            return Path(os.path.relpath(path, self._absolute_folder)).as_posix()
        except ValueError:
            return path

    def _current_path(self, path: str) -> str:
        """The absolute path of a stored path: a relative path joined onto the current folder."""
        if os.path.isabs(path):
            return path
        return os.path.normpath(os.path.join(self._absolute_folder, path))

    @classmethod
    def _with_paths(cls, record: FetchRecord, convert: Callable[[str], str]) -> FetchRecord:
        """`record` and its dependencies with `convert` applied to the path of each file read."""
        return dataclasses.replace(
            record,
            file_reads=tuple(dataclasses.replace(read, path=convert(read.path)) for read in record.file_reads),
            dependencies=tuple(cls._with_paths(dependency, convert) for dependency in record.dependencies),
        )

    @staticmethod
    def _read(path: Path) -> dict | None:
        """The content of the file at `path`, or `None` if it does not exist or cannot be read."""
        if not path.exists():
            return None
        try:
            with open(path, 'rb') as file:
                content = pickle.load(file)
            record, value = content['entry']
            return {'key': content['key'], 'entry': (record, value)}
        except Exception as error:
            logger.warning(f'Cache file {path} cannot be read and is ignored: {type(error).__name__}: {error}')
            return None

    def _write(self, key: CacheKey, entry: Entry) -> None:
        handle, temporary = tempfile.mkstemp(dir=self._absolute_folder, suffix='.tmp')
        try:
            with os.fdopen(handle, 'wb') as file:
                pickle.dump({'key': key, 'entry': entry}, file, protocol=pickle.HIGHEST_PROTOCOL)
            os.replace(temporary, self._path(key))
        finally:
            if os.path.exists(temporary):
                os.remove(temporary)

    def _keys(self) -> list[CacheKey]:
        contents = (self._read(path) for path in sorted(self._absolute_folder.glob(f'*{self.SUFFIX}')))
        return [content['key'] for content in contents if content is not None]

SUFFIX class-attribute instance-attribute

SUFFIX = '.entry.pickle'

The suffix of the files of the cache. list_keys and delete read only files with this suffix.

folder_path property

folder_path: Path

The folder of the files.

entry

entry(key: CacheKey) -> Entry | None

The entry of key, read from the file of the key. None if the file does not exist or cannot be read.

Source code in src/mesqual/cache/pickle_cache.py
72
73
74
75
76
77
78
def entry(self, key: CacheKey) -> Entry | None:
    """The entry of `key`, read from the file of the key. `None` if the file does not exist or cannot be read."""
    content = self._read(self._path(key))
    if content is None:
        return None
    record, value = content['entry']
    return self._with_paths(record, self._current_path), value

set

set(key: CacheKey, value: Series | DataFrame, record: FetchRecord) -> None

Replace the file of key with the new entry. If the entry cannot be written, raise and keep the file.

Source code in src/mesqual/cache/pickle_cache.py
80
81
82
def set(self, key: CacheKey, value: pd.Series | pd.DataFrame, record: FetchRecord) -> None:
    """Replace the file of `key` with the new entry. If the entry cannot be written, raise and keep the file."""
    self._write(key, (self._with_paths(record, self._stored_path), value))

delete

delete(dataset: Dataset | None = None, flag: Any = None) -> None

Delete the files of the keys of dataset and flag.

Source code in src/mesqual/cache/pickle_cache.py
84
85
86
87
def delete(self, dataset: Dataset | None = None, flag: Any = None) -> None:
    """Delete the files of the keys of `dataset` and `flag`."""
    for key in self.list_keys(dataset, flag):
        self._path(key).unlink(missing_ok=True)

list_keys

list_keys(dataset: Dataset | None = None, flag: Any = None) -> list[CacheKey]

The keys of dataset and flag, read from the files with SUFFIX. Files that cannot be read are skipped.

Source code in src/mesqual/cache/pickle_cache.py
89
90
91
def list_keys(self, dataset: Dataset | None = None, flag: Any = None) -> list[CacheKey]:
    """The keys of `dataset` and `flag`, read from the files with `SUFFIX`. Files that cannot be read are skipped."""
    return [key for key in self._keys() if key.matches(dataset, flag)]