Skip to content

MESQUAL Membership Property Enrichers

MembershipTagging

Bases: Enum

Controls how enriched property names are tagged when added to target DataFrames.

In energy system modeling, objects often have relationships to other model components (e.g., generators belong to nodes, lines connect nodes). When enriching a DataFrame with properties from related objects, this enum controls naming conventions to avoid column name conflicts and maintain clarity about property origins.

Values
  • NONE: Property names remain unchanged (may cause conflicts with existing columns)
  • PREFIX: Property names get membership name as prefix (e.g., 'node_voltage' from 'voltage')
  • SUFFIX: Property names get membership name as suffix (e.g., 'voltage_node' from 'voltage')

Examples:

For a generator DataFrame (target_df) with 'node' membership, enriching with node properties:

>>> MembershipPropertyEnricher().append_properties(target_df, dataset, MembershipTagging.NONE)
>>> # Returns target_df with new columns ['voltage', 'load'] (original names from node DataFrame)
>>>
>>> MembershipPropertyEnricher().append_properties(target_df, dataset, MembershipTagging.PREFIX)
>>> # Returns target_df with new columns ['node_voltage', 'node_load']
>>>
>>> MembershipPropertyEnricher().append_properties(target_df, dataset, MembershipTagging.SUFFIX)
>>> # Returns target_df with new columns ['voltage_node', 'load_node']
Source code in submodules/mesqual/mesqual/energy_data_handling/model_handling/membership_property_enrichers.py 
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
class MembershipTagging(Enum):
    """
    Controls how enriched property names are tagged when added to target DataFrames.

    In energy system modeling, objects often have relationships to other model components
    (e.g., generators belong to nodes, lines connect nodes). When enriching a DataFrame
    with properties from related objects, this enum controls naming conventions to avoid
    column name conflicts and maintain clarity about property origins.

    Values:
        - NONE: Property names remain unchanged (may cause conflicts with existing columns)
        - PREFIX: Property names get membership name as prefix (e.g., 'node_voltage' from 'voltage')
        - SUFFIX: Property names get membership name as suffix (e.g., 'voltage_node' from 'voltage')

    Examples:
        For a generator DataFrame (target_df) with 'node' membership, enriching with node properties:
        >>> MembershipPropertyEnricher().append_properties(target_df, dataset, MembershipTagging.NONE)
        >>> # Returns target_df with new columns ['voltage', 'load'] (original names from node DataFrame)
        >>>
        >>> MembershipPropertyEnricher().append_properties(target_df, dataset, MembershipTagging.PREFIX)
        >>> # Returns target_df with new columns ['node_voltage', 'node_load']
        >>>
        >>> MembershipPropertyEnricher().append_properties(target_df, dataset, MembershipTagging.SUFFIX)
        >>> # Returns target_df with new columns ['voltage_node', 'load_node']
    """
    NONE = "none"
    PREFIX = "prefix"
    SUFFIX = "suffix"

MembershipPropertyEnricher

Enriches energy system DataFrames with properties from related model objects.

In energy system modeling, entities often have membership relationships to other model components. For example: - Generators belong to nodes and have fuel types - Lines connect between nodes - Storage units are located at nodes and have technology types

This enricher automatically identifies membership columns in a target DataFrame and adds all properties from the corresponding model DataFrames. This enables comprehensive analysis by combining object properties with their relationships.

Key Features: - Automatic identification of membership columns using MESQUAL's flag index system - Support for multiple simultaneous memberships (node, fuel_type, company, etc.) - Preservation of NaN memberships in enriched data - Configurable property naming to avoid column conflicts - Integration with MESQUAL Dataset architecture

Parameters:

Name Type Description Default
membership_tag_separator str

Separator used between membership name and property when PREFIX or SUFFIX tagging is applied

 '_'

Examples:

>>> enricher = MembershipPropertyEnricher()
>>> # Generator DataFrame with 'node' column linking to node objects
>>> enriched_gen_df = enricher.append_properties(
...     generator_df, dataset, MembershipTagging.PREFIX
... )
>>> # enriched_gen_df now includes 'node_voltage', 'node_area' columns from node properties
Source code in submodules/mesqual/mesqual/energy_data_handling/model_handling/membership_property_enrichers.py 
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
class MembershipPropertyEnricher:
    """
    Enriches energy system DataFrames with properties from related model objects.

    In energy system modeling, entities often have membership relationships to other
    model components. For example:
    - Generators belong to nodes and have fuel types
    - Lines connect between nodes
    - Storage units are located at nodes and have technology types

    This enricher automatically identifies membership columns in a target DataFrame
    and adds all properties from the corresponding model DataFrames. This enables
    comprehensive analysis by combining object properties with their relationships.

    Key Features:
    - Automatic identification of membership columns using MESQUAL's flag index system
    - Support for multiple simultaneous memberships (node, fuel_type, company, etc.)
    - Preservation of NaN memberships in enriched data
    - Configurable property naming to avoid column conflicts
    - Integration with MESQUAL Dataset architecture

    Args:
        membership_tag_separator: Separator used between membership name and property
                                  when PREFIX or SUFFIX tagging is applied

    Examples:
        >>> enricher = MembershipPropertyEnricher()
        >>> # Generator DataFrame with 'node' column linking to node objects
        >>> enriched_gen_df = enricher.append_properties(
        ...     generator_df, dataset, MembershipTagging.PREFIX
        ... )
        >>> # enriched_gen_df now includes 'node_voltage', 'node_area' columns from node properties
    """
    def __init__(self, membership_tag_separator: str = '_'):
        """
        Initialize the membership property enricher.

        Args:
            membership_tag_separator: Character(s) used to separate membership names
                                     from property names in PREFIX/SUFFIX modes
        """
        self._membership_tag_separator = membership_tag_separator

    def identify_membership_columns(self, column_names: list[str], dataset: Dataset) -> list[str]:
        """
        Identifies columns that represent memberships to other model objects.

        Uses MESQUAL's flag index system to determine which columns in the target
        DataFrame represent relationships to other model components. This enables
        automatic discovery of enrichment opportunities without manual specification.

        Args:
            column_names: List of column names from the target DataFrame
            dataset: MESQUAL Dataset containing model definitions and flag mappings

        Returns:
            List of column names that represent memberships to other model objects

        Examples:
            For a generator DataFrame with columns ['name', 'capacity', 'node', 'fuel_type']:
            >>> membership_cols = enricher.identify_membership_columns(
            ...     generator_df.columns, dataset
            ... )
            >>> print(membership_cols)  # ['node', 'fuel_type']
        """
        return [
            col for col in column_names
            if dataset.flag_index.column_name_in_model_describes_membership(col)
        ]

    def append_properties(
            self,
            target_df: pd.DataFrame,
            dataset: Dataset,
            membership_tagging: MembershipTagging = MembershipTagging.NONE
    ) -> pd.DataFrame:
        """
        Enriches target DataFrame with properties from all linked model objects.

        Performs comprehensive enrichment by automatically identifying all membership
        relationships and adding corresponding properties. This is the primary method
        for energy system DataFrame enrichment, enabling complex multi-dimensional
        analysis by combining object properties with their relationships.

        The method preserves all original data while adding new property columns.
        Missing relationships (NaN memberships) are handled gracefully by preserving
        NaN values in the enriched properties.

        Args:
            target_df: DataFrame to enrich (e.g., generator, line, storage data)
            dataset: MESQUAL Dataset containing linked model DataFrames with properties
            membership_tagging: Strategy for naming enriched properties to avoid conflicts

        Returns:
            Enhanced DataFrame with all properties from linked model objects added.
            Original columns are preserved, new columns added based on memberships.

        Raises:
            Warning: Logged when membership objects are missing from source DataFrames

        Examples:
            Energy system use cases:

            >>> # Enrich generator data with node and fuel properties
            >>> enriched_generators = enricher.append_properties(
            ...     generators_df, dataset, MembershipTagging.PREFIX
            ... )
            >>> # Result: original columns + 'node_voltage', 'node_area', 'fuel_co2_rate', etc.

            >>> # Enrich transmission data with node characteristics
            >>> enriched_lines = enricher.append_properties(
            ...     transmission_df, dataset, MembershipTagging.SUFFIX
            ... )
            >>> # Result: line properties + node properties with '_node' suffix
        """
        membership_columns = self.identify_membership_columns(target_df.columns, dataset)
        result_df = target_df.copy()

        for column in membership_columns:
            result_df = self.append_single_membership_properties(
                result_df,
                dataset,
                column,
                membership_tagging
            )

        return result_df

    def append_single_membership_properties(
            self,
            target_df: pd.DataFrame,
            dataset: Dataset,
            membership_column: str,
            membership_tagging: MembershipTagging = MembershipTagging.NONE
    ) -> pd.DataFrame:
        """
        Enriches target DataFrame with properties from a specific membership relationship.

        This method provides fine-grained control over property enrichment by handling
        a single membership column. Useful when custom logic is needed for specific
        relationships or when processing memberships sequentially with different
        tagging strategies.

        The method uses MESQUAL's flag index to determine the source model DataFrame
        for the membership column, then performs a left join to preserve all target
        records while adding available properties.

        Args:
            target_df: DataFrame to enrich (must contain the membership column)
            dataset: MESQUAL Dataset with access to linked model DataFrames
            membership_column: Name of column containing object references (e.g., 'node', 'fuel_type')
            membership_tagging: Strategy for naming enriched properties

        Returns:
            DataFrame with properties from the linked model objects added.
            All original rows preserved; NaN memberships result in NaN properties.

        Raises:
            Warning: Logged when referenced objects are missing from the source DataFrame

        Examples:
            Targeted enrichment scenarios:

            >>> # Add only node properties to generators
            >>> gen_with_nodes = enricher.append_single_membership_properties(
            ...     generators_df, dataset, 'node', MembershipTagging.PREFIX
            ... )
            >>> # Result: generators + 'node_voltage', 'node_area', etc.

            >>> # Sequential enrichment with different tagging
            >>> result = generators_df.copy()
            >>> result = enricher.append_single_membership_properties(
            ...     result, dataset, 'node', MembershipTagging.PREFIX
            ... )
            >>> result = enricher.append_single_membership_properties(
            ...     result, dataset, 'fuel_type', MembershipTagging.SUFFIX
            ... )
        """
        source_flag = dataset.flag_index.get_linked_model_flag_for_membership_column(membership_column)
        source_df = dataset.fetch(source_flag)

        membership_objects = target_df[membership_column].dropna().unique()
        missing_objects = set(membership_objects) - set(source_df.index)

        if missing_objects:
            self._log_missing_objects_warning(missing_objects, membership_column)

        source_properties = source_df.copy()

        match membership_tagging:
            case MembershipTagging.PREFIX:
                source_properties = source_properties.add_prefix(f"{membership_column}{self._membership_tag_separator}")
            case MembershipTagging.SUFFIX:
                source_properties = source_properties.add_suffix(f"{self._membership_tag_separator}{membership_column}")

        result_df = target_df.merge(
            source_properties,
            left_on=membership_column,
            right_index=True,
            how="left"
        )

        return result_df

    def _log_missing_objects_warning(
            self,
            missing_objects: set,
            membership_column: str,
            max_show: int = 5
    ):
        """
        Logs warning about missing objects in the source DataFrame.

        In energy system modeling, missing references can indicate data quality
        issues, model inconsistencies, or incomplete datasets. This method provides
        informative warnings to help identify and resolve such issues.

        Args:
            missing_objects: Set of object identifiers missing from source DataFrame
            membership_column: Name of the membership column being processed
            max_show: Maximum number of missing objects to display in the warning
        """
        num_missing = len(missing_objects)
        warning_suffix = ", and more" if num_missing > max_show else ""
        logger.warning(
            f"{num_missing} objects missing in source dataframe for {membership_column}: "
            f"{list(missing_objects)[:max_show]}{warning_suffix}."
        )

__init__ 

__init__(membership_tag_separator: str = '_')

Initialize the membership property enricher.

Parameters:

Name Type Description Default
membership_tag_separator str

Character(s) used to separate membership names from property names in PREFIX/SUFFIX modes

 '_'
Source code in submodules/mesqual/mesqual/energy_data_handling/model_handling/membership_property_enrichers.py 
77
78
79
80
81
82
83
84
85
def __init__(self, membership_tag_separator: str = '_'):
    """
    Initialize the membership property enricher.

    Args:
        membership_tag_separator: Character(s) used to separate membership names
                                 from property names in PREFIX/SUFFIX modes
    """
    self._membership_tag_separator = membership_tag_separator

identify_membership_columns 

identify_membership_columns(column_names: list[str], dataset: Dataset) -> list[str]

Identifies columns that represent memberships to other model objects.

Uses MESQUAL's flag index system to determine which columns in the target DataFrame represent relationships to other model components. This enables automatic discovery of enrichment opportunities without manual specification.

Parameters:

Name Type Description Default
column_names list[str]

List of column names from the target DataFrame

 required
dataset Dataset

MESQUAL Dataset containing model definitions and flag mappings

 required

Returns:

Type Description
list[str]

List of column names that represent memberships to other model objects

Examples:

For a generator DataFrame with columns ['name', 'capacity', 'node', 'fuel_type']:

>>> membership_cols = enricher.identify_membership_columns(
...     generator_df.columns, dataset
... )
>>> print(membership_cols)  # ['node', 'fuel_type']
Source code in submodules/mesqual/mesqual/energy_data_handling/model_handling/membership_property_enrichers.py 
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
def identify_membership_columns(self, column_names: list[str], dataset: Dataset) -> list[str]:
    """
    Identifies columns that represent memberships to other model objects.

    Uses MESQUAL's flag index system to determine which columns in the target
    DataFrame represent relationships to other model components. This enables
    automatic discovery of enrichment opportunities without manual specification.

    Args:
        column_names: List of column names from the target DataFrame
        dataset: MESQUAL Dataset containing model definitions and flag mappings

    Returns:
        List of column names that represent memberships to other model objects

    Examples:
        For a generator DataFrame with columns ['name', 'capacity', 'node', 'fuel_type']:
        >>> membership_cols = enricher.identify_membership_columns(
        ...     generator_df.columns, dataset
        ... )
        >>> print(membership_cols)  # ['node', 'fuel_type']
    """
    return [
        col for col in column_names
        if dataset.flag_index.column_name_in_model_describes_membership(col)
    ]

append_properties 

append_properties(target_df: DataFrame, dataset: Dataset, membership_tagging: MembershipTagging = NONE) -> DataFrame

Enriches target DataFrame with properties from all linked model objects.

Performs comprehensive enrichment by automatically identifying all membership relationships and adding corresponding properties. This is the primary method for energy system DataFrame enrichment, enabling complex multi-dimensional analysis by combining object properties with their relationships.

The method preserves all original data while adding new property columns. Missing relationships (NaN memberships) are handled gracefully by preserving NaN values in the enriched properties.

Parameters:

Name Type Description Default
target_df DataFrame

DataFrame to enrich (e.g., generator, line, storage data)

 required
dataset Dataset

MESQUAL Dataset containing linked model DataFrames with properties

 required
membership_tagging MembershipTagging

Strategy for naming enriched properties to avoid conflicts

 NONE

Returns:

Type Description
DataFrame

Enhanced DataFrame with all properties from linked model objects added.
DataFrame

Original columns are preserved, new columns added based on memberships.

Raises:

Type Description
Warning

Logged when membership objects are missing from source DataFrames

Examples:

Energy system use cases:

>>> # Enrich generator data with node and fuel properties
>>> enriched_generators = enricher.append_properties(
...     generators_df, dataset, MembershipTagging.PREFIX
... )
>>> # Result: original columns + 'node_voltage', 'node_area', 'fuel_co2_rate', etc.

>>> # Enrich transmission data with node characteristics
>>> enriched_lines = enricher.append_properties(
...     transmission_df, dataset, MembershipTagging.SUFFIX
... )
>>> # Result: line properties + node properties with '_node' suffix
Source code in submodules/mesqual/mesqual/energy_data_handling/model_handling/membership_property_enrichers.py 
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
def append_properties(
        self,
        target_df: pd.DataFrame,
        dataset: Dataset,
        membership_tagging: MembershipTagging = MembershipTagging.NONE
) -> pd.DataFrame:
    """
    Enriches target DataFrame with properties from all linked model objects.

    Performs comprehensive enrichment by automatically identifying all membership
    relationships and adding corresponding properties. This is the primary method
    for energy system DataFrame enrichment, enabling complex multi-dimensional
    analysis by combining object properties with their relationships.

    The method preserves all original data while adding new property columns.
    Missing relationships (NaN memberships) are handled gracefully by preserving
    NaN values in the enriched properties.

    Args:
        target_df: DataFrame to enrich (e.g., generator, line, storage data)
        dataset: MESQUAL Dataset containing linked model DataFrames with properties
        membership_tagging: Strategy for naming enriched properties to avoid conflicts

    Returns:
        Enhanced DataFrame with all properties from linked model objects added.
        Original columns are preserved, new columns added based on memberships.

    Raises:
        Warning: Logged when membership objects are missing from source DataFrames

    Examples:
        Energy system use cases:

        >>> # Enrich generator data with node and fuel properties
        >>> enriched_generators = enricher.append_properties(
        ...     generators_df, dataset, MembershipTagging.PREFIX
        ... )
        >>> # Result: original columns + 'node_voltage', 'node_area', 'fuel_co2_rate', etc.

        >>> # Enrich transmission data with node characteristics
        >>> enriched_lines = enricher.append_properties(
        ...     transmission_df, dataset, MembershipTagging.SUFFIX
        ... )
        >>> # Result: line properties + node properties with '_node' suffix
    """
    membership_columns = self.identify_membership_columns(target_df.columns, dataset)
    result_df = target_df.copy()

    for column in membership_columns:
        result_df = self.append_single_membership_properties(
            result_df,
            dataset,
            column,
            membership_tagging
        )

    return result_df

append_single_membership_properties 

append_single_membership_properties(target_df: DataFrame, dataset: Dataset, membership_column: str, membership_tagging: MembershipTagging = NONE) -> DataFrame

Enriches target DataFrame with properties from a specific membership relationship.

This method provides fine-grained control over property enrichment by handling a single membership column. Useful when custom logic is needed for specific relationships or when processing memberships sequentially with different tagging strategies.

The method uses MESQUAL's flag index to determine the source model DataFrame for the membership column, then performs a left join to preserve all target records while adding available properties.

Parameters:

Name Type Description Default
target_df DataFrame

DataFrame to enrich (must contain the membership column)

 required
dataset Dataset

MESQUAL Dataset with access to linked model DataFrames

 required
membership_column str

Name of column containing object references (e.g., 'node', 'fuel_type')

 required
membership_tagging MembershipTagging

Strategy for naming enriched properties

 NONE

Returns:

Type Description
DataFrame

DataFrame with properties from the linked model objects added.
DataFrame

All original rows preserved; NaN memberships result in NaN properties.

Raises:

Type Description
Warning

Logged when referenced objects are missing from the source DataFrame

Examples:

Targeted enrichment scenarios:

>>> # Add only node properties to generators
>>> gen_with_nodes = enricher.append_single_membership_properties(
...     generators_df, dataset, 'node', MembershipTagging.PREFIX
... )
>>> # Result: generators + 'node_voltage', 'node_area', etc.

>>> # Sequential enrichment with different tagging
>>> result = generators_df.copy()
>>> result = enricher.append_single_membership_properties(
...     result, dataset, 'node', MembershipTagging.PREFIX
... )
>>> result = enricher.append_single_membership_properties(
...     result, dataset, 'fuel_type', MembershipTagging.SUFFIX
... )
Source code in submodules/mesqual/mesqual/energy_data_handling/model_handling/membership_property_enrichers.py 
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
def append_single_membership_properties(
        self,
        target_df: pd.DataFrame,
        dataset: Dataset,
        membership_column: str,
        membership_tagging: MembershipTagging = MembershipTagging.NONE
) -> pd.DataFrame:
    """
    Enriches target DataFrame with properties from a specific membership relationship.

    This method provides fine-grained control over property enrichment by handling
    a single membership column. Useful when custom logic is needed for specific
    relationships or when processing memberships sequentially with different
    tagging strategies.

    The method uses MESQUAL's flag index to determine the source model DataFrame
    for the membership column, then performs a left join to preserve all target
    records while adding available properties.

    Args:
        target_df: DataFrame to enrich (must contain the membership column)
        dataset: MESQUAL Dataset with access to linked model DataFrames
        membership_column: Name of column containing object references (e.g., 'node', 'fuel_type')
        membership_tagging: Strategy for naming enriched properties

    Returns:
        DataFrame with properties from the linked model objects added.
        All original rows preserved; NaN memberships result in NaN properties.

    Raises:
        Warning: Logged when referenced objects are missing from the source DataFrame

    Examples:
        Targeted enrichment scenarios:

        >>> # Add only node properties to generators
        >>> gen_with_nodes = enricher.append_single_membership_properties(
        ...     generators_df, dataset, 'node', MembershipTagging.PREFIX
        ... )
        >>> # Result: generators + 'node_voltage', 'node_area', etc.

        >>> # Sequential enrichment with different tagging
        >>> result = generators_df.copy()
        >>> result = enricher.append_single_membership_properties(
        ...     result, dataset, 'node', MembershipTagging.PREFIX
        ... )
        >>> result = enricher.append_single_membership_properties(
        ...     result, dataset, 'fuel_type', MembershipTagging.SUFFIX
        ... )
    """
    source_flag = dataset.flag_index.get_linked_model_flag_for_membership_column(membership_column)
    source_df = dataset.fetch(source_flag)

    membership_objects = target_df[membership_column].dropna().unique()
    missing_objects = set(membership_objects) - set(source_df.index)

    if missing_objects:
        self._log_missing_objects_warning(missing_objects, membership_column)

    source_properties = source_df.copy()

    match membership_tagging:
        case MembershipTagging.PREFIX:
            source_properties = source_properties.add_prefix(f"{membership_column}{self._membership_tag_separator}")
        case MembershipTagging.SUFFIX:
            source_properties = source_properties.add_suffix(f"{self._membership_tag_separator}{membership_column}")

    result_df = target_df.merge(
        source_properties,
        left_on=membership_column,
        right_index=True,
        how="left"
    )

    return result_df

DirectionalMembershipPropertyEnricher

Enriches energy system DataFrames with properties for directional relationships.

Energy networks inherently contain directional relationships - transmission lines connect from one node to another, flows have origins and destinations, and trade occurs between regions. This enricher handles such bidirectional memberships by identifying from/to column pairs and enriching with appropriate directional tags.

Common energy system applications: - Transmission lines: 'node_from' and 'node_to' linking to node properties - Inter-regional flows: 'region_from' and 'region_to' for trade analysis - Pipeline systems: 'hub_from' and 'hub_to' for gas network modeling - Market connections: 'market_from' and 'market_to' for price analysis

The enricher automatically identifies directional column pairs using configurable identifiers (default: '_from' and '_to') and adds properties from the linked model objects with appropriate directional suffixes.

Key Features
  • Automatic identification of from/to column pairs
  • Flexible directional identifiers (customizable beyond '_from'/'_to')
  • Support for multiple directional relationships in one DataFrame
  • Preservation of NaN relationships
  • Integration with MESQUAL's model flag system

Parameters:

Name Type Description Default
from_identifier str

Suffix/prefix identifying 'from' direction (default: '_from')

 '_from'
to_identifier str

Suffix/prefix identifying 'to' direction (default: '_to')

 '_to'
membership_tag_separator str

Separator for property name construction

 '_'

Examples:

>>> enricher = DirectionalMembershipPropertyEnricher()
>>> # Line DataFrame with 'node_from', 'node_to' columns
>>> enriched_lines = enricher.append_properties(
...     line_df, dataset, MembershipTagging.NONE
... )
>>> # Result includes node properties with '_from' and '_to' suffixes
Source code in submodules/mesqual/mesqual/energy_data_handling/model_handling/membership_property_enrichers.py 
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
class DirectionalMembershipPropertyEnricher:
    """
    Enriches energy system DataFrames with properties for directional relationships.

    Energy networks inherently contain directional relationships - transmission lines
    connect from one node to another, flows have origins and destinations, and trade
    occurs between regions. This enricher handles such bidirectional memberships by
    identifying from/to column pairs and enriching with appropriate directional tags.

    Common energy system applications:
    - Transmission lines: 'node_from' and 'node_to' linking to node properties
    - Inter-regional flows: 'region_from' and 'region_to' for trade analysis
    - Pipeline systems: 'hub_from' and 'hub_to' for gas network modeling
    - Market connections: 'market_from' and 'market_to' for price analysis

    The enricher automatically identifies directional column pairs using configurable
    identifiers (default: '_from' and '_to') and adds properties from the linked
    model objects with appropriate directional suffixes.

    Key Features:
        - Automatic identification of from/to column pairs
        - Flexible directional identifiers (customizable beyond '_from'/'_to')
        - Support for multiple directional relationships in one DataFrame
        - Preservation of NaN relationships
        - Integration with MESQUAL's model flag system

    Args:
        from_identifier: Suffix/prefix identifying 'from' direction (default: '_from')
        to_identifier: Suffix/prefix identifying 'to' direction (default: '_to')
        membership_tag_separator: Separator for property name construction

    Examples:

        >>> enricher = DirectionalMembershipPropertyEnricher()
        >>> # Line DataFrame with 'node_from', 'node_to' columns
        >>> enriched_lines = enricher.append_properties(
        ...     line_df, dataset, MembershipTagging.NONE
        ... )
        >>> # Result includes node properties with '_from' and '_to' suffixes
    """
    def __init__(
            self,
            from_identifier: str = "_from",
            to_identifier: str = "_to",
            membership_tag_separator: str = '_',
    ):
        """
        Initialize the directional membership property enricher.

        Args:
            from_identifier: String identifying source/origin columns (e.g., '_from', 'source_')
            to_identifier: String identifying destination/target columns (e.g., '_to', 'dest_')
            membership_tag_separator: Character(s) separating membership names from properties
        """
        self._from_identifier = from_identifier
        self._to_identifier = to_identifier
        self._membership_tag_separator = membership_tag_separator
        self._tag_finder = CommonBaseKeyFinder(from_identifier, to_identifier)

    def identify_from_to_columns(self, column_names: list[str], dataset: Dataset) -> list[str]:
        """
        Identifies base names for directional membership column pairs.

        Analyzes column names to find base membership types that have both 'from'
        and 'to' variants. For example, identifies 'node' as a base when both
        'node_from' and 'node_to' columns exist and represent valid memberships.

        Args:
            column_names: List of column names from the target DataFrame
            dataset: MESQUAL Dataset for membership validation

        Returns:
            List of base column names that have both from/to variants

        Examples:
            For a line DataFrame with ['name', 'capacity', 'node_from', 'node_to']:
            >>> base_columns = enricher.identify_from_to_columns(
            ...     line_df.columns, dataset
            ... )
            >>> print(base_columns)  # ['node']

            For inter-regional trade with ['flow', 'region_from', 'region_to', 'market_from']:
            >>> base_columns = enricher.identify_from_to_columns(
            ...     trade_df.columns, dataset  
            ... )
            >>> print(base_columns)  # ['region'] (market missing 'market_to')
        """
        potential_columns = self._tag_finder.get_keys_for_which_all_association_tags_appear(column_names)
        return [
            col for col in potential_columns
            if dataset.flag_index.column_name_in_model_describes_membership(col)
        ]

    def append_properties(
            self,
            target_df: pd.DataFrame,
            dataset: Dataset,
            membership_tagging: MembershipTagging = MembershipTagging.NONE
    ) -> pd.DataFrame:
        """
        Enriches DataFrame with properties from all directional relationships.

        Performs comprehensive directional enrichment by identifying all from/to
        column pairs and adding properties from both directions. Essential for
        network analysis where understanding characteristics of connected nodes,
        regions, or components is crucial for energy system modeling.

        Each directional relationship results in two sets of enriched properties:
        one for the 'from' direction and one for the 'to' direction, clearly
        distinguished by directional suffixes.

        Args:
            target_df: DataFrame with directional relationships (e.g., transmission lines)
            dataset: MESQUAL Dataset containing model objects and their properties
            membership_tagging: Strategy for property naming (applied before directional tags)

        Returns:
            Enhanced DataFrame with directional properties added. Original data preserved,
            new columns follow pattern: [prefix_]property_name[_suffix]_direction

        Raises:
            Warning: Logged when referenced objects missing from source DataFrames

        Examples:
            Network transmission analysis:

            >>> # Transmission lines with node endpoints
            >>> enriched_lines = enricher.append_properties(
            ...     transmission_df, dataset, MembershipTagging.NONE
            ... )
            >>> # Result: original columns + 'voltage_from', 'voltage_to', 
            >>> #         'area_from', 'area_to', etc.

            >>> # Inter-regional trade flows
            >>> enriched_trade = enricher.append_properties(
            ...     trade_df, dataset, MembershipTagging.PREFIX
            ... )
            >>> # Result: trade data + 'region_gdp_from', 'region_gdp_to', etc.
        """
        membership_base_columns = self.identify_from_to_columns(target_df.columns, dataset)
        result_df = target_df.copy()

        for base_column in membership_base_columns:
            result_df = self.append_directional_properties(
                result_df,
                dataset,
                base_column,
                membership_tagging
            )

        return result_df

    def append_directional_properties(
            self,
            target_df: pd.DataFrame,
            dataset: Dataset,
            base_column: str,
            membership_tagging: MembershipTagging = MembershipTagging.NONE
    ) -> pd.DataFrame:
        source_flag = dataset.flag_index.get_linked_model_flag_for_membership_column(base_column)
        source_df = dataset.fetch(source_flag)
        return self.append_directional_properties_in_source_to_target_df(
            target_df,
            source_df,
            base_column,
            membership_tagging,
        )

    def append_directional_properties_in_source_to_target_df(
            self,
            target_df: pd.DataFrame,
            source_df: pd.DataFrame,
            base_column: str,
            membership_tagging: MembershipTagging = MembershipTagging.NONE
    ) -> pd.DataFrame:
        """
        Enriches DataFrame with properties from a directional relationship.

        Handles a single from/to membership pair by adding the corresponding model
        DataFrame's properties with directional tags.

        The method processes both directions (from/to) for the specified base column,
        adding properties with appropriate directional suffixes. Missing references
        are handled gracefully with NaN preservation.

        Args:
            target_df: DataFrame containing the directional columns
            source_df: DataFrame containing the properties
            base_column: Base membership name (e.g., 'node' for 'node_from'/'node_to')
            membership_tagging: Property naming strategy (applied before directional tags)

        Returns:
            DataFrame with directional properties added for the specified relationship.
            Properties follow naming pattern: [prefix_]property[_suffix]_direction

        Raises:
            Warning: Logged when referenced objects are missing from source DataFrame

        Examples:
            Targeted directional enrichment:

            >>> # Add only node properties to transmission lines
            >>> lines_with_nodes = enricher.append_directional_properties(
            ...     line_df, node_df, 'node', MembershipTagging.NONE
            ... )
            >>> # Result: lines + 'voltage_from', 'voltage_to', 'area_from', 'area_to'
        """
        result_df = target_df.copy()
        for tag in [self._from_identifier, self._to_identifier]:
            membership_column = self._get_full_column_name(base_column, tag, target_df.columns)

            if membership_column not in target_df.columns:
                continue

            membership_objects = target_df[membership_column].dropna().unique()
            missing_objects = set(membership_objects) - set(source_df.index)

            if missing_objects:
                self._log_missing_objects_warning(missing_objects, membership_column)

            source_properties = source_df.copy()

            match membership_tagging:
                case MembershipTagging.PREFIX:
                    source_properties = source_properties.add_prefix(f"{base_column}{self._membership_tag_separator}")
                case MembershipTagging.SUFFIX:
                    source_properties = source_properties.add_suffix(f"{self._membership_tag_separator}{base_column}")

            source_properties = source_properties.add_suffix(tag)

            result_df = result_df.merge(
                source_properties,
                left_on=membership_column,
                right_index=True,
                how="left"
            )
        return result_df

    def _get_full_column_name(self, base_column: str, tag: str, df_columns: list[str]) -> str:
        """
        Determines the actual column name for a directional membership.

        Handles flexibility in directional column naming by testing both
        suffix and prefix patterns. Supports various naming conventions
        used across different energy modeling platforms.

        Args:
            base_column: Base membership name (e.g., 'node')
            tag: Directional identifier (e.g., '_from', '_to')
            df_columns: List of actual column names in the DataFrame

        Returns:
            Actual column name found in the DataFrame

        Examples:
            >>> # For base_column='node', tag='_from'
            >>> # Tests 'node_from' first, then 'from_node'
            >>> name = enricher._get_full_column_name('node', '_from', df.columns)
        """
        test_suffix = f"{base_column}{tag}"
        return test_suffix if test_suffix in df_columns else f"{tag}{base_column}"

    def _log_missing_objects_warning(
            self,
            missing_objects: set,
            membership_column: str,
            max_show: int = 5
    ):
        num_missing = len(missing_objects)
        warning_suffix = ", and more" if num_missing > max_show else ""
        logger.warning(
            f"{num_missing} objects missing in source dataframe for {membership_column}: "
            f"{list(missing_objects)[:max_show]}{warning_suffix}."
        )

__init__ 

__init__(from_identifier: str = '_from', to_identifier: str = '_to', membership_tag_separator: str = '_')

Initialize the directional membership property enricher.

Parameters:

Name Type Description Default
from_identifier str

String identifying source/origin columns (e.g., 'from', 'source')

 '_from'
to_identifier str

String identifying destination/target columns (e.g., 'to', 'dest')

 '_to'
membership_tag_separator str

Character(s) separating membership names from properties

 '_'
Source code in submodules/mesqual/mesqual/energy_data_handling/model_handling/membership_property_enrichers.py 
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
def __init__(
        self,
        from_identifier: str = "_from",
        to_identifier: str = "_to",
        membership_tag_separator: str = '_',
):
    """
    Initialize the directional membership property enricher.

    Args:
        from_identifier: String identifying source/origin columns (e.g., '_from', 'source_')
        to_identifier: String identifying destination/target columns (e.g., '_to', 'dest_')
        membership_tag_separator: Character(s) separating membership names from properties
    """
    self._from_identifier = from_identifier
    self._to_identifier = to_identifier
    self._membership_tag_separator = membership_tag_separator
    self._tag_finder = CommonBaseKeyFinder(from_identifier, to_identifier)

identify_from_to_columns 

identify_from_to_columns(column_names: list[str], dataset: Dataset) -> list[str]

Identifies base names for directional membership column pairs.

Analyzes column names to find base membership types that have both 'from' and 'to' variants. For example, identifies 'node' as a base when both 'node_from' and 'node_to' columns exist and represent valid memberships.

Parameters:

Name Type Description Default
column_names list[str]

List of column names from the target DataFrame

 required
dataset Dataset

MESQUAL Dataset for membership validation

 required

Returns:

Type Description
list[str]

List of base column names that have both from/to variants

Examples:

For a line DataFrame with ['name', 'capacity', 'node_from', 'node_to']:

>>> base_columns = enricher.identify_from_to_columns(
...     line_df.columns, dataset
... )
>>> print(base_columns)  # ['node']

For inter-regional trade with ['flow', 'region_from', 'region_to', 'market_from']:

>>> base_columns = enricher.identify_from_to_columns(
...     trade_df.columns, dataset  
... )
>>> print(base_columns)  # ['region'] (market missing 'market_to')
Source code in submodules/mesqual/mesqual/energy_data_handling/model_handling/membership_property_enrichers.py 
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
def identify_from_to_columns(self, column_names: list[str], dataset: Dataset) -> list[str]:
    """
    Identifies base names for directional membership column pairs.

    Analyzes column names to find base membership types that have both 'from'
    and 'to' variants. For example, identifies 'node' as a base when both
    'node_from' and 'node_to' columns exist and represent valid memberships.

    Args:
        column_names: List of column names from the target DataFrame
        dataset: MESQUAL Dataset for membership validation

    Returns:
        List of base column names that have both from/to variants

    Examples:
        For a line DataFrame with ['name', 'capacity', 'node_from', 'node_to']:
        >>> base_columns = enricher.identify_from_to_columns(
        ...     line_df.columns, dataset
        ... )
        >>> print(base_columns)  # ['node']

        For inter-regional trade with ['flow', 'region_from', 'region_to', 'market_from']:
        >>> base_columns = enricher.identify_from_to_columns(
        ...     trade_df.columns, dataset  
        ... )
        >>> print(base_columns)  # ['region'] (market missing 'market_to')
    """
    potential_columns = self._tag_finder.get_keys_for_which_all_association_tags_appear(column_names)
    return [
        col for col in potential_columns
        if dataset.flag_index.column_name_in_model_describes_membership(col)
    ]

append_properties 

append_properties(target_df: DataFrame, dataset: Dataset, membership_tagging: MembershipTagging = NONE) -> DataFrame

Enriches DataFrame with properties from all directional relationships.

Performs comprehensive directional enrichment by identifying all from/to column pairs and adding properties from both directions. Essential for network analysis where understanding characteristics of connected nodes, regions, or components is crucial for energy system modeling.

Each directional relationship results in two sets of enriched properties: one for the 'from' direction and one for the 'to' direction, clearly distinguished by directional suffixes.

Parameters:

Name Type Description Default
target_df DataFrame

DataFrame with directional relationships (e.g., transmission lines)

 required
dataset Dataset

MESQUAL Dataset containing model objects and their properties

 required
membership_tagging MembershipTagging

Strategy for property naming (applied before directional tags)

 NONE

Returns:

Type Description
DataFrame

Enhanced DataFrame with directional properties added. Original data preserved,
DataFrame

new columns follow pattern: [prefix_]property_name[_suffix]_direction

Raises:

Type Description
Warning

Logged when referenced objects missing from source DataFrames

Examples:

Network transmission analysis:

>>> # Transmission lines with node endpoints
>>> enriched_lines = enricher.append_properties(
...     transmission_df, dataset, MembershipTagging.NONE
... )
>>> # Result: original columns + 'voltage_from', 'voltage_to', 
>>> #         'area_from', 'area_to', etc.

>>> # Inter-regional trade flows
>>> enriched_trade = enricher.append_properties(
...     trade_df, dataset, MembershipTagging.PREFIX
... )
>>> # Result: trade data + 'region_gdp_from', 'region_gdp_to', etc.
Source code in submodules/mesqual/mesqual/energy_data_handling/model_handling/membership_property_enrichers.py 
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
def append_properties(
        self,
        target_df: pd.DataFrame,
        dataset: Dataset,
        membership_tagging: MembershipTagging = MembershipTagging.NONE
) -> pd.DataFrame:
    """
    Enriches DataFrame with properties from all directional relationships.

    Performs comprehensive directional enrichment by identifying all from/to
    column pairs and adding properties from both directions. Essential for
    network analysis where understanding characteristics of connected nodes,
    regions, or components is crucial for energy system modeling.

    Each directional relationship results in two sets of enriched properties:
    one for the 'from' direction and one for the 'to' direction, clearly
    distinguished by directional suffixes.

    Args:
        target_df: DataFrame with directional relationships (e.g., transmission lines)
        dataset: MESQUAL Dataset containing model objects and their properties
        membership_tagging: Strategy for property naming (applied before directional tags)

    Returns:
        Enhanced DataFrame with directional properties added. Original data preserved,
        new columns follow pattern: [prefix_]property_name[_suffix]_direction

    Raises:
        Warning: Logged when referenced objects missing from source DataFrames

    Examples:
        Network transmission analysis:

        >>> # Transmission lines with node endpoints
        >>> enriched_lines = enricher.append_properties(
        ...     transmission_df, dataset, MembershipTagging.NONE
        ... )
        >>> # Result: original columns + 'voltage_from', 'voltage_to', 
        >>> #         'area_from', 'area_to', etc.

        >>> # Inter-regional trade flows
        >>> enriched_trade = enricher.append_properties(
        ...     trade_df, dataset, MembershipTagging.PREFIX
        ... )
        >>> # Result: trade data + 'region_gdp_from', 'region_gdp_to', etc.
    """
    membership_base_columns = self.identify_from_to_columns(target_df.columns, dataset)
    result_df = target_df.copy()

    for base_column in membership_base_columns:
        result_df = self.append_directional_properties(
            result_df,
            dataset,
            base_column,
            membership_tagging
        )

    return result_df

append_directional_properties_in_source_to_target_df 

append_directional_properties_in_source_to_target_df(target_df: DataFrame, source_df: DataFrame, base_column: str, membership_tagging: MembershipTagging = NONE) -> DataFrame

Enriches DataFrame with properties from a directional relationship.

Handles a single from/to membership pair by adding the corresponding model DataFrame's properties with directional tags.

The method processes both directions (from/to) for the specified base column, adding properties with appropriate directional suffixes. Missing references are handled gracefully with NaN preservation.

Parameters:

Name Type Description Default
target_df DataFrame

DataFrame containing the directional columns

 required
source_df DataFrame

DataFrame containing the properties

 required
base_column str

Base membership name (e.g., 'node' for 'node_from'/'node_to')

 required
membership_tagging MembershipTagging

Property naming strategy (applied before directional tags)

 NONE

Returns:

Type Description
DataFrame

DataFrame with directional properties added for the specified relationship.
DataFrame

Properties follow naming pattern: [prefix_]property[_suffix]_direction

Raises:

Type Description
Warning

Logged when referenced objects are missing from source DataFrame

Examples:

Targeted directional enrichment:

>>> # Add only node properties to transmission lines
>>> lines_with_nodes = enricher.append_directional_properties(
...     line_df, node_df, 'node', MembershipTagging.NONE
... )
>>> # Result: lines + 'voltage_from', 'voltage_to', 'area_from', 'area_to'
Source code in submodules/mesqual/mesqual/energy_data_handling/model_handling/membership_property_enrichers.py 
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
def append_directional_properties_in_source_to_target_df(
        self,
        target_df: pd.DataFrame,
        source_df: pd.DataFrame,
        base_column: str,
        membership_tagging: MembershipTagging = MembershipTagging.NONE
) -> pd.DataFrame:
    """
    Enriches DataFrame with properties from a directional relationship.

    Handles a single from/to membership pair by adding the corresponding model
    DataFrame's properties with directional tags.

    The method processes both directions (from/to) for the specified base column,
    adding properties with appropriate directional suffixes. Missing references
    are handled gracefully with NaN preservation.

    Args:
        target_df: DataFrame containing the directional columns
        source_df: DataFrame containing the properties
        base_column: Base membership name (e.g., 'node' for 'node_from'/'node_to')
        membership_tagging: Property naming strategy (applied before directional tags)

    Returns:
        DataFrame with directional properties added for the specified relationship.
        Properties follow naming pattern: [prefix_]property[_suffix]_direction

    Raises:
        Warning: Logged when referenced objects are missing from source DataFrame

    Examples:
        Targeted directional enrichment:

        >>> # Add only node properties to transmission lines
        >>> lines_with_nodes = enricher.append_directional_properties(
        ...     line_df, node_df, 'node', MembershipTagging.NONE
        ... )
        >>> # Result: lines + 'voltage_from', 'voltage_to', 'area_from', 'area_to'
    """
    result_df = target_df.copy()
    for tag in [self._from_identifier, self._to_identifier]:
        membership_column = self._get_full_column_name(base_column, tag, target_df.columns)

        if membership_column not in target_df.columns:
            continue

        membership_objects = target_df[membership_column].dropna().unique()
        missing_objects = set(membership_objects) - set(source_df.index)

        if missing_objects:
            self._log_missing_objects_warning(missing_objects, membership_column)

        source_properties = source_df.copy()

        match membership_tagging:
            case MembershipTagging.PREFIX:
                source_properties = source_properties.add_prefix(f"{base_column}{self._membership_tag_separator}")
            case MembershipTagging.SUFFIX:
                source_properties = source_properties.add_suffix(f"{self._membership_tag_separator}{base_column}")

        source_properties = source_properties.add_suffix(tag)

        result_df = result_df.merge(
            source_properties,
            left_on=membership_column,
            right_index=True,
            how="left"
        )
    return result_df