MOTEL Methodology

1. Purpose

MOTEL (Methodology for Open Technology Data in Energy Models) provides a workflow for turning heterogeneous technology data into structured records that can be reviewed, harmonised, and reused in energy system modelling. In the current repository, the implemented core is the ingestion of source data into an unmapped staging format, the harmonisation of those staged records into a MOTEL database structure with controlled vocabularies, secondary entities, mapping tables, and a linked_entity schema, and a notebook-first exploration layer for inspecting the published data product.

The repository already includes machine-readable schemas in schema/unmapped_entity.yaml, schema/linked_entity.yaml, and the supporting entity schemas under schema/secondary and schema/controlled_vocabulary. It also includes human-readable blueprints under schema_human/unmapped_entity.yaml and schema_human/linked_entity.yaml. The current implementation does not yet include a populated ontology-linked graph database, backend API, or web application in the public tree. Those parts should therefore be treated as future work rather than current capabilities.

2. Overall workflow

The intended end-to-end MOTEL workflow can be described as:

raw technology data
    -> unmapped YAML records
    -> harmonised MOTEL entities and vocabularies
    -> linked entity records and mapping tables
    -> ontology / graph export (future work in current repo)
    -> backend query layer (future work in current repo)
    -> frontend inspection and export tools (future work in current repo)

In the current repository snapshot, the implemented steps are:

1. Collect or receive raw technology data.
2. Store unresolved records as unmapped data.
3. Harmonise data into the MOTEL database structure.
4. Create referenced entities and mapping tables for technologies, processes, sources, carriers, attributes, and scopes.
5. Create populated linked_entity records that link harmonised technologies, sources, scopes, carriers, and attributes.

Current limitation: Graph database construction, backend querying, frontend interaction, and model-ready export are still downstream work rather than implemented end-to-end features in the present public repository. The current tree does include an implemented ontology-mapping step under 3_ontology_mapping/, but no backend or frontend application code is present here.

3. Data ingestion and harmonisation

Data ingestion

The primary ingestion method implemented in MOTEL is schema-first staging into the unmapped format. For a new project or external user, the intended starting point is not the reFuel.ch notebook itself, but the generic unmapped contract defined in schema/unmapped_entity.yaml and explained in human-readable form in schema_human/unmapped_entity.yaml. The introductory notebook 1_ingest/1_data_ingestion.ipynb serves as the current guide to that structure.

In practice, a new contributor can work as follows:

1. Collect raw technology information from spreadsheets, reports, databases, or manual review.
2. Map each raw record into the unmapped schema with at least a technology_name, source references, attributes, balancing information, and metadata, while storing raw scope values plus paired scope descriptions where relevant.
3. Save those records as YAML in the unmapped structure.
4. Run the harmonisation workflow to resolve free-text names into MOTEL registries and controlled vocabularies.

This staging can be done manually or with project-specific transformation code:

• Manual route: a user can prepare YAML directly against schema/unmapped_entity.yaml.
• Scripted route: a project can create its own ingestion notebook or helper script, following the pattern used in the existing reFuel.ch example.
• LLM-supported route: in the current repository, LLM support is implemented mainly in harmonisation rather than in generic ingestion. The harmonisation helper in 2_harmonise/harmonise_helpers.py uses a local Ollama model to help standardise names, fill required fields, and match records against existing registries after the unmapped YAML has been created.

The main implemented project-specific example is the reFuel.ch workflow in 1_ingest/examples/refuel/ingestion_pipeline.ipynb, supported by 1_ingest/examples/refuel/scripts/ingestion_helper.py. This example reads an Excel workbook with multiple sheets, including ConvTech, StorTech, EmbeddedCarbon, and Reference, transforms those records into the unmapped schema, and writes sheet-level exports such as:

• 1_ingest/examples/refuel/output/unmapped_entities_refuel_convtech.yaml
• 1_ingest/examples/refuel/output/unmapped_entities_refuel_stortech.yaml
• 1_ingest/examples/refuel/output/unmapped_entities_refuel_embeddedcarbon.yaml

The same notebook then publishes a combined staging file to motel-db/unmapped_entity/unmapped_entities_refuel.yaml.

This reFuel.ch notebook should therefore be read as a worked example of how a project-specific adapter can populate the generic MOTEL staging format, not as the only allowed ingestion route.

Current supported input format:

• Excel workbook input through the reFuel.ch notebook workflow.

Current limitation:

• No general-purpose CSV, JSON, API, or form-based ingestion pipeline is implemented in the current public tree.
• Ingestion is notebook-driven rather than packaged as a reusable command-line tool or service.

Unmapped versus linked entities

unmapped data is the raw staging format. It keeps source-oriented names, raw scope values paired with human-readable scope descriptions, flexible attribute payloads, and source references before they are matched to MOTEL registries. This contract is defined in schema/unmapped_entity.yaml.

linked_entity is the target relational structure for harmonised records. It is defined in schema/linked_entity.yaml and explained in a human-readable form in schema_human/linked_entity.yaml. It is intended to reference standardised technologies, attributes, carriers, scopes, and sources through foreign-key style identifiers.

In practice, the harmonisation step currently writes:

• controlled vocabularies under motel-db/controlled_vocabulary
• referenced secondary entities under motel-db/secondary
• provenance and crosswalk tables under motel-db/mapping

The harmonisation notebook is 2_harmonise/2_data_harmonisation.ipynb, and its main logic lives in 2_harmonise/harmonise_helpers.py.

Current limitation:

• The linked-entity structure is implemented and populated, but some field names and schema details are still being aligned as the workflow evolves.

4. MOTEL data structure

The implemented data model in this repository is split across schemas and CSV/YAML registries.

Main entities currently represented:

• technology: standardised technology registry in schema/secondary/technology.yaml and motel-db/secondary/technology.csv
• process: standardised process registry in schema/secondary/process.yaml
• source: standardised reference registry in schema/secondary/source.yaml and motel-db/secondary/source.csv
• attribute: controlled parameter vocabulary in schema/controlled_vocabulary/attribute.yaml and motel-db/controlled_vocabulary/attribute.csv
• carrier: controlled carrier vocabulary in schema/controlled_vocabulary/carrier.yaml
• scope vocabularies: geographic_scope, temporal_scope, capacity_scope, system_boundary
• linked_entity: harmonised record structure in schema/linked_entity.yaml

Minimum fields for a technology in the current schema:

• tech_id
• technology_name

Additional currently defined fields include technology_description, technology_variant, main_process, main_operation_unit, and ontology_iri.

Minimum fields for an attribute in the current schema:

• attribute_id
• attribute_name
• attribute_description
• unit
• data_format

Additional currently defined fields include ontology_iri and note.

The unmapped record structure is broader and includes:

• technology_name
• nested technology
• nested scope
• sources
• attributes
• balancing
• metadata
• harmonisation_record

This makes unmapped suitable for preserving source-level context before standardisation.

5. Ontology mapping

The current repository includes an implemented ontology-mapping workflow as well as ontology-aware schema fields. In particular, the technology and attribute schemas both include an ontology_iri field:

• schema/secondary/technology.yaml
• schema/controlled_vocabulary/attribute.yaml

This indicates that MOTEL entities are designed to carry resolvable ontology links. The public repository now includes ontology mapping assets under 3_ontology_mapping, including the notebook 3_ontology_mapping/3_ontology_mapping.ipynb, mapping config 3_ontology_mapping/config/attribute_ontology_mapping.yaml, generator script 3_ontology_mapping/scripts/gen_ttl.py, and published TTL output 3_ontology_mapping/output_ttl/cls_atr_motel.ttl.

The current mapping tables in motel-db/mapping support harmonisation traceability rather than ontology conversion. For example:

Current limitation:

• The repository includes ontology-ready TTL generation, but it does not yet document a full repository-local ontology vocabulary package such as a complete .owl or .ttl ontology source tree.
• The public docs still describe the mapping outcome at a high level rather than fully enumerating field-to-predicate mappings.
• Ontology export is implemented, while GraphDB loading and application-layer use remain downstream work.

6. Graph database construction

The current repository does not contain an implemented graph database build pipeline. What it does contain is the upstream ontology-mapping stage in 3_ontology_mapping, which generates ontology-ready TTL from harmonised MOTEL data. No GraphDB loading, SPARQL service, or Neo4j configuration files were found in the active tree.

What is implemented today is the relational precondition for graph construction:

• harmonised entities in motel-db/secondary/
• controlled vocabularies in motel-db/controlled_vocabulary/
• source mappings in motel-db/mapping/
• linked-entity schema in schema/linked_entity.yaml

If a graph layer is added later, the likely relationship pattern would be:

technology -> has parameter -> attribute
attribute -> has value/unit/source -> value record
technology/attribute -> maps to ontology IRI
source -> provides provenance for parameter values

Current limitation:

• No implemented graph construction script exists in the current public tree.
• No serialized RDF or graph database export artifact is present.

7. Backend and frontend workflow

No backend API or frontend application code is present in the current public repository snapshot. The repository contains a static documentation site in docs/index.html, but no service code for querying or editing the MOTEL database. That application-facing layer is maintained in the separate public repository uesl-empa/motel-webapp.

Therefore, the following workflow is not yet implemented in code in this tree:

1. Open a webapp.
2. Browse or select technologies.
3. Inspect parameters and provenance through an API.
4. Edit or customise values through the interface.
5. Export model-ready configuration from the interface.

Current limitation:

• No backend endpoints exist in the active repository.
• No frontend pages or components exist beyond static documentation.
• No implemented browser-based editor or query UI was found.

8. Export to model-ready input

The implemented export in the current repository is YAML export for staging data rather than model-ready solver configuration export.

Implemented exports:

• per-sheet unmapped YAML exports in 1_ingest/examples/refuel/ingestion_pipeline.ipynb
• consolidated staging export to motel-db/unmapped_entity/unmapped_entities_refuel.yaml
• harmonised CSV registries under motel-db/secondary/, motel-db/controlled_vocabulary/, and motel-db/mapping/

The current export preserves, depending on the stage:

• technology name
• raw or harmonised parameter names
• values
• units
• source references and linked attributes
• harmonisation mappings
• selected provenance metadata

Current limitation:

• No implemented export to model-specific YAML, TOML, JSON, or solver configuration files was found.
• No API-based export function exists in the current public tree.
• No graph-enriched export is implemented.

9. ORD and FAIR contribution

The implemented workflow already contributes to ORD and FAIR goals in several concrete ways.

First, it improves traceability by preserving raw source references in unmapped records and by generating explicit mapping tables such as motel-db/mapping/source_map.csv. Second, it makes assumptions more visible by storing raw scope values, semantic scope descriptions, metadata notes, and source-linked attributes before and during harmonisation. Third, it creates reusable structured data through explicit YAML schemas and CSV registries rather than leaving technology assumptions only in spreadsheets.

The workflow also supports reproducibility because the transformation from spreadsheet input to staging YAML and then to harmonised registries is encoded in notebooks and helper scripts rather than being entirely manual. Finally, the presence of ontology_iri fields in the schemas establishes a concrete interoperability path, even though the ontology export layer is not yet implemented in this repository snapshot.

10. Current limitations and future work

• Current limitation: only the reFuel.ch ingestion workflow is implemented as a full example.
• Current limitation: ingestion and harmonisation are notebook-driven rather than packaged as stable command-line or service workflows.
• Current limitation: the linked_entity output is populated, but parts of the schema and downstream documentation still need alignment with the latest field names and workflow behavior.
• Current limitation: ontology mapping is implemented in 3_ontology_mapping/, but the public documentation still leaves some mapping details and ontology packaging implicit.
• Current limitation: graph database construction is not implemented in the current public repository.
• Current limitation: no backend API or frontend application is present in the public repository snapshot; those components are maintained in uesl-empa/motel-webapp.
• Current limitation: no model-ready export beyond YAML staging files and harmonised CSV registries is implemented.
• Future work: add a formal ontology vocabulary and explicit field-to-predicate mappings.
• Future work: implement graph export and query infrastructure.
• Future work: add a backend service and frontend interface for inspection, editing, and export.
• Future work: broaden ingestion beyond the current notebook example and add stronger automated validation.