Skip to content

Conversation

@JGrothoff
Copy link
Contributor

Many AASX packages include semantic ids with external or model references.

Concept descriptions are lost when reading and writing back to AASX

Problem

When a semantic id with a ModelReference is deserialized its type is currently set to the abstract type Referable.
When the aasx is serialized again and the semantic id refers to a concept description, it is skipped, because only concrete ModelReference types are looked up and serialized.

Solutions

  • Suggested: Set the type of the semantic id to the concrete type determined by the last key element.
  • Alternative: Skip the type check as errors are caught anyway (EAFP style). Maybe lowering the loglevel to warning or info.

Writing aasx with external references as semantic ids produce many info logs

Problem

Often an external reference with a global id (e.g. to ECLASS IRDIs) is used as semantic id.
A typical technical data submodel may easily contain around 100 of them.
Writing to an aasx will print an info message for each semantic id, that is not a model reference to an concept description, which clutters the output.

Solutions

  • Suggested: skip external references without log message
  • Alternative: lower logging level to debug or delete log info.

@JGrothoff JGrothoff marked this pull request as draft November 19, 2024 08:47
@JGrothoff JGrothoff force-pushed the fix/write-aasx-cd-external branch from 968597b to 9f0dea6 Compare November 19, 2024 08:49
@s-heppner
Copy link
Contributor

Thank you very much for your input!
This is indeed a quite complicated problem, as the specification was not clear about which ReferenceType to use and therefore we have lots of weird AASX files in the wild.

I agree that our SDK should not discard content without warning when reading and writing again, so that is definitly something that needs to be addressed.

Regarding your suggestions: I need to do some more thinking about which solution is the one with the least repercussions. I'll bring this discussion to our dev meeting and will get back to you.

@JGrothoff
Copy link
Contributor Author

Thanks for investigating the topic so far!
I will try to add or update the tests to cover the issue (thats why i marked this as Draft).

@JGrothoff JGrothoff force-pushed the fix/write-aasx-cd-external branch 2 times, most recently from 904e747 to 381d6a9 Compare November 21, 2024 16:36
@JGrothoff JGrothoff marked this pull request as ready for review November 21, 2024 16:38
@JGrothoff
Copy link
Contributor Author

I added a test that covers all semantic ids that are expected to be ModelReferences for the same type.

@s-heppner
Copy link
Contributor

This is what we concluded in our meeting:

  • We cannot allow writing wrong AASXs. If the ConceptDescription of the ModelReference is not found, we should raise an Error as the AASX is wrong.
  • However, we should obviously not silently discard content, as in not warning the user about the issue.
  • We should try to declutter our output (if there's not actually an issue with the ExternalReference). This message may be now unnecessary and might have been historically grown

Sadly, I did not have time yet to look deeper into the matter.

@JGrothoff JGrothoff force-pushed the fix/write-aasx-cd-external branch 2 times, most recently from ee006dc to a441d20 Compare December 9, 2024 13:41
@JGrothoff
Copy link
Contributor Author

  • We cannot allow writing wrong AASXs. If the ConceptDescription of the ModelReference is not found, we should raise an Error as the AASX is wrong.

I rebased on main and changed to a re-raise for those cases. It will raise an error if the reference is a ModelReference and has the type ConceptDescription, but it can't be resolved in the given object_store.
Of couse, we could change the Exeception class here, but i will leave that up to you.

This will still not check if

  • an abstract Reference or
  • a ModelReference with type other than a ConceptDescription
    can be resolved and is contained in the object_store (or in the objects_to_be_written).
    But it will log an info message about that, like before.

@s-heppner s-heppner changed the base branch from main to develop January 6, 2025 08:59
Don't log `ModelReferences` not pointing to `ConceptDescriptions`, as it is not contrained by the spec.
Warn about missing references, as the `object_store` seems to be incomplete.
@JGrothoff JGrothoff force-pushed the fix/write-aasx-cd-external branch from a441d20 to f0c7eb1 Compare January 6, 2025 12:04
@s-heppner s-heppner merged commit 933b9fa into eclipse-basyx:develop Jan 15, 2025
14 checks passed
@JGrothoff JGrothoff deleted the fix/write-aasx-cd-external branch January 15, 2025 11:09
@s-heppner s-heppner added this to the Release v1.1.1 milestone Jan 29, 2025
Frosty2500 pushed a commit to rwth-iat/basyx-python-sdk that referenced this pull request Mar 30, 2025
…ce` (eclipse-basyx#337)

Previously, if the `semantic_id` was a 
`ModelReference`, it very often had the generic 
`type` "`model.Referable`" instead of the actual
type of the object it points to. This lead to a 
bug in the AASX writer, where 
`ConceptDescription`s were not written to the 
AASX, even though they existed in the 
`ObjectStore`.

This fixes this problem by tackling three separate
points: 

1. In `json_deserialization.py`, we now infer the
`type` of the `ModelReference` via the Reference's
last `Key`. This is more of a hotfix and issue 
eclipse-basyx#367 tracks the creation of a better solution. 

2. In `aasx.py`, an unneccessary `logger.info` 
call is removed, that previously cluttered the 
output of the writer if a `semantic_id` was not 
pointing to a `ConceptDescription`, which is not
something worth noting. Furthermore, we improve
other log messages. 

3. In `examples.data._helper`, we improve the test
for `semantic_id` equality, namely by comparing
the `ModelReference`'s attributes explicitly, when
it happens to be one, instead of relying on simple
string comparision.
s-heppner pushed a commit that referenced this pull request Apr 23, 2025
…ce` (#337)

Previously, if the `semantic_id` was a 
`ModelReference`, it very often had the generic 
`type` "`model.Referable`" instead of the actual
type of the object it points to. This lead to a 
bug in the AASX writer, where 
`ConceptDescription`s were not written to the 
AASX, even though they existed in the 
`ObjectStore`.

This fixes this problem by tackling three separate
points: 

1. In `json_deserialization.py`, we now infer the
`type` of the `ModelReference` via the Reference's
last `Key`. This is more of a hotfix and issue 
#367 tracks the creation of a better solution. 

2. In `aasx.py`, an unneccessary `logger.info` 
call is removed, that previously cluttered the 
output of the writer if a `semantic_id` was not 
pointing to a `ConceptDescription`, which is not
something worth noting. Furthermore, we improve
other log messages. 

3. In `examples.data._helper`, we improve the test
for `semantic_id` equality, namely by comparing
the `ModelReference`'s attributes explicitly, when
it happens to be one, instead of relying on simple
string comparision.
zrgt added a commit that referenced this pull request Sep 8, 2025
* fix some typos

* fix README.md

* tutorial_serialization_deserialization: Fix missing import (#363)

We add a missing import statement of `basyx.aas.adapter.json`. 

Fixes #352

* README.md: Fix typo in Markdown Link

* compliance-tool: Fix compliance tool imports and unitests (#356)

Currently, there are several wrong imports and
outdated unittests in the `compliance-tool` 
package.

This fixes these issues.

Fixes #354

* aasx.adapter: Fix `semantic_id` type deserialization of `ModelReference` (#337)

Previously, if the `semantic_id` was a 
`ModelReference`, it very often had the generic 
`type` "`model.Referable`" instead of the actual
type of the object it points to. This lead to a 
bug in the AASX writer, where 
`ConceptDescription`s were not written to the 
AASX, even though they existed in the 
`ObjectStore`.

This fixes this problem by tackling three separate
points: 

1. In `json_deserialization.py`, we now infer the
`type` of the `ModelReference` via the Reference's
last `Key`. This is more of a hotfix and issue 
#367 tracks the creation of a better solution. 

2. In `aasx.py`, an unneccessary `logger.info` 
call is removed, that previously cluttered the 
output of the writer if a `semantic_id` was not 
pointing to a `ConceptDescription`, which is not
something worth noting. Furthermore, we improve
other log messages. 

3. In `examples.data._helper`, we improve the test
for `semantic_id` equality, namely by comparing
the `ModelReference`'s attributes explicitly, when
it happens to be one, instead of relying on simple
string comparision.

* sdk: Update version of pyecma dependency in pyproject.toml

This updates the minimum version of the dependency `pyecma376` in the `pyproject.toml`, ensuring that our AASX files allow for spaces in file names. 

Fixes #236

* sdk: Move testing depencies to dev section in pyproject.toml (#369)

Previously, some `sdk` dependencies, that are only 
necessary for testing were included in the 
mandatory dependencies.

For improving package footprint, these are moved 
into the `[dev]` section of optional dependencies
in the `pyproject.toml`.

---------

Co-authored-by: s-heppner <iat@s-heppner.com>

* basyx.provider: Add SetObjectStore to provider (#340)

This adds a new `SetObjectStore` that is backed by 
a set. Its main advantage is that it does not have 
the problem with not updated `Identifier`s
that the `DictObjectStore` has (See #216).

We make sure to document the differences between
`DictObjectStore` and `SetObjectStore` in their
respective class docstrings.

* scripts.set_copyright_year: Add a CI check to ensure the copyright is updated (#344)

Previously, there was no check if the copyright
statement on top of each file (due to the MIT
license) was up to date. 

This adapts our `update_copyright_year.sh` script
with a checking mode to determine if any file
needs adapting and then adds this check to the CI.

At the same time, this also updates all necessary
files to the year 2025.

Fixes #260
Fixes #331

---------

Co-authored-by: s-heppner <iat@s-heppner.com>

* sdk/.readthedocs.yaml: Install dependencies from pyproject.toml (#359)

Previously, we were accidentally still trying to 
install dependencies from the already deleted 
`requirements.txt` when generating the 
documentation.

This fixes this and installs them instead from 
the `pyproject.toml`.

Co-authored-by: s-heppner <iat@s-heppner.com>

* compliance_tool: add pyproject.toml (#361)

This refactors away from `setup.py` and towards
`pyproject.toml` for the `compliance_tool` 
package.

At the same step, we also update the relevant CI 
checks as well.

* http.py: Fix redirects, bugs, and SDK installation (#362)

This fixes the redirects from the AAS repository 
paths to the Submodel repository paths to work
properly.

Furthermore, the installation of dependencies 
inside the server Docker image was previously 
using the latest release of the SDK. This means,
that we could not ensure that each commit in the
`main` branch of our monorepo would be 
interoperable between SDK and server, as a PR 
would have to be closed in order for the server CI
would not report any errors. In order to fix this,
issue in the development process, the server 
Docker image now installs the SDK from the local
repository, rather than from GitHub.

Lastly, this fixes a wrong status code reported
when specifing a model that is malformed or 
missing information. The specification expects a
400 (Bad Request) response, but the server sent a 
422 (Unprocessable Content). 

Fixes #315

* CONTRIBUTING.md: Improve codestyle and testing section (#376)

Previously, the section "Codestyle and Testing"
was outdated, still from the time where there was
only the SDK inside this repository.

This greatly extends the section, renaming it to
"Code Quality" and introducing subsections for
"Codestyle" and "Testing", where we describe how
to run the necessary tests locally on the
developer's machine for each of the packages
inside this monorepository.

Fixes #353

* compliance_tool: Remove setup.py (#377)

We previously refactored the compliance_tool from
`setup.py` to `pyproject.toml` (See #361).
However, somehow the `setup.py` slipped through
and is still in the `compliance_tool` directory.

This removes the `setup.py` from the
`compliance_tool` completely.

* Add CI job to release compliance-tool to PyPI (#382)

This adds a job `compliance-tool-publish` to the `release.yml` that
automatically publishes the compliance-tool package to PyPI upon
release. It is simply copied from the SDK job, with adapted paths.

* sdk: Update lxml and mypy dependency in pyproject.toml (#392)

* sdk: Update lxml dependency in pyproject.toml

Previously, the `lxml` dependency in the `pyproject.toml` was pinned to
versions `>=4.2,<5`. This caused a faulty installation via `pip` on
Windows due to missing binary wheels.

This updates the `lxml` dependency to versions `>=5.3` to resolve the
issue.

Additionally, the `mypy` dependency in the `pyproject.toml` was unpinned,
leading to the installation of the latest version. The recent release of
`mypy 1.16.0` introduced changes not yet supported by our codebase,
resulting in CI pipeline failures.

This restricts the `mypy` dependency to version `1.15.0` until we have
proper support for `1.16.0` and its new features.

Fixes #391

---------

Co-authored-by: Sercan Sahin <s.sahin@iat.rwth-aachen.de>
Co-authored-by: Frosty2500 <125310380+Frosty2500@users.noreply.github.com>
Co-authored-by: Julian Vogel <julian.vogel@sms-group.com>
Co-authored-by: Igor Garmaev <56840636+zrgt@users.noreply.github.com>
Co-authored-by: Moritz Sommer <moritz.sommer@rwth-aachen.de>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants