GitLab client: specify artifact name? (.simg vs .sif)

[MarkusLohmayer]

First of all, thanks for developing this tool!

I have used https://gitlab.com/singularityhub/gitlab-ci to build a Singularity image based on a Def file in a repo.

Now I would like to use this tool to pull the image.

sregistry search ltd/some-repo
[client|gitlab] [database|sqlite:////home/ltd-admin/.singularity/sregistry.db]
Artifact Browsers (you will need path and job id for pull)
1  job_id       browser
2  96   https://git.ltd.uni-erlangen.de/ltd/some-repo/-/jobs/96/artifacts/browse/build

So I want to pull the image of job 96:

sregistry pull 96,ltd/singularity
[client|gitlab] [database|sqlite:////home/ltd-admin/.singularity/sregistry.db]
Looking for artifact build/Singularity.simg for job name build, 96
https://git.ltd.uni-erlangen.de/ltd/some-repo/-/jobs/96/artifacts/raw/build/Singularity.simg/?inline=false
ERROR Invalid url or permissions https://git.ltd.uni-erlangen.de/ltd/singularity/-/jobs/96/artifacts/raw/build/Singularity.simg/?inline=false
ERROR Cannot find /home/ltd-admin/.singularity/shub/ltd/singularity/latest.sif

I think this doesn't work because the file is called Singularity.sif and not Singularity.simg.
So in some sense, I see a mismatch between this project and the template.
Since the file ending .sif seems to be standard, I would think that this tool should expect images to be called Singularity.sif.

Further, I would like to know if there is any way to specify the artifact name when pulling the image
or is it the case that Singularity must be its name for the tool to work.

[vsoch]

hey @MarkusLohmayer let's get this fixed up for you! The GitLab sregistry client hasn't been updated in a long time so I'm not surprised that we have issues. It looks like the expectation is to provide a job id, collection, and job nameL

https://github.com/singularityhub/sregistry-cli/blob/master/sregistry/main/gitlab/pull.py#L52

and then you are correct it assumes a Singularity.simg https://github.com/singularityhub/sregistry-cli/blob/master/sregistry/main/gitlab/pull.py#L68. Let's refactor this to be able to specify the artifact directly. To work on this, are you able to ensure that one of your image builds is publicly accessible, and then describe how you would see doing the pull? If we can decide on the interaction (what you type to pull) together, I can test/develop for the changes, and then open a PR for you to test.

[MarkusLohmayer]

@vsoch thanks for your feedback!

The documentation (http://sylabs.io/docs/..) mentions .sif as the extension for Singularity containers. So I think the easiest way to make progress would just be to change line 68 (.simg to .sif), isn't it?

It's slightly off-topic here, but also the build.sh (https://gitlab.com/singularityhub/gitlab-ci/-/blob/master/.gitlabci/build.sh) should expect a file with the extension .def. It currently expects a file to be called just Singularity without any extension.

From a usability perspective, do you see a significant disadvantage when being forced to the names Singularity.def and Singularity.sif?
I guess that rarely will it be necessary to have multiple containers for a given project/repo.
The use-case I have in mind is simply a more elegant/scriptable way to get the image from the GitLab instance to the workstation (avoiding the need for the user to download it through the browser).

Unfortunately, I currently don't have any means to make the image builds publicly available. I was doing this for a GitLab instance which pertains just to a specific university lab.

Thanks again for your help!

[vsoch]

I’ll need you to help me set up a repo and build to reproduce your use case, otherwise I can’t be of much help because I can’t develop this.

[vsoch]

And we can definitely update simg to sif, and a different artifact name (or allowing the user to specify it and default to something else) if you like that approach. What I'd ask for your help with is to take your current build, and turn it into a dummy build on a public GitLab account (you don't need to set up a runner, GitLab provides them and you can use the default) and then point me to the repository and artifact and I'll get started. Thank you!

[MarkusLohmayer]

Dear Vanessa, wouldn't it make sense to take your template https://gitlab.com/singularityhub/gitlab-ci as the "reference use case"? What I currently have is not much different from your template!
Thank you!

[vsoch]

That was run years ago. There is good chance that it's different. If we want to be absolutely sure that the build logic lines up with the issue you are having, then we need a dummy example for it. I can update the extension to be sif easily, but if we aren't testing it's mostly shooting in the dark. If you don't want to help, then I'm happy to close the issue.

[MarkusLohmayer]

Don't get me wrong, please. I am referring to your template because it is essentially exactly what I have and because it is what other people will probably look at as well. Therefore I think that the template and the client should work together. Therefore I would prefer to work with it, rather than with a short-lived project.
I could help you to make the required changes regarding the file extensions though, if you like.
I assume, when these changes are made, the pipeline will anyhow run again, giving us a fresh build, right?
Then some adjustments to the client can be made to work correctly with the template.
Is this fine with you? Maybe I am still missing some point.

[vsoch]

That would work for me - if you want to make changes to the template on Gitlab, then we can build there, then I can test development here. Please add builds that will test the features you want (e.g., a specific artifact name).