Instamatic 2.2.0 does not install properly from pip

[Baharis]

Thanks to Andrew Stewart's bug report I have found out that Instamatic version 2.2.0 is not properly packaged, at least not for pip. Compared to pip install -v "Instamatic==2.1.1", the latest release does not include most of the files: some directories and src/*.py files are present, but it seems basically any src/*/*.py files, ones in non-source directories, are absent. I do not remember messing with packaging, but my best guess is that I have accidentally removed some recursive flag for packaging discovery search when adding new scripts. Unfortunately, PyPI does not allow re-uploading version with the same name, so once I locate an issue this will likely warrant version 2.2.1 with little to no code changes.

Edit: the most relevant difference I can see is the addition of the following in pyproject.toml:

packages = ["instamatic"]
package-dir = {"" = "src"}
include-package-data = true

I guess this replaces some defaults while does not pointing to the .py files.

[Baharis]

I think I found the culprit. Somewhere in-between 2.1.1 and 2.2.0 the following lines were added to the pyproject.toml:

[tool.setuptools]
packages = ["instamatic"]
package-dir = {"" = "src"}
include-package-data = true

Apparently by themselves they break the recursive discovery and need to be supplemented to work instead:

[tool.setuptools]
package-dir = {"" = "src"}
include-package-data = true

[tool.setuptools.packages.find]
where = ["src"]
include = ["instamatic*"]

or

[tool.setuptools]
packages = { find = { where = ["src"], include = ["instamatic*"] } }
package-dir = {"" = "src"}
include-package-data = true

This comes from ChatGPT so is likely true but should be confirmed elsewhere. I suggest (and can take care of) adding the first solution and pushing version 2.2.1 ASAP. Alternatively removing the block may revert to defaults which apparently somehow work.

[stefsmeets]

The files are managed via manifest.in: https://github.com/instamatic-dev/instamatic/blob/main/MANIFEST.in

I don't know if it conflicts with the pyproject config added in aafc04e.

EDIT: This page explains how files are included. It seems that maybe the file declaration in pyproject.toml conflicts with manifest.in?
https://setuptools.pypa.io/en/latest/userguide/datafiles.html#interplay-between-these-keywords

[Baharis]

My guess is that MANIFEST.in adds all non-python files via include src/instamatic/... commands, whether current pyproject.toml adds src/instamatic/*.py files. None explicitly adds other .py files, hence they are absent. MANIFEST.in does not point to python files.

I personally believe that "explicit is better then implicit" and I would try adding the ChatGPT suggested lines. Then I will test using TestPyPI before creating a release. I think I already mentioned it once, IMO TestPyPI should be a part of the release pipeline to prevent issues like that.

[stefsmeets]

None explicitly adds other .py files, hence they are absent. MANIFEST.in does not point to python files.

Python files are included by default by pointing at the scr directory, typically this is enough:

[tool.setuptools]
package-dir = {"" = "src"}

Then I will test using TestPyPI before creating a release.

It's not necessarily a pypi issue, it's a build issue. You can test this locally via python -m build. The command line lists the files added.

[Baharis]

Today I located the action that published 2.2.0, https://github.com/instamatic-dev/instamatic/actions/runs/20032624674/job/57445582881, and indeed it is missing all the files. So I recreated the environment on my PC (setuptools, build, pip, packaging etc.) and run python -m build. And lo and behold, the full package builds as usual, including all the python files! Which would be great if the python action behaved the same way, but it clearly does not. I guess once I find time, I will make an action on a custom branch that builds locally and see how it fares.

[stefsmeets]

That's odd, the logs are pretty vocal about the packages missing (and apparently setuptools hates how we define the licenses 😅). Is this something that was changed in a recent version of setuptools?

[stefsmeets]

FYI I yanked the 2.2.0 release: https://pypi.org/project/instamatic/2.2.0/