CI is using up too much git-lfs bandwidth

[chatcannon]

GitHub puts a quota on git-lfs bandwidth usage, as well as storage space. We are nowhere near filling the storage limit but because every CI run pulls down all the LFS data once for every version of python tested, it eats up the free quota very quickly.

We wanted to let you know that you’ve used 80% of your data plan for Git LFS on the organization echemdata. No immediate action is necessary, but you might want to consider purchasing additional data packs to cover your bandwidth and storage usage:

https://github.com/organizations/echemdata/billing/data/upgrade

Current usage as of 03 Feb 2024 04:21PM UTC:

Bandwidth: 0.8 GB / 1 GB (80%)
Storage: 0.01 GB / 1 GB (1%)

[chatcannon]

@ml-evs I don't know much about the new github actions CI setup - can you look into this? There ought to be a way to cache data between runs.

[ml-evs]

I've just opened #101 which hopefully fixes this, making use of https://github.com/nschloe/action-cached-lfs-checkout until GitHub adds native support for this (or ideally stops charging for internal network bandwidth). I think on the free "academic team" plan on GitHub the LFS quota is much more generous. Depending on how you view things, the echemdata organisation might be upgradable to this tier as long as someone in the organisation retains an academic affiliation (and as long as the org remains to be used for stuff like galvani).

[ml-evs]

#101 doesn't seem to be having much effect, I'm not sure if this is because the bandwidth for the month is already exceeded. Unfortunately this is a product of not having an up to date release on PyPI, so anyone who wants to install the dev version is either failing and using up git lfs bandwidth, or is having to disable LFS.

[chatcannon]

I still got github actions errors yesterday even though it is the beginning of a new month.

I think this might be the last straw that finally pushes me over the edge to migrate to https://sourcehut.org/ - I would certainly rather pay them than Microsoft.

[chatcannon]

I finally found a free time slot (at 5:30 am) where I could try to make a release. But GitHub won't let me pull the latest code because the git-lfs quota is taken. I'm definitely switching to another platform.

[ml-evs]

GIT_LFS_SKIP_SMUDGE=1 will prevent it from downloading the test files again (see the README), which you don't need for the release, though I understand the frustration. The CI no longer uses that bandwidth at least, and presumably its just normal users cloning it that causes the issue.

[chatcannon]

I finally found a free time slot (at 5:30 am) where I could try to make a release. But GitHub won't let me pull the latest code because the git-lfs quota is taken. I'm definitely switching to another platform.

I set up a fork at https://codeberg.org/echemdata/galvani

If that turns out to be more usable than GitHub then I will archive this repo and switch development to CodeBerg.

[ml-evs]

Seems like a shame, any PyPI release in the last year would have mitigated the git lfs bandwidth problem, which is only a problem when people naively clone the repo without reading the instructions to get the latest unreleased changes. I'll probably maintain a fork at https://github.com/datalab-org/galvani and release to PyPI under a separate package name (galvani-mirror or something, rather than galvani2) though happy to sync any changes in either direction (and I doubt we will add much more functionality ourselves).