Contributing: Decrease the size of the source code needed to be downloaded

[gziolo]

Similar to #26993.

What problem does this address?

It takes ages to finish:

git clone git@github.com:WordPress/gutenberg.git

At the moment the size of the repository is over 2GB!!!!

If you add to the mix that you need to run on every brand new repository:

npm install

It adds another 1GB of data that needs to be downloaded as reported in #26993.

What is your proposed solution?

It makes me think that maybe gh-pages branch is one of the reasons why the size of the repository has grown so much. We replace the content of gh-pages with the new build of Storybook on every commit to the main branch.

I don't know how this sort of issues are usually solved in git-based repositories, but the comment from WordPress Slack (link requires registration at https://make.wordpress.org/chat/) authored by @ocean90 should be a good start:

https://wordpress.slack.com/archives/C5UNMSU4R/p1609864617204200?thread_ts=1609770083.149700&cid=C5UNMSU4R

Yes by creating a new orphan branch from gh-pages. You have to add the files there and the gh-pages branch needs to be deleted. Then rename the new branch to gh-pages which finally gets force-pushed.
This post documents the steps

The link included:
https://medium.com/@sangeethkumar.tvm.kpm/cleaning-up-a-git-repo-for-reducing-the-repository-size-d11fa496ba48

This article contains some techniques that could help with gh-pages where we don't care about history at all. There are also several interesting references to other similar articles that try to address similar issues.

[iandunn]

This command lists files that are larger than 1MB (requires brew install coreutils on OS X):

git rev-list --objects --all | git cat-file --batch-check='%(objecttype) %(objectname) %(objectsize) %(rest)' | sed -n 's/^blob //p' | awk '$2 >= 2^20' | sort --numeric-sort --key=2 | gcut -c 1-12,41- | $(command -v gnumfmt || echo numfmt) --field=2 --to=iec-i --suffix=B --padding=7 --round=nearest

Results are at https://pastebin.com/hQaYcHwE

Those includes files in HEAD, though. I wasn't able to filter them out, but it should be possible.

I suspect there's a long tail of files < 1MB that could also be removed for a significant boost.

Removing files from the history would break the hashes, but might be worth it in this case.

[gziolo]

@iandunn, that's very helpful. Thank you for doing a more in-depth investigation.

Results are at https://pastebin.com/hQaYcHwE

@hypest, I see a lot of mobile-related files with the highest impact that don't look like source code. Would it be possible to remove some of them from the git history so we could make the Gutenberg repository faster to download?

59MiB diffcheck.txt

This one alone looks like a quick win if we erase the full history.

[hypest]

59MiB diffcheck.txt

This one alone looks like a quick win if we erase the full history.

Aha, I'm not familiar with what that file is but I'm sure there are savings to be made along those lines. @ceyhun , you think you can take a look when you get some chance, possibly after HACK week of March 2021? Thanks!

[iandunn]

git clone https://github.com/WordPress/gutenberg.git --depth 10 might also be interesting. That clones the repo with just the last 10 commits. It only took ~5 seconds to download on my machine.

I'm guessing someone could use that, then send a PR, and it wouldn't have any problems. I haven't tested that, though. The downside is that people would have to intentionally do it, since it's not the default behavior. Scripts and docs could be updated, though.

If we do remove stuff from history, I'd recommend getting rid of everything all at once if possible. Changing the history will break lots of stuff, so we'd probably only want to do it once every few years, at most.

[windhamdavid]

I tested the --depth. Faster but doen't do much for size. 2.2GB is 2much ✂️ Might consider something like SVN subtree using a filter-branch -f --prune-empty --subdirectory-filter and split off the docs(gh-pages) and tests. cc: #26993 (comment)

[ceyhun]

59MiB diffcheck.txt

This one alone looks like a quick win if we erase the full history.

Aha, I'm not familiar with what that file is but I'm sure there are savings to be made along those lines. @ceyhun , you think you can take a look when you get some chance, possibly after HACK week of March 2021? Thanks!

@gziolo I found the commits for this file using git log --all --full-history -- diffcheck.txt command. This seems to be the latest one deleting it: 397e645. It seems like an out of a git diff. I think we can safely erase this.

@hypest I also checked the pastebin and saw a lot of mobile bundles and binaries. I think it's also fine to erase these ones as we're not using them and they can also be regenerated if needed:

bundle/android/App.js
bundle/android/App.js.map
bundle/ios/App.js
bundle/ios/App.js.map
test/native/gutenberg-mobile-demo-app.apk
ios/Gutenberg.app.zip

[gziolo]

@ceyhun, this is a great finding. How can we perform this cleanup? Is it something you could do yourself?

[hypest]

@hypest I also checked the pastebin and saw a lot of mobile bundles and binaries. I think it's also fine to erase these ones as we're not using them and they can also be regenerated if needed

+1 for removing the Android ones, the APK and the app.zip, no questions asked.

For the iOS ones though, it will probably make trying out/debugging older WPiOS versions harder, right? Recreating them will probably be cumbersome. I actually don't like that we had to commit the JS bundles at all so, if you feel confident about iOS debugging without having the bundles readymade @ceyhun then I'm +1.

[ceyhun]

@ceyhun, this is a great finding. How can we perform this cleanup? Is it something you could do yourself?

@gziolo I'm not really sure what git magic is needed for this to happen 🪄 I also do not consider myself a git magician 😃 So any help would be appreciated.

For the iOS ones though, it will probably make trying out/debugging older WPiOS versions harder, right? Recreating them will probably be cumbersome. I actually don't like that we had to commit the JS bundles at all so, if you feel confident about iOS debugging without having the bundles readymade @ceyhun then I'm +1.

@hypest I think WPiOS was always using the bundles on gutenberg-mobile repo and seems like that one goes back as far as 2018, so maybe that's enough?

[jonathanbossenger]

@gziolo @ceyhun based on this SO answer, git filter-branch should allow you to completely remove those files from the repo history https://stackoverflow.com/questions/43762338/how-to-remove-file-from-git-history#43762489

[gziolo]

We discussed options on WordPress Slack in the #meta channel (link requires registration at https://make.wordpress.org/chat/):
https://wordpress.slack.com/archives/C02QB8GMM/p1616519854024400

@dd32 shared the following:

Playing with git rev-list --disk-usage shows that the gh-pages branch is 10x the next:

26.67MB refs/tags/@wordpress/warning@1.4.0
26.67MB refs/tags/@wordpress/wordcount@2.15.0
112.10MB refs/remotes/origin/import-gutenberg-mobile
117.89MB refs/remotes/origin/rnmobile/import-mobile-lint
119.86MB refs/remotes/origin/rnmobile/import-mobile-fix-ci
158.28MB refs/remotes/origin/rnmobile/try-fix-android-build
169.98MB refs/remotes/origin/feat/import-gutenberg-mobile-no-squash-E2E-TESTS-fix-ios-ci
182.22MB refs/remotes/origin/rnmobile/experiment-monorepo-new-setup-update-node
213.09MB refs/tags/rnmobile/monorepo-commit-history
2053.26MB refs/remotes/origin/gh-pages

I started with the first step and rewrote the history of gh-pages branch:
https://github.com/WordPress/gutenberg/commits/gh-pages

It looks like it mostly generates new bundle files for the Storybook instance available at https://wordpress.github.io/gutenberg/.

Can you check if we can remove the mobile branches listed completely?

[hypest]

@hypest I think WPiOS was always using the bundles on gutenberg-mobile repo and seems like that one goes back as far as 2018, so maybe that's enough?

Oh, right @ceyhun. I don't think WPiOS was ever using the bundle directly from Gutenberg's repo, only from gutenberg-mobile. I see what you mean now so yeah, no need for the native mobile (RN) bundle inside Gutenberg's repo 👍.