Automate builds

[towerofnix]

@recordcrash reports that across the board, one way or another, this is probably all possible. So here's the outline!

Environment stuff

All hsmusic commands expect you to provide a data, media, and cache path.

Prepare your working directory as follows:

git clone git@github.com:hsmusic/hsmusic-wiki.git code
git clone git@github.com:hsmusic/hsmusic-data.git data
git clone git@github.com:hsmusic/hsmusic-media.git media
git clone git@github.com:hsmusic/hsmusic-lang.git lang
mkdir cache

Provide them either with environment variables or shell options:

export HSMUSIC_DATA=data
export HSMUSIC_MEDIA=media
export HSMUSIC_LANG=lang
export HSMUSIC_CACHE=cache
hsmusic # …additional options…

hsmusic \
  --data-path data --media-path media \
  --lang-path lang --cache-path cache \
  # …additional options…

For AWS commands we personally use a ~/.aws/config file. These config options may work for you and can probably be adjusted according to system and network capacity. They can probably be provided via enviroment variables or CLI options, but we don't know how.

[default]
endpoint_url = https://nyc3.digitaloceanspaces.com
s3 =
    max_concurrent_requests = 40
    max_queue_size = 20000

AWS also takes a credentials file in this format.

[default]
aws_access_key_id = …secret…
aws_secret_access_key = …secret…

Install hsmusic with npm install and npm link. Use latest Node.js LTS.

Install ImageMagick however. It's needed for thumbnail generation. It should be available in the command line via magick.

Install AWS CLI however. Homepage is here and you're looking for the CLI that can do this stuff. Or adjust for some other desired AWS S3 interface.

When to run a build?

Currently, it's complicated (on our part) to architect performing both preview and release builds, so we're only interested in automating preview builds.

Builds are desired on

• new pushed commits to preview branch on hsmusic-wiki (code repo)
• new pushed commits to preview branch on hsmusic-data (data repo)

Builds take several hours. Depending on the speed of the device, 1.5-3hrs is expected to build a single localization language. As a result, we'd like to make a schedule more along the lines of the following:

• Every day at some desired time of day (e.g. 8PM EST), check if there are any new commits since the latest build, and perform a new build
• Perform a new build manually e.g. we're done working for the day and want to have it up a little sooner

Having two builds occur at the same time, esp. on the same device, is probably a terrible idea, so preventing that would be cool. If that just means "no manual builds" then that's fine. These are preview builds, not release ones, so urgency / responsiveness isn't a big deal.

Builds should build all localization languages, but put the english one online first. This effectively means repeating the "build the website" and "publish the website" sections w/ --lang en, then --lang es, iterating over whatever languages are hard-coded or as detected from hsmusic-lang.

Process and publish media

This step comes in two parts, and the first part always precedes all other steps. The second part is actually uploading the new media files, and this can be done parallel with building the website, but it also typically takes like a couple minutes max, so if that's annoying you can just treat this whole step as one part, taken before "build the website" and "publish the website".

Pull latest media, code, and data. Then...

# generate thumbnails for new and updated original files
# this step writes `thumbnail-cache.json` (and all the thumbnails!)
hsmusic --thumbs-only

# this step writes `file-size-cache.json`
hsmusic --skip-thumbs --live-dev-server --skip-serving

# upload these two important cache files
# which wlil be read by the actual build
aws s3 cp --acl public-read cache/media-cache/thumbnail-cache.json s3://hsmusic-media/
aws s3 cp --acl public-read media/file-size-cache.json s3://hsmusic-media/

# these will be immediately be visible online:
# https://media.hsmusic.wiki/thumbnail-cache.json
# https://media.hsmusic.wiki/file-size-cache.json

Then upload the original media files and thumbnails. This is the second part and may be run in the background/parallel, but also doesn't take ages, so it's also fine to wait on it.

# upload original media files
aws s3 sync --acl public-read --exclude '.git/*' media s3://hsmusic-media

# upload thumbnail files
aws s3 sync --acl public-read cache/media-cache s3://hsmusic-media

Build the website

Pull latest code and data. The media folder must exist (be provided to hsmusic) but can just be a blank folder. If you're running on the same device then you can pass in the usual (totally populated) media folder.

# clean output directory
rm -rf out
mkdir out
# ^
# it's fine to do this between builds of different languages
# because `aws s3 sync` won't delete the files uploaded
# from preceding builds unless you use `--delete`
# (and you should not be using `--delete`)

# build one language, i.e. run one of the following
hsmusic --lang en --skip-thumbs --urls online-media --static-build
hsmusic --lang es --skip-thumbs --urls online-media --static-build
# ^ or etc. for whichever language

Publish the website

This one's easy.

aws s3 sync --acl public-read out s3://hsmusic-preview

The preview website — https://preview.hsmusic.wiki — is always serving from the same, single bucket, so uploaded files - new pages, javascript etc - will immediately be available.

As outlined in "when to run a build?", ideally the website is published as soon as the initial, English-only build is complete, with subsequent per-language builds uploaded later (when those are ready).