The README example includes cache-key: flatpak-builder-${{ github.sha }} since c0f26db.
Because the github.sha changes with each commit, this effectively prevents the cache from being ever restored, so dependencies are rebuilt every time.
When cache-key isn't set, the default is to use a hash of the manifest.