Skip to content

migrate-db: automate more steps of bbolt to SQL migration process #76

@ZZiigguurraatt

Description

@ZZiigguurraatt

Right now there are a number of steps that need to be manually run before lndinit migrate-db can be run. These are mentioned at

3. (optional) Make sure to add config options like
`gc-canceled-invoices-on-startup=true` and `db.bolt.auto-compact=true` to
your `lnd.conf` to optimize the source database size by removing canceled
invoices and compacting it on startup.
4. Remove any data from the source database that you can. The fewer
entries are in the source database, the quicker the migration will complete.
For example failed payments (or their failed HTLC attempts) can be removed
with `lncli deletepayments --all`. This can make a huge difference for
routing nodes which rebalance a lot. Make sure you restart LND and compact
the db after the failed payments were deleted so it has an effect on the size
of the db.
5. (optional) Also make sure to migrate the revocation log for all channels
active prior to `lnd@0.15.0` by activating the config setting `--db.prune-revocation`.
This version introduced an optimized revocation log storage system that
reduces the storage footprint. All channels will be automatically migrated
to this new format when the setting is enabled.
6. If you have ever run the watchtower client you need to make sure before
starting the migration that you either activate it again and restart LND so
that the db is up to date (the newest migrations have been applied) or you
delete the `wtclient.db` manually so that the migration can proceed
successfully because you do not plan to use the wtclient anymore.
7. Start `lnd` normally, using the flags mentioned above but not yet changing
any database backend related configuration options. Check the log that the
database schema was migrated successfully, for example: `Checking for
schema update: latest_version=XX, db_version=XX`. This relates to the
kv-schema NOT any SQL schema, this makes sure all inital migration of the
old database were performed. This migration tool is only allowed for DBs
which have all the latest db migrations applied up to LND 19. This makes sure
that all LND nodes which want to migrate to the SQL world have the latest db
modifications in place before migrating to a different DB type. If that is
not the case the migration will be refused.
8. Stop `lnd` again and make sure it isn't started again by accident during the
data migration (e.g. disable any `systemd` or other scripts that start/stop
`lnd`).

I believe that we should make lndinit migrate-db smart enough to handle all of these steps for us automatically (inside of lndinit) without having to manually start LND extra times. Also, if they have been complete already, lndinit migrate-db should be smart enough to quietly move on and not error out.

We may also want to allow some flags to be passed to lndinit migrate-db to force the skipping of any of those steps (especially the optional ones) if we want, but we should default to them all being attempted.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions