Conversation
t-bast
force-pushed
the
audit-db-rework
branch
from
057acbb to
a093279
Compare
We improve the following events: - `TransactionPublished` includes more details about mining fees and an optional liquidity purchase - all channel events include the latest `channel_type` - `PaymentRelayed` exposes the `relayFee` earned
We improve the `channel_events` table in the `AuditDb` by: - adding the `funding_txid` from each event - adding the `channel_type` from each event - using hex instead of blobs in sqlite - providing APIs to list these events - adding indexes on `channel_id` and `remote_node_id` The added data isn't available in past events. We decide to simply rename the older tables and start fresh: previous data will not be available from the API, but can still be queried directly in SQL if necessary.
We improve the `transactions_published` and `transactions_confirmed` tables in the `AuditDb` by: - adding more details about mining fees and feerates - adding input and output counts - using hex instead of blobs in sqlite - providing APIs to list transactions by `remote_node_id` - adding indexes for listing APIs The added data isn't available in past events. We decide to simply rename the older tables and start fresh: previous data will not be available from the API, but can still be queried directly in SQL if necessary.
We improve the `sent` and `received` payment tables in the `AuditDb` by: - adding `start_at` / `settled_at` timestamps to measure duration - adding `remote_node_id` for every channel used - using hex instead of blobs in sqlite - reordering columns and renaming them wherever useful The added data isn't available in past events. We decide to simply rename the older tables and start fresh: previous data will not be available from the API, but can still be queried directly in SQL if necessary.
We improve the `relayed` and `relayed_trampoline` payment tables in the `AuditDb` by: - unifying the way we handle multi-part payments, by having one row per payment, whether incoming or outgoing, and reconciling based on type and direction when listing events - adding `remote_node_id` for every channel used - using hex instead of blobs in sqlite - renaming columns for better clarify The added data isn't available in past events. We decide to simply rename the older tables and start fresh: previous data will not be available from the API, but can still be queried directly in SQL if necessary.
We change the relay statistics we compute, to more accurately reflect nodes that are good or bad peers and inform liquidity allocation decisions.
We now track liquidity purchases as well in transaction events, as their fees must be taken into account when evaluating the overall fees earned from a node. We also add the number of on-chain transactions made with a given peer to its relay stats.
t-bast
force-pushed
the
audit-db-rework
branch
from
a093279 to
a50fbbb
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
In this PR, we make a collection of changes to all tables of the
AuditDb. The main change is that it is way more relevant to track statistics for peer nodes instead of individual channels, so we want to track thenode_idassociated with each event. We also track more data about transactions we make and relayed payments, to more easily score peers based on the fees we're earning vs the fees we're paying (for on-chain transactions or for liquidity purchases).Each commit should be reviewed individually, as they are scoped to different tables of the DB and will make the review easier.
Note that we cannot migrate existing data (since it is lacking information that we now need), so we simply rename older tables and create new ones. Past data will thus not be accessible through the APIs, but can be queried directly using SQL if necessary. It should be fine, since liquidity decisions should be taken based on relatively recent data (a few weeks) in order to be economically relevant (nodes that generated fees months ago but aren't generating any new fees since then are probably not good peers).
Fixes #2977
Builds on top of #3246