switchrpc: improve TrackOnion error handling#10472
switchrpc: improve TrackOnion error handling#10472calvinrzachman wants to merge 2 commits intolightningnetwork:elle-base-branch-payment-servicefrom
Conversation
Summary of ChangesHello @calvinrzachman, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request significantly refactors the error handling mechanism for the Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request significantly improves error handling for the TrackOnion RPC by introducing structured error messages using protobuf's oneof feature. This is a great improvement over the previous string-based error reporting, making the API more robust and easier for clients to consume. The changes correctly separate RPC-level errors (communicated via gRPC status codes) from application-level payment failures. The implementation is solid, but I found one issue where the ErrorCode field is not being set for certain failure types, which I've commented on. Overall, this is a high-quality change that enhances the switchrpc API.
|
This leads to a code structure in the RPC client code like: switch result := trackResp.Result.(type) {
case *switchrpc.TrackOnionResponse_Preimage:
preimage, err := lntypes.MakePreimage(result.Preimage)
if err != nil {
// This is a data corruption error from the server. It
// is NOT a definitive payment failure.
}
// Do something with pre-image...
case *switchrpc.TrackOnionResponse_FailureDetails:
details := result.FailureDetails
translatedErr, err := switchrpc.UnmarshallFailureDetails(
details, deobfuscator,
)
// Do something to handle attempt result (either settle/fail or retry if uncertain) |
calvinrzachman
force-pushed
the
switchrpc-error-handle
branch
from
5575fec to
a70b135
Compare
ziggie1984
force-pushed
the
elle-base-branch-payment-service
branch
from
307e665 to
3e0967d
Compare
calvinrzachman
force-pushed
the
switchrpc-error-handle
branch
from
a70b135 to
7b98465
Compare
| hex.EncodeToString(buf.Bytes())), nil | ||
| // marshallFailureDetails creates the FailureDetails message for the | ||
| // TrackOnion response body. | ||
| func marshallFailureDetails(err error) *FailureDetails { |
There was a problem hiding this comment.
The code repetition here and in translageErrorForRPC isn't ideal, but I like the clarity of using one_of. Mabye we could do a similar approach in the SendOnion error to also add a ClearTextFailure.
There was a problem hiding this comment.
Yep, for sure. I have a similar PR incoming which updates the way SendOnion rpc communicates error information. It will remove translateErrorForRPC. Just wanted to keep the PRs as small as possible to facilitate review.
There was a problem hiding this comment.
One thing to consider is that SendOnion semantics differ a bit from TrackOnion. For TrackOnion, the gRPC request can itself be successful, but deliver information to the client about the failure of the given attempt ID being queried. For such failures, it makes sense to transport those via the RPC proto response message.
The SendOnion PR will likely move away from transporting error information via the proto response message. Instead we can reserve gRPC status status.OK for successful dispatches only and then transport structured error information via Details as mentioned in these docs:
I think this has added benefit to observability in that we'll be able to track gRPC failures/non-OK status for SendOnion endpoint more clearly to see if the remote router is having issues communicating with Switch RPC servers.
|
Before we were good to have separate commits for proto definitions and implementation. Now that the |
lnrpc/switchrpc/switch.proto
Outdated
| // The ErrorCode provides a high-level classification of the failure. | ||
| ErrorCode error_code = 1; |
There was a problem hiding this comment.
I wonder if we can get rid of this error code as it makes it harder to use the API. Perhaps this could be an alternative approach https://github.com/bitromortac/lnd/tree/re-10472-upstream-260108-17, not sure how much it interferes with the grpc error status based approach for SendOnion. This way we'd only need to switch on the failure details.
There was a problem hiding this comment.
Thanks for the exploration of this alternate approach. After trying a few things on the rpc client side, I expect the ErrorCode will be the primary discriminator used by SendOnion rpc client. The switch statements are a bit less verbose when done over an integer like ErrorCode than a proto Message type. I think it can also improve the forward compatibility a bit in that new definitive failure types (e.g., a new LinkError) can be added within the same code/category without breaking older clients.
calvinrzachman
force-pushed
the
switchrpc-error-handle
branch
from
c1cbfdc to
3a92256
Compare
|
@calvinrzachman, remember to re-request review from reviewers when ready |
calvinrzachman
force-pushed
the
switchrpc-error-handle
branch
from
3a92256 to
7f49bb4
Compare
🟠 PR Severity: HIGH
🟠 High (2 files)
🟢 Low (5 files)
AnalysisThis PR improves error handling in the
The PR does not warrant a bump to CRITICAL as it:
Recommendation: Requires review by an engineer knowledgeable in the RPC layer and payment tracking subsystems to ensure error handling changes maintain backward compatibility and don't introduce regressions in client error handling. To override, add a |
calvinrzachman
force-pushed
the
switchrpc-error-handle
branch
2 times, most recently
from
c24c471 to
3372d12
Compare
🟠 PR Severity: HIGH
🟠 High (1 file)
🟡 Medium (1 file)
🟢 Low (2 files)
⚪ Auto-generated (3 files, excluded from analysis)
AnalysisThis PR improves error handling in the
The PR does not meet the criteria for a severity bump (would need >20 files or >500 non-excluded lines changed). The majority of the line changes are in auto-generated protobuf code, which is expected. Recommendation: This PR should be reviewed by an engineer familiar with the switchrpc interface and error handling patterns in lnd's RPC layer. To override, add a |
calvinrzachman
force-pushed
the
switchrpc-error-handle
branch
from
3372d12 to
61b60cc
Compare
🟠 PR Severity: HIGH
🟠 High (2 files)
🟢 Low (3 files)
📦 Auto-generated (2 files, excluded from severity)
AnalysisThis PR is classified as HIGH severity because it modifies RPC API definitions and server implementations in the Key changes:
No severity bump applied: The PR has only 2 substantive files (excluding tests and auto-generated code) with 339 lines changed, which doesn't meet the bump thresholds (>20 files or >500 lines). Recommendation: This should be reviewed by an engineer familiar with the Lightning RPC layer and onion routing functionality to ensure the error handling changes maintain API compatibility and correctness. To override, add a |
bitromortac
left a comment
bitromortac
left a comment
There was a problem hiding this comment.
I think this is a nice improvement as the errors we encounter can be manifold 🙏.
lnrpc/switchrpc/switch_server.go
Outdated
| log.Criticalf("Payment %v completed without a valid preimage "+ | ||
| "or error", hash) |
There was a problem hiding this comment.
Critical will lead to a shutdown. Not sure we want that
There was a problem hiding this comment.
Ahh my mistake. Definitely don't want a shut down. Updated ✅
| // An internal error occurred. | ||
| InternalError internal_error = 6; |
There was a problem hiding this comment.
Does it make sense to get rid of that and use the internal error status code instead, so maybe add a return error to marshallFailureDetails?
calvinrzachman
force-pushed
the
switchrpc-error-handle
branch
from
61b60cc to
576257c
Compare
🟠 PR Severity: HIGH
🟠 High (2 files)
🟡 Medium (0 files)None 🟢 Low (1 file)
AnalysisThis PR is classified as HIGH severity because it modifies the switchrpc RPC server implementation (
The PR touches RPC/API definitions and server implementations, which fall under the HIGH severity category per the classification rules. The changes improve error handling for the TrackOnion RPC, which is important for payment tracking reliability. No severity bump applied: The PR has 3 non-test/non-generated files and approximately 340 lines changed, both below the thresholds for bumping (20 files or 500 lines). Review Focus: Reviewers should pay attention to backward compatibility of the API changes, error handling completeness, and proper error propagation in the TrackOnion implementation. To override, add a |
The new structure uses a top-level `oneof` to provide a compile-time distinction between a successful payment (preimage) and a failed one. Additional information on a failed attempt can be found in FailureDetails. We now also use a structured ForwardingFailure type for communicating the failure index and wire message from failures which occur during htlc forwarding downstream in the route.
calvinrzachman
force-pushed
the
switchrpc-error-handle
branch
from
576257c to
77c3327
Compare
🟠 PR Severity: HIGH
🟠 High (2 files)
🟡 Medium (0 files)None 🟢 Low (5 files)
AnalysisThis PR improves error handling in the TrackOnion RPC endpoint. The changes are focused in the Key considerations:
Review focus areas:
To override, add a |
3 participants
Change Description
Following up on some review comments from #9489, we explore an update to the communication of error information via
TrackOnionResponse.TrackOnionResponsethrough a top-leveloneof.ForwardingError, instead opting for structured protobuf messages (FailureDetails) to transport rich error details without loss of information.oneoffailure) withinFailureDetailsto carry specific failure types:ForwardingFailure,ClearTextFailure, or rawencrypted_error_databytes.TrackOnionquery itself (e.g., attempt not found, server unavailable) via standard gRPC status codes (codes.NotFound,codes.Unavailable), cleanly separating them from application-level payment outcomes in the response body.NOTE: This is a breaking change for clients. Though given that this RPC is not included in any release, this seems perfectly fine. A follow-up PR is planned for SendOnion.
Steps to Test
make itest icase=track_oniongo test -v -timeout 30s -tags switchrpc -run ^TestTrackOnion$ github.com/lightningnetwork/lnd/lnrpc/switchrpcgo test -v -timeout 30s -tags switchrpc github.com/lightningnetwork/lnd/lnrpc/switchrpc