Should we try to implement Rust?

[chrisdebian]

Rust has many advantages, but these are only good if we can find devs that are willing/ able to use Rust.

Should we try to introduce Rust?

[manuelkasper]

I thought we'd need it for performance, but it looks like it's not really necessary after all.

[chrisdebian]

Closed, as not necessary at this time.

[dchristle]

Thinking more on Rust – current capacity is good, yes, but what if Rust makes the backend simpler and lets us add those heavy-duty matching features users want?

Node.js needs multiple forked matcher processes because it's hard to share the big trigger data structures between parallel threads in one Node process. So each of the ~4 processes hits MongoDB and rebuilds its own conditionMaps and Roaring Bitmaps every minute. That's a lot of repeated work and memory.

A single Rust matcher process could use proper threads that share one copy of that data. Build it once, update it once. That would let us keep things simpler, e.g. there's no need to migrate to change streams/sync trigger data structure deltas, if the "brute force" reload-every-minute doesn't need to happen for every worker thread.

Plus, users want regex on callsigns/gridsquares. That stuff chews CPU and doesn't map to efficient roaring bitmap data structures. A faster Rust matcher gives us the "CPU budget" for those features.

So, it's not about scaling 10x (which I am in full agreement isn't needed right now), but more about cleaner matcher core and room for more complex features. Worth another look? Could do it incrementally with gRPC/Protobuf before we even invoke an external message queue. As an aside: it is a good idea IMHO to use Protobuf from the get-go if the intent is to migrate to an external queue. The use of Protobuf is separate from the Rust question - happy to discuss additional detail elsewhere.

[chrisdebian]

Hi, David.

I think you make some excellent points, and I think that Manuel was interested in exploring Rust.

As a non-developer, I am totally sold on the advantages of Rust, but I'm not the one that will be writing code. I wonder how feasible it is to look at the current solution in a modular way, and then seeing what the options are for each part.

Main thoughts:

1. Someone has to write the code, otherwise it's purely hypothetical.
2. This is an open source project, but Manuel has the final say. If people don't like this, they can fork the code. Manuel has done brilliant work on this project.

Hope this helps.

Chris.

[manuelkasper]

@dchristle Interesting thoughts, thanks for sharing! I fully agree that a matcher written in Rust would be cleaner and more efficient. I think it's primarily a matter of priority and developer resources. Or, as @chrisdebian put it: “Someone has to write the code, otherwise it's purely hypothetical.”. My Rust is a bit rusty (ahem); it's been a few years since I last used it. But it would be a good exercise, and perhaps not that big of a task after all, especially to a seasoned Rust developer.

Another way to look at it: if we find an additional developer who writes both Rust backend code and makes beautiful modern responsive web frontends, then having that developer tackle the latter task first would probably add more user value. If said developer “only” writes Rust backend code, then rewriting the matcher would certainly be a good investment in the future.

Regarding heavy-duty matching features – at the moment I can't think of a whole lot that is really needed and would add much value to HamAlert. Regex matching on callsigns is usually requested by users who mistakenly believe they need this, when they would be much better served with DXCC or Prefix conditions. And the main issue holding us back from implementing a grid square condition is not the regex matching, but the (public/free) availability of a reliable callsign to grid square lookup database. Finally, judging from the support requests that I get via email and on the forum, I doubt that more than 5% of HamAlert users would understand what a regex is and how it is used 😂

[chrisdebian]

"Finally, judging from the support requests that I get via email and on the forum, I doubt that more than 5% of HamAlert users would understand what a regex is and how it is used 😂"

1. :-)
2. We need to stop users emailing @manuelkasper. I know we all want to help users, but it would be better if they raised a bug or issue on GitHub or the forum.

So, Rust could be a thing...