Modular probes

[RicYaben]

I am opening this one to discuss the idea of separating modules/probes from the framework. TL;DR:

• Separate modules from the main framework (plugins-ish)
• SDK to develop probes in other languages

Motivation:

• Conversation started here
• Probes are painful to develop
• They all come bundled within ZGrab on compilation
• Limited upstream on the repo from the many forks with scattered probes. Contributors seem reluctant to submit their work
• Nmap already does this quite successfully

Open questions:

• What are the tradeoffs of moving in this direction?
• What technologies could help and how?

Examples:
Right now, zgrab2 compiles with certain probes, so if we develop a new probe, we have to compile zgrab2 with it. Nmap solves this by giving an interface for probe authors to build their own probes without touching the core of nmap, so probes become drag-and-drop binaries or scripts, in a plugin-like style.

In nmap we can do

# find all the scripts within the iot category and run them with some arguments
nmap [targets] --script 'iot' --script-args-file 'iot.conf'

The idea is to get closer to this syntax model while keeping the good features from the multiple module.

Maybe we can open different issues along with this discussion.

[Seanstoppable]

Thanks for opening this.

Warning: big brain dump as I have though about some of this in a different application a few years ago and some of it is coming back.

I did want to say from earlier that I agree that nmap "by itself" isn't good for doing internet wide scans, but I have definately used zmap + nmap and masscan + nmap to do internet wide observations across a large number of ports in an automated fashion.
But I currently complete my initial zmap independent of zgrab, which afford me the flexibility of subbing in other tools as I desire, rather than immediately forcing me to implement a probe in . I also haven't benchmarked this approach, particularly where pieces overlap, to see what overheads I am gifting to myself that may be unnecessary :)

But I definately see the benefit from the perspective of the multiple module, which I have started more heavily using as of late, as being able to do more of this in the framework of zgrab means you don't have to layer on some other layer to handle running multiple different things as part of a single scan.

In that context, I can see two sorts of solutions.
One would be more or less shell execution, which is obviously the most flexible, since you could invoke pretty much any command. Even in the event of inconcistent command line params, minimally someone could just wrap their use case up in a shell script to do a little param conversion and do whatever.
The other would be more or less like nmap, which is actually embedding (or binding) something like lua.
Or I guess you could somehow try to use go plugins, but those don't seem to have evolved much since I last looked at them.

in both cases, the API seems to be the same:
Inputs: Which are already well defined in zmap, so this would be ip, domain, port, tags
Outputs: error (if available) and result? I am assuming we probably want the zgrab to fill in some of the 'base' fields defined in

zgrab2/zgrab2_schemas/zgrab2/zgrab2.py

Line 53 in 52041e5

base_scan_response = SubRecord(

for consistency.

And then as part of processing, zgrab just needs to validate that error matches the schema, and the result if valid json, and should just fill in the bits.

In the other thread, you also mention script chaining in nmap. Do you mean chaining, or just running multiple scripts in parallel ala the multiple module?

The biggest negative I kind of see is with the usage of zcrypto. I am not sure how we'd be able to re-use that (and I feel like you absolutely want to for anything TLS based for consistency), unless zgrab/the plugin api somehow hands off the connection to the plugin (so more of the embedded lua approach), which probably has other benefits. This doesn't impact all scripts though, as not everything is over TLS.

Also curious as to what is preventing you from contributing back. Note I am not affiliated with the zmap team. For my own org it was originally the development lull, where a number of PRs languished and we ended up forking to address some functionality that was lacking, notably a modified http module and a few other things, and I am now in the opposite world where maintaining my own modules is extra effort/zgrab has fixed a bunch of things that obsolete my own code, but migrating all my own schedules to simplify is also extra work I am trying to put off.

[RicYaben]

We tested some other setups, but nothing comes close to what we need like zmap+zgrab with a pipe in between (I will release this tooling later in ~August). Our use-case is primarily identifying misconfigurations and widespread security issues, sort of "systemic vulnerabilities", so getting the full load of the communication is very important to us without needing tcpdump next to it. We did fork both tools a couple of times to play with things like multiple triggers per target, capture and reuse TCP handshakes, blocklist for zgrab2 (very much needed for UDP scans)... Not many things pin-out to be honest, but hot-swappable probes are something I am willing to dedicate time to :)

For what I have tried in other projects:

• Golang plugins are very restrictive, they only work in linux (not that anybody uses zgrab2 on windows/mac though), they have to be compiled to the same version of the primary app, and they are very heavy (they vendor third-party repos, so a probe can go into the Mb very fast)
• Embedding Lua is a good compromise in my opinion, fast and simple. The problem is the same as with nmap; every time the program touches Lua, it slows down by a lot, so all the logic needs to be loaded into zgrab2, and the Lua script should be "detached" as soon as the probe is loaded. This means that scripts cannot process things in the middle of the run, e.g., zgrab2 - target -> lua script - result -> zgrab2. We can measure this as a PoC.
• gRPC is a very good solution for this, but you need to implement a bunch of interfaces. This way probes can "pop-off" at any moment without necessarily breaking zgrab2, which may be something too? Same issues as with embedding lua though, but with the caveat that probes can be written in Rust, for example, which may be an improvement over Go. This may open other opportunities though, like middlewares. Pocketbase has a very cool way of implementing plugins.

I mentioned that nmap can chain scripts because more and more often I find myself in a situation where I scan for protocol X, and based on the data, now I want to do Y. That's why I would like to see if zgrab2 would be open to accept input in json form, so I can re-feed zgrab2 as soon as that condition is met (somewhere else down the processing pipeline). I am currently working on an engine to show how these things can be orchestrated for multiple monitoring strategies.

I also resonate with the issues of maintaining probes, and I believe that incentivizing users to contribute is the right way to go. However, the process has to be easier.

[Seanstoppable]

Cool paper. I love hearing about how people are orchestrating their scans. And as you mention it is probably annoying/time intensive for every research group that is doing internet scans to build their own orchestration from scratch.

Having read it, are you really looking for plugins, or are you looking for a way for zgrab to potentially trigger messages into your orchestration engine to drive the behavior described?

Assuming we are still on plugins:

So I had to go and look up gRPC as a plugin mechanism. I read a few things, notable:

• Plugins in Go
• go-plugin by Hashicorp
• RPC Based Plugins in Go
  Also plugin systems in general, and since you mention rust:
• Rust Plugin Systems

It certainly is interesting, and I can see its uses in a lot of projects. Ultimately it looks like you turn a standalone app into a microservices architecture? Given that each "plugin" has to run its own gRPC server. With a wrapper so that the main process starts up all of these.
This seems to turn zgrab into a plugin loader and scan orcestrator?

From my own background working with microservices, for zgrab in particular this seems to have the worst set of tradeoffs.

The main advantage is, as you say, decoupling, letting anyone write a plugin in any language as long as it speaks gRPC and implements whatever contract we define for plugins. In a traditional microservices architecture, this is usually done to split concerns at some reasonable set of service boundaries so that multiple teams can take a portion of responsibilities and iterate independently, as long as the service boundary contract is unchanged (similar to the problem you are describing)

The downsides are performance and long term maintenance.

If all scanner modules are plugins, then we encounter the following performance issues:

You are going to slow down considerably when you serialize and make a network call and back, vs invoking methods in-process. You invite higher failure rates because of the extra network calls, even if locally. Obviously requires testing, but I suspect that high volume network calls that zgrab already makes effectively doubling with a gRPC plugin layer would lead to further performance problems.

It would have additional memory and CPU overhead, because all of those things are gRPC servers. This springs to my mind, as my scans are done on pretty low specced commodity hardware.

We also gain the following maintenance challenges:

Since our app here is mostly for making network calls, it seems like all code sharing would be via using other libraries. Those libraries, when upgraded, would then need to be rolled out eventually to all the plugins.
For example, zcrypto has had a lot of recent development and fixed bugs. Notable adding new ciphers/TLS 1.3 support. Any plugin that uses TLS would need to eventually upgrade to reap those benefits. Until that happens you have a mix of zcrypto versions across your modules, leading to different behavior for differet modules than you might expect. For users, this could even mean breaking changes across a longer time span, as for example zcrypto made a breaking change in the output schema recently.
Nmap solves this by sticking to one language and providing common libraries in that language for usage. But those libraries are bundled by nmap and the plugins use the bundled version rather than their own bundled versions. Any scripts that are not contributed back to nmap have the risk of breaking if there are refactors to those libraries, but a ton of these scripts are just contributed back and maintained by the project.

This also means that any plugin implemented in a different language would not be able to use zcrypto. Maybe this is not actually a problem and maybe zcrypto's usage is mostly centered on the TLS module. But for scan result symmetry, anything returning the HandshakeLog should be compatible with the current implemention, which is derived from zcrypto. So any TLS plugin will either break the handshake log contract and do "their own thing", or developers in other languages will have to write more code to maintain that contract. Or they avoid it and just write the plugin in go.

Another example is the recent dialer changes. They either wouldn't be possible, they would only happen on the 'core' plugins, or would potentially be considered a 'breaking change' in the plugins api/shared libraries, forcing people to do perform those upgrades anyway. This goes back to if the zgrab team can try to be a bit more careful to deliniate between non-breaking changes and breaking changes, so downstreams can be on stable over a longer period of time before a breaking change is made. Having a 'stable' branch for bug fixes would be nice, it just also adds overhead in deciding 'what is a bug fix vs not' as well as makes some changes, like the dialer changes again, much more complicated if someone submits a new PR for a module to 'stable', but then it also needs re-working for the upcoming refactor.

[RicYaben]

I agree, it's a tough cookie, but I still believe this is the way to go (in some way or form). I am willing to play around with this and see what are the technical tradeoffs :)

If I had to choose one approach, it would be the Hashicorp go-plugins! But if it turns out to be significantly slower, I would be up for a hybrid setting in zgrab2?

What about having built-in modules (like until now), and support Go native plugins? There is the compiling issue and a million other compromises, but it would be a step forward. As long as users are informed of the limitations, I don't see why not?

Do you have any other suggestions? :)

Here is a small example of where we could land this: (this would be a cold-loaded plugin, only need to invoke 'Scanner' once)

package main

import (
	"github.com/zmap/zgrab2"
)

const maintainer = "awesome contributor"

type probeResult struct {
	TopicMessages map[string][]string
}

func grab(scanner *zgrab2.Scanner, target *zgrab2.ScanTarget) *zgrab2.ScanResult {
	// ScanTargets includes a context.Context and a zgrab2.Target
       result := &probeResult{}
       scanner.Result = result
	if err := scanner.Connect(target); err != nil { // returns a wrapped zgrab2.CONNECTION_ERROR
		return scanner.ScanError("failed to connect", err) // wraps scanner.Result with the error
	}
	// do something
	return scanner.Success()
}

func scan(scanner *zgrab2.Scanner, target zgrab2.ScanTarget) (result *zgrab2.ScanResult) {
	// retry with tcp, then upgrade to tls
	for scn := range scanner.RetryIterator() {
		if result = grab(scn, &target); result.Status == zgrab2.SUCCESS {
			return // return a wrapped zgrab2.SUCCESS with the banner
		}
	}
	return
}

func Scanner() zgrab2.Scanner {
	// Comes with base flags + TLS, e.g., retry-tls and use-tls
	flags := zgrab2.TLSFlagSet()
	flags.StringArray("topics", []string{"$SYS"}, "Topics to subscribe to")

	return zgrab2.Scanner{
		Name: "mqtt-topics",
		Scan: scan,
		Dialer: zgrab2.TLSDialer("tcp")
		Flags: flags,
		Description: "MQTT probe that subscribes to brokers internal topics"
	}
}

** Some report embedding Lua is very slow: https://engineering.grab.com/faster-using-the-go-plugin-to-replace-Lua-VM