feat: add auto-wildcard detection flag#935
feat: add auto-wildcard detection flag#935erdogan98 wants to merge 3 commits intoprojectdiscovery:devfrom
Conversation
Implements automatic wildcard detection similar to PureDNS (projectdiscovery#924). New flags: - --auto-wildcard (-aw): Enable automatic wildcard detection and filtering - --auto-wildcard-tests: Number of random subdomain tests (default 3) How it works: 1. For each domain being resolved, extracts parent domains 2. Tests parent domains with random subdomains (using xid for randomness) 3. If random subdomains resolve to same IPs, marks domain as wildcard 4. Filters out any resolved subdomains matching wildcard IPs This approach is similar to PureDNS wildcard detection and helps reduce false positives when scanning domains with wildcard DNS records.
WalkthroughAdds an AutoWildcardDetector that performs concurrent, per-parent wildcard DNS detection at runtime, integrates it into the Runner to filter wildcard-derived results during processing, and exposes flags to enable/configure the behavior and report detected roots and filtered counts. Changes
Sequence DiagramsequenceDiagram
participant Runner
participant Detector as AutoWildcardDetector
participant DNSX
participant Output
Runner->>Detector: New(dnsxClient, testCount)
activate Detector
Detector->>Detector: init maps (wildcardRoots, testedDomains, inFlight)
deactivate Detector
loop For each domain
Runner->>Detector: DetectAndFilter(host, hostIPs)
activate Detector
Detector->>Detector: getParentDomains(host)
loop For each parent
Detector->>Detector: ensureWildcardTested(parent)
alt Parent not tested
Detector->>DNSX: Query multiple random subdomains
activate DNSX
DNSX-->>Detector: A/AAAA records (IPs)
deactivate DNSX
Detector->>Detector: aggregate wildcard IPs for parent
end
end
Detector->>Detector: isWildcardMatch(host, hostIPs)
alt Match found
Detector->>Detector: IncrementFilteredCount()
Detector-->>Runner: true (filter)
else No match
Detector-->>Runner: false (process)
end
deactivate Detector
end
Runner->>Detector: GetWildcardRoots() / GetFilteredCount()
Detector-->>Runner: results
Runner->>Output: Print summary (root count, filtered count)
Estimated Code Review Effort🎯 3 (Moderate) | ⏱️ ~22 minutes Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Fix all issues with AI agents
In `@internal/runner/autowildcard.go`:
- Around line 168-188: getParentDomains currently includes public suffixes like
"co.uk" as parents; change it to compute the registrable domain using
golang.org/x/net/publicsuffix.EffectiveTLDPlusOne and stop generating parents at
that boundary so you never return the public suffix itself. Update
getParentDomains to call publicsuffix.EffectiveTLDPlusOne(host) (handle and log
the error case) to get the base registrable domain, derive parent labels only up
to that registrable domain, and import the publicsuffix package; ensure any
error falls back to the current behavior but avoids appending pure public
suffixes like "co.uk".
🧹 Nitpick comments (2)
internal/runner/autowildcard.go (1)
86-113: Consider simplifying the IP collection logic.The
ipCountsmap andcount >= 1check are redundant. Since any IP that resolves for a randomly-generated subdomain indicates a wildcard, you could simplify to directly add IPs towildcardIPs.♻️ Simplified implementation
func (d *AutoWildcardDetector) testWildcard(parent string) map[string]struct{} { wildcardIPs := make(map[string]struct{}) - ipCounts := make(map[string]int) // Query multiple random subdomains for i := 0; i < d.testCount; i++ { randomHost := xid.New().String() + "." + parent result, err := d.dnsx.QueryOne(randomHost) if err != nil || result == nil { continue } for _, ip := range result.A { - ipCounts[ip]++ + wildcardIPs[ip] = struct{}{} } } - // An IP is considered a wildcard if it appears in at least one random subdomain query - // (if a random subdomain resolves, it indicates wildcard) - for ip, count := range ipCounts { - if count >= 1 { - wildcardIPs[ip] = struct{}{} - } - } - return wildcardIPs }internal/runner/runner.go (1)
756-763: Consider also checking AAAA records for wildcard detection.Currently, wildcard detection only uses A records (
dnsData.A). If a wildcard domain is configured with only AAAA records (IPv6), it won't be detected or filtered. Consider extending the detection to include AAAA records if the--aaaaflag is enabled or if the user expects IPv6 wildcard detection.♻️ Example extension to include AAAA records
// auto wildcard detection and filtering if r.options.AutoWildcard && r.autoWildcardDetector != nil { - if r.autoWildcardDetector.DetectAndFilter(domain, dnsData.A) { + // Combine A and AAAA records for wildcard detection + allIPs := append([]string{}, dnsData.A...) + allIPs = append(allIPs, dnsData.AAAA...) + if r.autoWildcardDetector.DetectAndFilter(domain, allIPs) { r.autoWildcardDetector.IncrementFilteredCount() gologger.Debug().Msgf("Filtered wildcard subdomain: %s\n", domain) continue } }Note: The
testWildcardmethod inautowildcard.gowould also need to collect AAAA records from DNS responses for this to work fully.
- Use publicsuffix.EffectiveTLDPlusOne to stop at registrable domain (prevents treating public suffixes like 'co.uk' as wildcard parents) - Simplify testWildcard(): remove redundant ipCounts map, add IPs directly - Add AAAA record support for IPv6 wildcard detection
|
Thanks for the review feedback! I've addressed all the points:
Ready for another review! |
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Fix all issues with AI agents
In `@internal/runner/autowildcard.go`:
- Around line 87-110: The testWildcard function currently unions IPs from each
successful random-subdomain probe, causing false positives; change it to collect
the IP set for each successful probe (using d.dnsx.QueryOne in
AutoWildcardDetector), track only successful probes, compute the intersection of
IP sets across probes, and only declare wildcard IPs if there are at least two
successful probes and the intersection is non-empty; return that intersection
map[string]struct{} instead of the current union and keep using d.testCount to
drive probe attempts.
- Around line 57-85: ensureWildcardTested currently sets testedDomains before
calling testWildcard which allows other goroutines to skip testing and read
incomplete wildcardRoots; change the flow to track in-flight tests: add an
in-flight map (e.g., inflight map[string]chan struct{} or similar) guarded by
d.mutex, so when a goroutine wants to test parent it first checks testedDomains,
then checks inflight—if an inflight channel exists it waits for it to close and
returns (or reads the completed wildcardRoots), otherwise it creates and stores
a new channel in inflight, releases the lock, runs d.testWildcard(parent), then
re-acquires the lock to set d.testedDomains[parent], set d.wildcardRoots[parent]
if any, close the inflight channel and delete the entry; update
ensureWildcardTested to use these symbols (testedDomains, inflight,
wildcardRoots, testWildcard, d.mutex) and handle locking/unlocking around waits
and updates.
…se positives 1. Fix race condition in ensureWildcardTested: - Added inFlight map to track domains currently being tested - Concurrent goroutines now wait for in-flight tests to complete - testedDomains is now set AFTER testWildcard completes - Prevents race where wildcardRoots isn't populated yet 2. Fix false positives in testWildcard: - Changed from union-based to intersection-based IP collection - Track IP sets per successful probe separately - Require at least 2 successful probes before declaring wildcard - Only IPs consistent across ALL probes are considered wildcard IPs - Added intersectIPSets helper function These changes ensure: - Thread-safe wildcard detection with proper synchronization - Reduced false positives by requiring consistent DNS responses
CodeRabbit Feedback Addressed ✅I've pushed commit f06d2ed to address the two major issues identified: 1. Concurrency Bug Fix (ensureWildcardTested)Problem: Solution:
2. False Positives Fix (testWildcard)Problem: Union-based IP collection was wrong - if probes returned different IPs, all were considered wildcards. Solution:
These changes ensure thread-safe wildcard detection with proper synchronization and significantly reduce false positives by requiring consistent DNS responses across multiple probes. |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Fix all issues with AI agents
In `@internal/runner/autowildcard.go`:
- Around line 23-35: NewAutoWildcardDetector currently clamps testCount to 3
only when <1, but testWildcard requires at least 2 probes so passing 1 silently
disables detection; update the clamping in NewAutoWildcardDetector to ensure
testCount is at least 2 (not 1) — i.e. if testCount < 2 set testCount = 2 — and
keep references to dnsx, testCount, wildcardRoots, testedDomains, and inFlight
initialization unchanged so testWildcard receives a usable probe count.
| // NewAutoWildcardDetector creates a new auto wildcard detector | ||
| func NewAutoWildcardDetector(dnsxClient *dnsx.DNSX, testCount int) *AutoWildcardDetector { | ||
| if testCount < 1 { | ||
| testCount = 3 | ||
| } | ||
| return &AutoWildcardDetector{ | ||
| dnsx: dnsxClient, | ||
| testCount: testCount, | ||
| wildcardRoots: make(map[string]map[string]struct{}), | ||
| testedDomains: make(map[string]struct{}), | ||
| inFlight: make(map[string]chan struct{}), | ||
| } | ||
| } |
There was a problem hiding this comment.
Clamp testCount to ≥2 to avoid a silent no‑op.
testWildcard requires at least two successful probes, so a value of 1 can never succeed and effectively disables detection.
🛠️ Proposed fix
- if testCount < 1 {
+ if testCount < 2 {
testCount = 3
}🤖 Prompt for AI Agents
In `@internal/runner/autowildcard.go` around lines 23 - 35,
NewAutoWildcardDetector currently clamps testCount to 3 only when <1, but
testWildcard requires at least 2 probes so passing 1 silently disables
detection; update the clamping in NewAutoWildcardDetector to ensure testCount is
at least 2 (not 1) — i.e. if testCount < 2 set testCount = 2 — and keep
references to dnsx, testCount, wildcardRoots, testedDomains, and inFlight
initialization unchanged so testWildcard receives a usable probe count.
dogancanbakir
requested review from
Ice3man543,
Mzack9999,
dwisiswant0 and
tarunKoyalwar
and removed request for
dogancanbakir
|
I think we should try to extend or import the logic from https://github.com/projectdiscovery/shuffledns/tree/dev/pkg/wildcards which already has a fairly stable wildcard implementation rather than re-implementing it? cc @dogancanbakir |
2 participants
/claim #924
Summary
Implements automatic wildcard detection similar to PureDNS, as requested in #924.
New Flags
--auto-wildcard(-aw): Enable automatic wildcard detection and filtering--auto-wildcard-tests: Number of random subdomain tests (default 3)How It Works
Usage
Implementation Details
-debugflagTesting
Tested with various wildcard domains and verified:
Closes #924
Summary by CodeRabbit
New Features
Notes