go-secret-hunter

A high-performance Go-based scanner that detects leaked secrets, API keys, tokens, and credentials in source code and Git history. Built for speed with concurrent workers, configurable patterns, and CI/CD integration through SARIF output.

Features

• 🔍 Multi-pattern Detection: Pre-configured patterns for AWS keys, GitHub tokens, API keys, private keys, and more
• 🧮 Entropy Analysis: Advanced Shannon entropy calculations to detect high-entropy strings
• ⚡ Concurrent Scanning: Worker pool architecture for fast parallel file processing
• 📜 Git History: Deep scanning of Git repository history using go-git
• 🎯 Low False Positives: Entropy thresholds and pattern validation reduce noise
• 🚫 Allowlisting: Support for allowlisted secrets via configuration or files
• 📁 Ignore Patterns: Flexible file and directory exclusion (node_modules, vendor, etc.)
• 📊 Multiple Output Formats: Text, JSON, and SARIF for CI/CD integration
• ⚙️ Configurable: YAML/JSON configuration files with custom pattern support
• 🚀 CI/CD Ready: SARIF output integrates with GitHub Security, Azure DevOps, and more

Installation

Build from Source

git clone https://github.com/BaseMax/go-secret-hunter.git
cd go-secret-hunter
go build -o secret-hunter ./cmd/secret-hunter

Using Go Install

go install github.com/BaseMax/go-secret-hunter/cmd/secret-hunter@latest

Quick Start

Basic Scan

Scan current directory for secrets:

secret-hunter

Scan Specific Paths

secret-hunter --paths=/path/to/code,/another/path

Include Git History

secret-hunter --git --verbose

Output to SARIF (for CI/CD)

secret-hunter --format=sarif --output=secrets-report.sarif

Using Configuration File

# Generate example config
secret-hunter --generate-config=config.yaml

# Run with config
secret-hunter --config=config.yaml

Command-Line Options

Flag	Description	Default
--config	Path to configuration file (YAML or JSON)	-
--paths	Comma-separated paths to scan	.
--format	Output format: text, json, sarif	text
--output	Output file (default: stdout)	-
--workers	Number of concurrent workers	4
--verbose	Verbose output	false
--git	Include Git history scanning	false
--allowlist	Path to allowlist file	-
--version	Show version	-
--generate-config	Generate example config file	-

Configuration File

Example config.yaml:

paths:
  - .
  - src/
include_git: true
max_depth: -1
workers: 8

ignore_patterns:
  - node_modules/
  - .git/
  - vendor/
  - '*.min.js'
  - '*.lock'
  - '*.log'

allowed_secrets:
  - "EXAMPLE_NOT_A_REAL_SECRET"

file_extensions:
  - .go
  - .py
  - .js
  - .ts
  - .java
  - .env
  - .yaml
  - .json

output_format: sarif
output_file: secrets-report.sarif
verbose: true

custom_patterns:
  - name: Custom API Key
    pattern: 'custom_api_key[:=]\s*[''"]?([a-zA-Z0-9]{32})[''"]?'
    description: Custom API Key Pattern
    entropy: 3.5

Detected Secret Types

Built-in Patterns

• AWS Access Keys (AKIA...)
• AWS Secret Keys
• GitHub Tokens (ghp_...)
• GitHub OAuth Tokens
• Google API Keys (AIza...)
• Google OAuth Client IDs
• Slack Tokens (xox...)
• Slack Webhooks
• Stripe API Keys
• Heroku API Keys
• Twilio API Keys
• NPM Tokens
• PyPI Tokens
• JWT Tokens
• Private Keys (RSA, SSH, PGP)
• Azure Client Secrets
• Docker Auth Tokens
• MailChimp API Keys
• MailGun API Keys
• Generic API Keys
• Generic Secrets/Passwords
• Generic Tokens

Custom Patterns

Add your own patterns in the configuration file:

custom_patterns:
  - name: My Custom Secret
    pattern: 'my_secret[:=]\s*[''"]?([a-zA-Z0-9]{20,})[''"]?'
    description: Organization-specific secret pattern
    entropy: 4.0  # Minimum entropy (0 = no entropy check)

Entropy Analysis

The scanner uses Shannon entropy to detect high-entropy strings that may be secrets:

• Entropy Score: Calculated as bits per character (0-8 scale)
• Thresholds: Configurable per pattern
• False Positive Reduction: Filters repeated patterns ("aaaa", "1111")

Allowlisting

Via Configuration File

allowed_secrets:
  - "NOT_A_REAL_SECRET"
  - "EXAMPLE_KEY_FOR_TESTS"

Via Allowlist File

Create a file with one entry per line:

# allowlist.txt
NOT_A_REAL_SECRET
EXAMPLE_KEY_FOR_TESTS
test_api_key_12345

Use it:

secret-hunter --allowlist=allowlist.txt

Output Formats

Text (Human-Readable)

Found 3 potential secret(s):

[1] AWS Access Key
    File: src/config.go:15:10
    Description: AWS Access Key ID
    Match: AKIAIOSFODNN7EXAMPLE
    Entropy: 3.68
    Context: awsKey := "AKIAIOSFODNN7EXAMPLE"

JSON

[
  {
    "type": "AWS Access Key",
    "description": "AWS Access Key ID",
    "file": "src/config.go",
    "line": 15,
    "column": 10,
    "match": "AKIAIOSFODNN7EXAMPLE",
    "context": "awsKey := \"AKIAIOSFODNN7EXAMPLE\"",
    "entropy": 3.68
  }
]

SARIF (Static Analysis Results Interchange Format)

Integrates with:

• GitHub Code Scanning
• Azure DevOps Security
• GitLab Security Dashboard
• Jenkins
• Many other CI/CD tools

Example usage with GitHub Actions:

- name: Run Secret Scanner
  run: secret-hunter --format=sarif --output=secrets.sarif

- name: Upload SARIF
  uses: github/codeql-action/upload-sarif@v2
  with:
    sarif_file: secrets.sarif

CI/CD Integration

GitHub Actions

name: Secret Scanning
on: [push, pull_request]

jobs:
  scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
        with:
          fetch-depth: 0  # Get full history for Git scanning
      
      - name: Set up Go
        uses: actions/setup-go@v4
        with:
          go-version: '1.21'
      
      - name: Install Secret Hunter
        run: go install github.com/BaseMax/go-secret-hunter/cmd/secret-hunter@latest
      
      - name: Scan for Secrets
        run: secret-hunter --git --format=sarif --output=secrets.sarif
      
      - name: Upload Results
        uses: github/codeql-action/upload-sarif@v2
        if: always()
        with:
          sarif_file: secrets.sarif

GitLab CI

secret_scan:
  stage: security
  image: golang:1.21
  script:
    - go install github.com/BaseMax/go-secret-hunter/cmd/secret-hunter@latest
    - secret-hunter --git --format=json --output=secrets.json
  artifacts:
    reports:
      secret_detection: secrets.json

Performance

• Concurrent Workers: Adjustable worker pool (default: 4)
• Binary Detection: Automatically skips binary files
• Smart Filtering: Ignore patterns reduce unnecessary scans
• Efficient Git: Uses go-git for optimized repository traversal

Example performance:

• ~1000 files/second on typical hardware
• Full Git history scan: depends on repository size
• Memory usage: ~50-100MB for most projects

Best Practices

1. Use Allowlisting Carefully: Only allowlist confirmed false positives
2. Scan Git History: Use --git flag to catch historical leaks
3. CI/CD Integration: Run on every commit and PR
4. Configure Ignores: Exclude build artifacts, dependencies
5. Custom Patterns: Add organization-specific secret patterns
6. Review Findings: Manually verify high-priority findings
7. Rotate Secrets: Immediately rotate any exposed credentials

Exit Codes

• 0: No secrets found
• 1: Secrets found or error occurred

Development

Build

go build -o secret-hunter ./cmd/secret-hunter

Run Tests

go test ./...

Add Custom Patterns

Edit internal/patterns/patterns.go to add new built-in patterns.

Contributing

Contributions are welcome! Please:

1. Fork the repository
2. Create a feature branch
3. Add tests for new functionality
4. Submit a pull request

License

This project is licensed under the MIT License - see the LICENSE file for details.

Security

If you discover a security issue, please email security@example.com instead of using the issue tracker.

Acknowledgments

• Built with go-git for Git integration
• Inspired by tools like truffleHog, gitleaks, and git-secrets

Support

• 📫 Issues: GitHub Issues
• 💬 Discussions: GitHub Discussions

---

⚠️ Important: This tool helps detect secrets but is not foolproof. Always follow security best practices and never commit secrets to version control.