Add CI validation and project documentation by jontsai · Pull Request #1 · jontsai/claude-plugins

jontsai · 2026-01-23T06:48:26Z

Summary

Add GitHub Actions workflow to validate plugin structure on push/PR
Add local validation scripts for pre-push testing
Add project documentation (CONTRIBUTING, AGENTS, TESTING, CLAUDE)
Update README with CI badge and documentation links

CI Validation

Validates on every push/PR to main:

JSON syntax for all .json files
marketplace.json required fields (name, owner, plugins)
Plugin structure (plugin.json exists, skills have SKILL.md)
SKILL.md frontmatter (name, description required)
hooks.json format (object not array, valid event names)

Documentation

File	Purpose
`CONTRIBUTING.md`	Contribution workflow, plugin structure guide
`AGENTS.md`	Guidelines for LLMs working on this codebase
`TESTING.md`	Validation and testing instructions
`CLAUDE.md`	Quick reference pointing to other docs

Files Added

scripts/
├── validate.sh              # Local runner
├── validate_json.py         # JSON syntax
├── validate_marketplace.py  # Marketplace + plugin structure
├── validate_skills.py       # SKILL.md frontmatter
└── validate_hooks.py        # hooks.json format

CONTRIBUTING.md
AGENTS.md
TESTING.md
CLAUDE.md

Test Plan

Run ./scripts/validate.sh locally - all checks pass
Verify GitHub Actions workflow runs on PR

Generated with Claude Code

Copilot

Pull request overview

This PR adds comprehensive CI validation and project documentation to ensure plugin quality and consistency. It introduces automated validation through GitHub Actions and provides clear guidelines for contributors and AI agents working on the codebase.

Changes:

Added GitHub Actions workflow to validate plugin structure on push/PR to main
Created 5 validation scripts (JSON syntax, marketplace structure, SKILL.md frontmatter, hooks.json format, and a runner script)
Added 4 documentation files (CONTRIBUTING.md, AGENTS.md, TESTING.md, CLAUDE.md)
Updated README with CI badge and documentation links

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
`.github/workflows/validate.yml`	GitHub Actions workflow that runs validation scripts on push/PR
`scripts/validate.sh`	Bash script runner that executes all validation checks locally
`scripts/validate_json.py`	Validates JSON syntax for all `.json` files in repository
`scripts/validate_marketplace.py`	Validates marketplace.json structure and plugin directory structure
`scripts/validate_skills.py`	Validates SKILL.md files have proper YAML frontmatter
`scripts/validate_hooks.py`	Validates hooks.json files have correct format and event names
`CONTRIBUTING.md`	Contribution guidelines with plugin structure requirements
`TESTING.md`	Testing and validation instructions
`AGENTS.md`	Guidelines for LLMs working on the codebase
`CLAUDE.md`	Quick reference pointing to other documentation
`README.md`	Updated with CI badge and documentation links

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-01-23T07:20:27Z

scripts/validate_marketplace.py

+    with open(".claude-plugin/marketplace.json") as f:
+        data = json.load(f)


The script doesn't handle file reading errors (e.g., encoding issues, permission errors, missing file). If marketplace.json is missing or has encoding issues, the script will crash with an unhandled exception rather than providing a clear validation error message. Consider wrapping the file operations and JSON parsing in a try-except block to catch and report these errors gracefully.

Suggested change

with open(".claude-plugin/marketplace.json") as f:

data = json.load(f)

try:

with open(".claude-plugin/marketplace.json", encoding="utf-8") as f:

data = json.load(f)

except FileNotFoundError:

print("Error: .claude-plugin/marketplace.json not found.")

sys.exit(1)

except PermissionError:

print("Error: Permission denied when reading .claude-plugin/marketplace.json.")

sys.exit(1)

except (OSError, UnicodeDecodeError) as e:

print(f"Error: Failed to read .claude-plugin/marketplace.json: {e}")

sys.exit(1)

except json.JSONDecodeError as e:

print(f"Error: Invalid JSON in .claude-plugin/marketplace.json: {e}")

sys.exit(1)

.github/workflows/validate.yml

Copilot · 2026-01-23T07:20:28Z

scripts/validate_json.py

+    result = subprocess.run(
+        ["find", ".", "-name", "*.json", "-not", "-path", "./.git/*"],
+        capture_output=True,
+        text=True,
+    )


The subprocess.run call uses shell=False (implicitly), which is correct for security. However, the script relies on the find command being available, which may not be on all systems (particularly Windows without WSL). Consider adding a fallback using Python's pathlib or os.walk for better cross-platform compatibility, or at minimum, document the dependency on Unix-like systems.

TESTING.md

Copilot · 2026-01-23T07:20:28Z

scripts/validate_json.py

+    files = result.stdout.strip().split("\n")
+    errors = 0
+
+    for f in files:
+        if not f:
+            continue


The validation script does not handle empty file list correctly. If find returns an empty result, files will be [''] (a list with one empty string), not an empty list. The check if not f: on line 19 handles this, but it's inefficient to iterate over a list with an empty string. Consider filtering the list: files = [f for f in result.stdout.strip().split("\n") if f] to avoid unnecessary iterations.

Suggested change

files = result.stdout.strip().split("\n")

errors = 0

for f in files:

if not f:

continue

files = [f for f in result.stdout.strip().split("\n") if f]

errors = 0

for f in files:

Copilot · 2026-01-23T07:20:29Z

scripts/validate_skills.py

+                frontmatter = match.group(1)
+                if "name:" not in frontmatter:
+                    errors.append(f"{path}: Missing 'name' in frontmatter")
+                if "description:" not in frontmatter:
+                    errors.append(f"{path}: Missing 'description' in frontmatter")


The validation of frontmatter fields using string matching ("name:" not in frontmatter) is fragile and could produce false negatives. For example, if a SKILL.md has name: with no space after the colon, or if there's a YAML comment containing the word "name:", the validation would incorrectly pass or fail. Consider parsing the frontmatter as actual YAML using the yaml module (e.g., import yaml; parsed = yaml.safe_load(frontmatter)) and checking for the presence of keys in the parsed dictionary.

Copilot · 2026-01-23T07:20:29Z

scripts/validate_skills.py

+                with open(path) as fp:
+                    content = fp.read()


The script doesn't handle file reading errors (e.g., encoding issues, permission errors). If a SKILL.md file has encoding issues or is unreadable, the script will crash with an unhandled exception rather than reporting it as a validation error. Consider wrapping the file operations in a try-except block to catch and report these errors gracefully.

Suggested change

with open(path) as fp:

content = fp.read()

try:

with open(path, encoding="utf-8") as fp:

content = fp.read()

except (OSError, UnicodeError) as exc:

errors.append(f"{path}: Unable to read file ({exc})")

continue

Copilot · 2026-01-23T07:20:29Z

scripts/validate_marketplace.py

+            for skill_name in os.listdir(skills_dir):
+                skill_path = os.path.join(skills_dir, skill_name)
+                if os.path.isdir(skill_path):
+                    skill_md = os.path.join(skill_path, "SKILL.md")
+                    if not os.path.exists(skill_md):
+                        errors.append(
+                            f"Plugin '{name}': skill '{skill_name}' missing SKILL.md"
+                        )


The script doesn't handle potential OS errors when listing directories. If os.listdir(skills_dir) fails due to permission errors or other OS-level issues, the script will crash with an unhandled exception. Consider wrapping the directory operations in a try-except block to handle these cases gracefully and report them as validation errors.

Suggested change

for skill_name in os.listdir(skills_dir):

skill_path = os.path.join(skills_dir, skill_name)

if os.path.isdir(skill_path):

skill_md = os.path.join(skill_path, "SKILL.md")

if not os.path.exists(skill_md):

errors.append(

f"Plugin '{name}': skill '{skill_name}' missing SKILL.md"

)

try:

skill_entries = os.listdir(skills_dir)

except OSError as exc:

errors.append(

f"Plugin '{name}': cannot list skills directory '{skills_dir}': {exc}"

)

else:

for skill_name in skill_entries:

skill_path = os.path.join(skills_dir, skill_name)

if os.path.isdir(skill_path):

skill_md = os.path.join(skill_path, "SKILL.md")

if not os.path.exists(skill_md):

errors.append(

f"Plugin '{name}': skill '{skill_name}' missing SKILL.md"

)

Copilot · 2026-01-23T07:20:30Z

scripts/validate_skills.py

+                    errors.append(f"{path}: Missing YAML frontmatter")
+                    continue
+
+                match = re.match(r"^---\n(.*?)\n---", content, re.DOTALL)


The regex pattern assumes the frontmatter ends with exactly \n--- but doesn't account for possible trailing whitespace or different line endings. Files with \r\n (Windows line endings) or trailing spaces after --- would fail validation incorrectly. Consider using a more flexible pattern like r"^---\s*\n(.*?)\n---\s*" to handle these edge cases.

Suggested change

match = re.match(r"^---\n(.*?)\n---", content, re.DOTALL)

match = re.match(r"^---\s*\n(.*?)\n---\s*", content, re.DOTALL)

Copilot · 2026-01-23T07:20:30Z

scripts/validate_hooks.py

+                with open(path) as fp:
+                    data = json.load(fp)


The script doesn't handle file reading errors (e.g., encoding issues, permission errors). If a hooks.json file has encoding issues or cannot be parsed, the script will crash with an unhandled exception rather than reporting it as a validation error. Consider wrapping the file operations and JSON parsing in a try-except block to catch and report these errors gracefully.

Suggested change

with open(path) as fp:

data = json.load(fp)

try:

with open(path, encoding="utf-8") as fp:

data = json.load(fp)

except (OSError, UnicodeError, json.JSONDecodeError) as exc:

errors.append(f"{path}: Failed to load JSON: {exc}")

continue

Copilot

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 8 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-01-23T08:04:03Z

scripts/validate_skills.py

+                if "name:" not in frontmatter:
+                    errors.append(f"{path}: Missing 'name' in frontmatter")
+                if "description:" not in frontmatter:
+                    errors.append(f"{path}: Missing 'description' in frontmatter")


The frontmatter validation uses a simple string search ("name:" in frontmatter) which could produce false positives. For example, it would incorrectly match if "name:" appears in a multi-line description or comment. Consider using a proper YAML parser (like PyYAML) to parse and validate the frontmatter structure more reliably.

Copilot · 2026-01-23T08:04:04Z

scripts/validate_skills.py

+    if errors:
+        print("Errors found:")
+        for e in errors:
+            print(f"  ✗ {e}")
+        sys.exit(1)
+    print("✓ All SKILL.md files have valid frontmatter")


When no SKILL.md files are found in the repository, the script silently exits with success (exit code 0). Consider adding a check to warn if no SKILL.md files were found, as this might indicate a problem with the repository structure or the script not running from the correct directory.

Copilot · 2026-01-23T08:04:04Z

scripts/validate_hooks.py

+    if errors:
+        print("Errors found:")
+        for e in errors:
+            print(f"  ✗ {e}")
+        sys.exit(1)
+    print("✓ All hooks.json files are valid")


When no hooks.json files are found in the repository, the script silently exits with success. Consider adding a check to report if no hooks.json files were found, to help distinguish between "no files found" and "all files valid".

Copilot · 2026-01-23T08:04:04Z

scripts/validate_json.py

+    if errors:
+        print(f"  ✗ {errors} JSON errors found")
+        sys.exit(1)
+    print("✓ All JSON files are valid")


The script doesn't check if any JSON files were found. When run in an empty repository or wrong directory, it will report success even though no validation actually occurred. Consider tracking the count of files processed and reporting it or warning if zero files are found.

Copilot · 2026-01-23T08:04:04Z

scripts/validate_hooks.py

+                hooks = data.get("hooks", {})
+                if isinstance(hooks, list):
+                    errors.append(f"{path}: 'hooks' must be an object, not array")
+                    continue
+


The validation doesn't verify that the "hooks" field is actually required or present in the data. If a hooks.json file doesn't contain a "hooks" key at all, it will pass validation with an empty dictionary from data.get("hooks", {}). Consider adding a check to ensure the "hooks" field exists in hooks.json files.

Suggested change

hooks = data.get("hooks", {})

if isinstance(hooks, list):

errors.append(f"{path}: 'hooks' must be an object, not array")

continue

if "hooks" not in data:

errors.append(f"{path}: Missing required 'hooks' field")

continue

hooks = data["hooks"]

if not isinstance(hooks, dict):

if isinstance(hooks, list):

errors.append(f"{path}: 'hooks' must be an object, not array")

else:

errors.append(f"{path}: 'hooks' must be an object")

continue

TESTING.md

Copilot · 2026-01-23T08:04:05Z

scripts/validate_skills.py

+    errors = []
+
+    for root, dirs, files in os.walk("."):
+        if ".git" in root:


The check for ".git" in root will incorrectly skip any directory path that contains ".git" anywhere in its path (e.g., ".github" directory). This should use a more precise check to avoid skipping legitimate directories like ".github" which contains the workflows.

Suggested change

if ".git" in root:

path_parts = os.path.normpath(root).split(os.sep)

if ".git" in path_parts:

Copilot · 2026-01-23T08:04:05Z

scripts/validate_hooks.py

+        if ".git" in root:
+            continue


The check for ".git" in root will incorrectly skip any directory path that contains ".git" anywhere in its path (e.g., ".github" directory). This should use a more precise check to avoid skipping legitimate directories like ".github" which may contain hooks.json files for workflow configurations.

Suggested change

if ".git" in root:

continue

# Skip version control metadata directories named '.git' but still

# traverse similarly named directories like '.github' that may contain

# hooks.json files for workflow configurations.

if ".git" in dirs:

dirs.remove(".git")

Copilot

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 2 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-01-23T08:19:59Z

scripts/validate_hooks.py

+    errors = []
+
+    for root, dirs, files in os.walk("."):
+        if ".git" in root:


The check if ".git" in root could incorrectly skip directories that contain ".git" as a substring (e.g., "my.github" or ".gitignore-backup"). Consider using a more precise check such as if root.split(os.sep).__contains__(".git") or if "/.git/" in root or root.startswith(".git") to ensure only the actual .git directory is skipped.

Suggested change

if ".git" in root:

if ".git" in root.split(os.sep):

Copilot · 2026-01-23T08:19:59Z

scripts/validate_skills.py

+                    errors.append(f"{path}: Missing YAML frontmatter")
+                    continue
+
+                match = re.match(r"^---\s*\n(.*?)\n---\s*", content, re.DOTALL)


The regex pattern r"^---\s*\n(.*?)\n---\s*" may not correctly handle edge cases where the closing --- delimiter is at the end of the file without a trailing newline, or where there's content immediately after --- on the same line. Consider making the pattern more robust by using r"^---\s*\n(.*?)\n---\s*(?:\n|$)" to explicitly handle both newline and end-of-string after the closing delimiter.

Suggested change

match = re.match(r"^---\s*\n(.*?)\n---\s*", content, re.DOTALL)

match = re.match(r"^---\s*\n(.*?)\n---\s*(?:\n|$)", content, re.DOTALL)

CI validation (GitHub Actions + local scripts): - JSON syntax for all .json files - marketplace.json required fields (name, owner, plugins) - Plugin structure (plugin.json exists, skills have SKILL.md) - SKILL.md frontmatter (name, description required) - hooks.json format (object not array, valid event names) - Cross-platform: uses os.walk instead of find - Robust error handling for file/JSON operations Documentation: - CONTRIBUTING.md: Contribution workflow, plugin structure guide - AGENTS.md: Guidelines for LLMs working on this codebase - TESTING.md: Validation and testing instructions - CLAUDE.md: Quick reference pointing to other docs - README.md: Add CI badge and documentation section Also fixes: - Default RESEARCHER_DIR to $HOME/research - .git check to not skip .github directories Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

jontsai changed the title ~~Add CI validation for plugin structure~~ Add CI validation and project documentation Jan 23, 2026

jontsai force-pushed the add-ci-checks branch from eb6459b to 21e7c3e Compare January 23, 2026 07:00

jontsai requested a review from Copilot January 23, 2026 07:14

Copilot started reviewing on behalf of jontsai January 23, 2026 07:14 View session

Copilot AI reviewed Jan 23, 2026

View reviewed changes

jontsai requested a review from Copilot January 23, 2026 07:58

jontsai force-pushed the add-ci-checks branch from 20579b5 to e527214 Compare January 23, 2026 07:58

Copilot started reviewing on behalf of jontsai January 23, 2026 07:58 View session

jontsai force-pushed the add-ci-checks branch from e527214 to 4b198cd Compare January 23, 2026 08:02

Copilot AI reviewed Jan 23, 2026

View reviewed changes

jontsai requested a review from Copilot January 23, 2026 08:15

Copilot started reviewing on behalf of jontsai January 23, 2026 08:15 View session

Copilot AI reviewed Jan 23, 2026

View reviewed changes

jontsai force-pushed the add-ci-checks branch from 4b198cd to 89ecc5e Compare January 23, 2026 08:40

jontsai merged commit 8494221 into main Jan 23, 2026
1 check passed

jontsai deleted the add-ci-checks branch January 23, 2026 08:55

		with open(".claude-plugin/marketplace.json") as f:
		data = json.load(f)

-    with open(".claude-plugin/marketplace.json") as f:
-        data = json.load(f)
+    try:
+        with open(".claude-plugin/marketplace.json", encoding="utf-8") as f:
+            data = json.load(f)
+    except FileNotFoundError:
+        print("Error: .claude-plugin/marketplace.json not found.")
+        sys.exit(1)
+    except PermissionError:
+        print("Error: Permission denied when reading .claude-plugin/marketplace.json.")
+        sys.exit(1)
+    except (OSError, UnicodeDecodeError) as e:
+        print(f"Error: Failed to read .claude-plugin/marketplace.json: {e}")
+        sys.exit(1)
+    except json.JSONDecodeError as e:
+        print(f"Error: Invalid JSON in .claude-plugin/marketplace.json: {e}")
+        sys.exit(1)

	match = re.match(r"^---\n(.*?)\n---", content, re.DOTALL)
	match = re.match(r"^---\s\n(.?)\n---\s*", content, re.DOTALL)

	if ".git" in root:
	path_parts = os.path.normpath(root).split(os.sep)
	if ".git" in path_parts:

-        if ".git" in root:
-            continue
+        # Skip version control metadata directories named '.git' but still
+        # traverse similarly named directories like '.github' that may contain
+        # hooks.json files for workflow configurations.
+        if ".git" in dirs:
+            dirs.remove(".git")

Conversation

jontsai commented Jan 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

CI Validation

Documentation

Files Added

Test Plan

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Jan 23, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI Jan 23, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI Jan 23, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 23, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 23, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 23, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 23, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 23, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Jan 23, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 23, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 23, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 23, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 23, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI Jan 23, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 23, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Jan 23, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 23, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

jontsai commented Jan 23, 2026 •

edited

Loading