Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
64 changes: 64 additions & 0 deletions TODO.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
# Mutant Killer: Remaining Issues

## Security

- **Token visible in process listings**: `--token` CLI flag and `GitProvider.injectAuth()` embed tokens in URLs passed to `ProcessBuilder`, making them visible via `ps`. Consider using `git credential-store` or a credential helper instead.
- **`git push --force` in RepositoryManager**: A bug in branch naming could force-push to protected branches. Add a safeguard to refuse force-push to `main`/`master`.
- **MutantKillerConfig record**: The custom `toString()` now excludes the API key, but the record's auto-generated `hashCode`/`equals` still include it. Not a security risk, but worth noting.

## Missing Verification

- **Generated code is never compiled before committing**: Both `RunCommand` and `KillCommand` apply LLM-generated test code without running `mvn test` or `gradle test` first. This means PRs may contain code that does not compile. Add a verification step between `improvement.apply()` and `commitAndPush()`.
- **No validation that applied tests actually kill the mutant**: After generating and applying test improvements, the tool should re-run PIT (or at least the new test) to confirm the mutant is actually killed.

## Error Handling

- **HTTP response null safety in git providers**: `responseJson.get("html_url")`, `responseJson.get("web_url")`, and `responseJson.get("pullRequestId")` could return null if the API response shape changes. Add null checks with descriptive error messages.
- **MutantAnalyzer.analyze() line number validation**: If `mutation.lineNumber()` is 0, negative, or exceeds the file length, the context extraction produces incorrect results silently.

## Incomplete Features

- **`analyze.md` prompt template unused**: The Handlebars-style template at `src/main/resources/prompts/analyze.md` is never referenced in code. `MutantAnalysis.buildAnalysisPrompt()` constructs the prompt inline. Either wire up a template engine or remove the file.
- **BuildExecutor verbose flag ignored**: `MavenExecutor.runMutationTesting()` always passes `false` to `execute(command, false)`. The verbose flag from CLI options should be threaded through so users can see build output.
- **Source file search is limited**: `MutantAnalyzer.findSourceFile()` only checks the exact package path. No recursive search or multi-module project support.
- **Test file discovery is limited**: Only looks for `ClassNameTest.java`. Does not check `TestClassName.java`, `ClassNameTests.java`, or tests in different packages.

## Configuration

- **No project-level config file**: The tool only accepts configuration via CLI flags and environment variables. A `.mutant-killer.yml` or similar would let teams persist settings in their repository.
- **maxTokens hardcoded to 2048**: In `TestImprover.java`, the Claude API max tokens is fixed. For complex test improvements this may be insufficient. Should be configurable.
- **Context window hardcoded to 5 lines**: `MutantAnalyzer` uses `mutationLine +/- 5` as magic numbers. Should be configurable.
- **Build timeout hardcoded to 30 minutes**: In `BuildExecutor.execute()`. Should be configurable via CLI or config.
- **Git timeout hardcoded to 5 minutes**: In `RepositoryManager`. Should be configurable.

## Resource Management

- **HttpClient instances never closed**: All three git providers (`GitHubProvider`, `GitLabProvider`, `AzureDevOpsProvider`) create `HttpClient` instances that are never closed. Since Java 21, `HttpClient` implements `AutoCloseable`.
- **AnthropicClient never closed**: `TestImprover` creates an `AnthropicOkHttpClient` that wraps OkHttp connection/thread pools but is never closed.
- **Temporary directories never cleaned up**: Cloned repositories in the work directory are never deleted, even on successful completion.

## CLI Usability

- **No progress reporting for long operations**: Mutation testing can run for 30+ minutes with zero output when verbose is off. Add a spinner or periodic status message.
- **No `--format` option on `analyze` command**: Output is only human-readable text. JSON or CSV output would help CI/CD integration.
- **Subcommands lack `mixinStandardHelpOptions`**: Only the top-level command has `--help`/`--version` via picocli mixin.

## Architecture

- **No retry logic for API calls**: Claude API and git provider API calls have no retry on transient failures (429, 500, 503).
- **No rate limiting for Claude API calls**: Processing multiple mutants fires API calls in a tight loop. Should add a delay or respect rate limit headers.
- **Each mutant gets its own branch and PR**: Related mutants in the same class/method should be grouped into a single PR to avoid PR spam.
- **No idempotency checking**: Repeated runs re-analyze and re-call the Claude API for mutants that already have open PRs.
- **No Bitbucket support**: Only GitHub, GitLab, and Azure DevOps are supported.
- **No multi-module Maven/Gradle project support**: Source and test directories are hardcoded to `src/main/java` and `src/test/java`.

## Code Quality

- **TestImproverTest tests a reimplemented copy of extractCodeBlock()**: The test class reimplements the private method rather than using reflection to call the actual one. If the real method changes, tests still pass.
- **RunCommandTest uses heavy reflection**: Private utility methods (`extractRepoName`, `simpleClassName`, `generateMutantId`, `buildPrBody`) are tested via `setAccessible(true)`. These should be extracted to a package-private utility class.
- **Logging framework configured but unused**: SLF4J/Logback are dependencies with `logback.xml` configured, but no source file uses `Logger`. All output goes through `System.out`/`System.err`.

## Build/Packaging

- **picocli-codegen generates native-image config but no GraalVM build**: The annotation processor generates `reflect-config.json` for native images, but there is no `native-maven-plugin` or GraalVM build profile. Either add native-image support or remove the annotation processor.
- **No mocking library for tests**: Without Mockito or similar, HTTP-based classes and the Claude API client cannot be properly unit tested without hitting real APIs.
10 changes: 9 additions & 1 deletion mvnw

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

17 changes: 17 additions & 0 deletions pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,12 @@
<version>5.10.1</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-params</artifactId>
<version>5.10.1</version>
<scope>test</scope>
</dependency>
</dependencies>

<build>
Expand Down Expand Up @@ -132,7 +138,18 @@
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>io.github.dubthree.mutantkiller.MutantKiller</mainClass>
</transformer>
<transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
</transformers>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
<createDependencyReducedPom>false</createDependencyReducedPom>
</configuration>
</execution>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
@Command(
name = "mutant-killer",
mixinStandardHelpOptions = true,
version = "0.1.0",
version = "0.1.0-SNAPSHOT",
description = "Autonomous agent that kills surviving mutants by improving your tests.",
subcommands = {
RunCommand.class,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -79,13 +79,22 @@ private Path findSourceFile(String className) {
}

private Path findTestFile(String className) {
String simpleClassName = className.substring(className.lastIndexOf('.') + 1);
int lastDot = className.lastIndexOf('.');
String simpleClassName = lastDot >= 0 ? className.substring(lastDot + 1) : className;
String testClassName = simpleClassName + "Test";
String packagePath = className.substring(0, className.lastIndexOf('.')).replace('.', '/');

Path candidate = config.testDir().resolve(packagePath).resolve(testClassName + ".java");
if (Files.exists(candidate)) {
return candidate;

if (lastDot >= 0) {
String packagePath = className.substring(0, lastDot).replace('.', '/');
Path candidate = config.testDir().resolve(packagePath).resolve(testClassName + ".java");
if (Files.exists(candidate)) {
return candidate;
}
} else {
// Default package
Path candidate = config.testDir().resolve(testClassName + ".java");
if (Files.exists(candidate)) {
return candidate;
}
}
return null;
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,25 @@ public static BuildExecutor detect(Path projectDir) {
*/
public abstract Path testDir();

/**
* Recursively search for mutations.xml in a directory.
*/
protected File findMutationsXml(File dir) {
File[] files = dir.listFiles();
if (files == null) return null;

for (File file : files) {
if (file.getName().equals("mutations.xml")) {
return file;
}
if (file.isDirectory()) {
File found = findMutationsXml(file);
if (found != null) return found;
}
}
return null;
}

/**
* Execute a command in the project directory.
*/
Expand Down Expand Up @@ -143,22 +162,6 @@ public Path sourceDir() {
public Path testDir() {
return projectDir.resolve("src/test/java");
}

private File findMutationsXml(File dir) {
File[] files = dir.listFiles();
if (files == null) return null;

for (File file : files) {
if (file.getName().equals("mutations.xml")) {
return file;
}
if (file.isDirectory()) {
File found = findMutationsXml(file);
if (found != null) return found;
}
}
return null;
}
}

/**
Expand Down Expand Up @@ -217,21 +220,5 @@ public Path sourceDir() {
public Path testDir() {
return projectDir.resolve("src/test/java");
}

private File findMutationsXml(File dir) {
File[] files = dir.listFiles();
if (files == null) return null;

for (File file : files) {
if (file.getName().equals("mutations.xml")) {
return file;
}
if (file.isDirectory()) {
File found = findMutationsXml(file);
if (found != null) return found;
}
}
return null;
}
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
import io.github.dubthree.mutantkiller.analysis.MutantAnalyzer;
import io.github.dubthree.mutantkiller.codegen.TestImprover;
import io.github.dubthree.mutantkiller.config.MutantKillerConfig;
import static io.github.dubthree.mutantkiller.config.MutantKillerConfig.DEFAULT_MODEL;
import io.github.dubthree.mutantkiller.pit.MutationResult;
import io.github.dubthree.mutantkiller.pit.PitReportParser;
import picocli.CommandLine.Command;
Expand Down Expand Up @@ -31,7 +32,7 @@ public class KillCommand implements Callable<Integer> {
@Option(names = {"-t", "--test"}, description = "Test source directory", required = true)
private File testDir;

@Option(names = {"--model"}, description = "LLM model to use", defaultValue = "claude-sonnet-4-20250514")
@Option(names = {"--model"}, description = "LLM model to use", defaultValue = DEFAULT_MODEL)
private String model;

@Option(names = {"--dry-run"}, description = "Show proposed changes without applying")
Expand Down
28 changes: 15 additions & 13 deletions src/main/java/io/github/dubthree/mutantkiller/cli/RunCommand.java
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
import io.github.dubthree.mutantkiller.codegen.TestImprovement;
import io.github.dubthree.mutantkiller.codegen.TestImprover;
import io.github.dubthree.mutantkiller.config.MutantKillerConfig;
import static io.github.dubthree.mutantkiller.config.MutantKillerConfig.DEFAULT_MODEL;
import io.github.dubthree.mutantkiller.git.GitProvider;
import io.github.dubthree.mutantkiller.git.RepositoryManager;
import io.github.dubthree.mutantkiller.build.BuildExecutor;
Expand Down Expand Up @@ -35,7 +36,7 @@ public class RunCommand implements Callable<Integer> {
@Option(names = {"-b", "--base-branch"}, description = "Base branch to work from", defaultValue = "main")
private String baseBranch;

@Option(names = {"--model"}, description = "LLM model to use", defaultValue = "claude-sonnet-4-20250514")
@Option(names = {"--model"}, description = "LLM model to use", defaultValue = DEFAULT_MODEL)
private String model;

@Option(names = {"--max-mutants"}, description = "Maximum mutants to process", defaultValue = "10")
Expand All @@ -53,15 +54,21 @@ public class RunCommand implements Callable<Integer> {
@Option(names = {"-v", "--verbose"}, description = "Verbose output")
private boolean verbose;

@Option(names = {"--github-token"}, description = "GitHub token (or set GITHUB_TOKEN env var)")
private String githubToken;
@Option(names = {"--token", "--github-token"}, description = "Git provider token (or set GIT_TOKEN / GITHUB_TOKEN env var)")
private String gitToken;

@Override
public Integer call() throws Exception {
// Resolve GitHub token
String token = githubToken != null ? githubToken : System.getenv("GITHUB_TOKEN");
// Resolve git provider token: CLI flag, then GIT_TOKEN, then GITHUB_TOKEN
String token = gitToken;
if (token == null || token.isBlank()) {
System.err.println("GitHub token required. Set --github-token or GITHUB_TOKEN env var.");
token = System.getenv("GIT_TOKEN");
}
if (token == null || token.isBlank()) {
token = System.getenv("GITHUB_TOKEN");
}
if (token == null || token.isBlank()) {
System.err.println("Git provider token required. Set --token, GIT_TOKEN, or GITHUB_TOKEN env var.");
return 1;
}

Expand Down Expand Up @@ -129,7 +136,7 @@ public Integer call() throws Exception {
System.out.println(" Processing: " + survived.size() + " mutants");

if (survived.isEmpty()) {
System.out.println("\nNo surviving mutants! Your tests are strong. 💪");
System.out.println("\nNo surviving mutants! Your tests are strong.");
return 0;
}

Expand Down Expand Up @@ -257,12 +264,7 @@ private String simpleClassName(String fullClassName) {
}

private String extractRepoName(String url) {
String name = url.replaceAll("\\.git$", "");
int lastSlash = name.lastIndexOf('/');
if (lastSlash >= 0) {
name = name.substring(lastSlash + 1);
}
return name;
return RepositoryManager.extractRepoName(url);
}

private String buildPrBody(MutationResult mutant, MutantAnalysis analysis, TestImprovement improvement) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -113,28 +113,34 @@ private String appendToTestClass(String existingCode, String newMethods) {

private void createNewTestFile() throws IOException {
String className = analysis.mutation().mutatedClass();
String simpleClassName = className.substring(className.lastIndexOf('.') + 1);
String packageName = className.substring(0, className.lastIndexOf('.'));

int lastDot = className.lastIndexOf('.');
String simpleClassName = lastDot >= 0 ? className.substring(lastDot + 1) : className;
String packageName = lastDot >= 0 ? className.substring(0, lastDot) : null;

StringBuilder testClass = new StringBuilder();
testClass.append("package ").append(packageName).append(";\n\n");
if (packageName != null) {
testClass.append("package ").append(packageName).append(";\n\n");
}
testClass.append("import org.junit.jupiter.api.Test;\n");
testClass.append("import static org.junit.jupiter.api.Assertions.*;\n\n");
testClass.append("/**\n * Tests generated by mutant-killer to improve mutation coverage.\n */\n");
testClass.append("class ").append(simpleClassName).append("Test {\n\n");
testClass.append(indentCode(generatedCode, 4));
testClass.append("\n}\n");

// Determine path
String packagePath = packageName.replace('.', '/');
Path testDir = analysis.mutation().sourceFile() != null
? Path.of(analysis.mutation().sourceFile()).getParent()
: Path.of("src/test/java").resolve(packagePath);

// Determine path using the test directory from the analysis config
Path testDir;
if (packageName != null) {
String packagePath = packageName.replace('.', '/');
testDir = Path.of("src/test/java").resolve(packagePath);
} else {
testDir = Path.of("src/test/java");
}

Files.createDirectories(testDir);
Path testFile = testDir.resolve(simpleClassName + "Test.java");
Files.writeString(testFile, testClass.toString());

System.out.println("Created new test file: " + testFile);
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,10 @@

import com.anthropic.client.AnthropicClient;
import com.anthropic.client.okhttp.AnthropicOkHttpClient;
import com.anthropic.models.messages.*;
import com.anthropic.models.messages.ContentBlock;
import com.anthropic.models.messages.MessageCreateParams;
import com.anthropic.models.messages.MessageParam;
import com.anthropic.models.messages.Message;
import io.github.dubthree.mutantkiller.analysis.MutantAnalysis;
import io.github.dubthree.mutantkiller.config.MutantKillerConfig;
import io.github.dubthree.mutantkiller.pit.MutationResult;
Expand Down Expand Up @@ -78,8 +81,8 @@ public Optional<TestImprovement> improve(MutationResult mutation, MutantAnalysis
Message response = client.messages().create(params);

String content = response.content().stream()
.filter(block -> block instanceof TextBlock)
.map(block -> ((TextBlock) block).text())
.filter(ContentBlock::isText)
.map(block -> block.asText().text())
.findFirst()
.orElse("");

Expand Down
Loading