Should the command fixture test inputs be self-testing?

[JayBazuzi]

Two problems we maybe care about:

• How do we know that the command fixture test inputs are valid programs?
• How do we know that the output valid and equivalent to the input?

Proposed solution:

• Every input includes an assertion.
• The test runner compiles and runs the input, to ensure that the input is valid and does what we think.
• The test runner compiles and runs the actual result, to ensure that the results is valid and does the same thing

For example, for rename the input program might look like:

const x = 42;
expect(x).toBe(42);

[devill]

I really like this idea! It would definitely make me feel safer. Whats not quite clear for me is how we can ensure that the input and the expected result are equivalent. Unless we include tests with the input and run the tests for each received directory as part of the test run, but I worry it might make the already pretty heavy build even longer. One option would be to only run the tests inside the fixture projects when we approve a new received as accepted?

[JayBazuzi]

The input has two parts: the code that is being refactored, and 1 or more assertions. If you run it the assertions should pass.

The refactoring should not modify the assertions. If the assertions pass, then you know the code still does the same thing.

The test infra could enforce this: every input should have 1+ assertions at the end, and every expected should end with those same exact assertions.

(In the wild, with arbitrarily-complicated code that we can't understand, tests are not sufficient to guarantee that a refactoring is correct, but in this case we were writing the input code ourselves, so we can count on tests.)

[devill]

So the test code would be part of the main project test code? Right now the fixtures are deliberately excluded from the ts project, so that the IDE doesn't get confused by them. I would want to keep it that way.

Either way, I like the concept and I'm sure there is a good solution for it. It's likely that if you come up with it I'll be like: "oh... that's cool" :D

[JayBazuzi]

Ok, I'll see if I can get this going