Requiring tests to be in a folder named 'test' is a terrible quality metric

[myndzi]

To an extent I can agree that containing tests in a subfolder is useful as a quality metri, but requiring a name for that subfolder is plain awful. It doesn't speak at all to any kind of quality metric about the code itself, only shoehorns authors who are paying attention into a rigid structure, and authors who aren't paying attention into falsely low scores.

Two alternative suggestions:

1. The intent of this checker seems to be about code organization / keeping things tidy. This can be achieved without any specific structure by simply testing the number of "loose files" in the root directory. This should be done on the product of npm packaging so that npmignore is accounted for, and possibly excluding common files such as grunt, gulp, bower, package.json, etc. This method has the advantage of not penalizing small, self-contained packages that only have one file and one test file, for example.

2. Curate a list of acceptable folder names and be open to adding others to the list, as long as they serve the intended purpose. Some opinion is okay with this method (for example "no, 't' is not a good folder name), but it shouldn't be overbearing.

[konobi]

@othiym23 @isaacs got any input?

[tkissing]

While I agree that the "one true folder name" is too strict, I do think conventions are valuable. If I check out your project from GitHub, I don't want to have to read half an hour just to figure out that you put your tests in files ending in .howitshouldbe in the src/lib/extras/verification folder.
As I said in #9 any commonly used folder (test, tests, spec) should be acceptable (but it should be exactly one, no more, no less)

[konobi]

Well... the naming of those folders tend to be convention from other langauges, frameworks, etc. but the name itself doesn't provide any value. So, is there something wrong with saying "instead of 'flibblegoobletestsforme' lets use 'test' since we all understand that things in there will be tests"? I've seen several projects where "spec" isn't tests at all, but contains data that is used to generate APIs and the like. It's also nice to have just one place to be able to run a bunch of checks against (vs maybe a bunch or a wrong directory).

[konobi]

@myndzi is your concern about it being detrimental to current available packages (and therefore their score) or is it more about every author having to choose the same folder? These are two different issues.

[myndzi]

@tkissing: Exactly one is exactly my complaint. I don't have a problem with requiring it in the package root, and I don't have a problem with requiring it to be named sensibly. I don't think that "making it easy on the kwalitee module should have any say in the matter.

@konobi: My concern is both, really.

I don't think stylistic choices have a place in a module like this (what to name a folder). The kind of opinionated choices made in linters are okay in linters because linters affect the user/developer, not the consumers, of the package. This module will affect the user and consumers if it does what you want it to.

And yes, I also have a concern that it will affect existing packages should the "rating" become available in some package-search interface: it does the package authors a disservice by unfairly lowering their rating based on a metric that has no bearing on the quality of their product, and it does package searchers a disservice by same.

Ask yourself the obvious question: if I'm looking at a package that put their tests in a subfolder named 'tests', is it in any way a worse package than one that put their tests in a subfolder named 'test'? I posit that the answer is no, and that no matter which single folder name you choose, there will always be cases where the answer is no -- and that's what makes this a poor choice of metric.

[myndzi]

If you want to get around some of the naming/content problems, you could easily just require an extra field in package.json to indicate the location of the tests directory, by the way.

https://docs.npmjs.com/files/package.json#directories

For whatever reason, this seems to have everything but the test directory in it. It would be easy to extend and you could probably get it included in the official docs / spec too. Then your requirement becomes "must specify the directories.test key" and you can mark them down if it's not something within a specific group if you want to be extra pedantic about it.

[konobi]

@myndzi "it does the package authors a disservice by unfairly lowering their rating based on a metric that has no bearing on the quality of their product"... That's their issue... the point of a "kwalitee" metric, is to ensure that the registry as a whole is more easily consumable and verifiable by users.

The point of some of checks is not that it fits in with everything that anyone what's to do ever... It's to ensure that the package can be easily used and identified by users.

For example, have a look at the addon checker... that's probably something that some authors will take on board, others may not... but it's based on confirmation from the core team.

[myndzi]

A package is not more difficult to be used or identified by users by virtue of whether its test directory is plural or singular. I can keep hammering on that point if you want to keep dancing around it, but at the very least as a lowest common denominator it makes a strong point against a single acceptable directory name.

[konobi]

Now or in the future? If it was declared as the go-to name for the future, would it still be a bad check?

[myndzi]

We cross-posted. Re: "That's their issue" -- no, it really isn't. You're aiming to impose an external, somewhat objective, quality metric on everyone who writes modules. The least you can do is not make the metrics arbitrary and worthless, forcing everyone who doesn't agree with your viewpoint to jump through hoops or be "de-rated". In terms of value from a quality perspective, this metric is about as good as "turned three times widdershins before hitting publish" and about as annoying to any author it affects.

Re: now or future, I mean now. Adding a key to package.json is less onerous than restructuring your project, which can easily involve multiple edits (even if it's only two). As well, it provides the flexibility to allow authors to use their preferred structure (within constraints, if you must have them), while informing kwalitee of the information it may want to perform more detailed checks.

[konobi]

An authors preferred layout does not nessecarily have to map to what is published on NPM.

Fairly sure there's plenty of folks who have gulp style things for doing stuff before publishing.

[myndzi]

If you're suggesting that authors should write extra code/configuration in order to work with their tests in one directory and then change that directory specifically to publish to npm, that's just ... I don't even know what to say.

Most peoples' interactions with test directories won't be through npm to begin with. One could go so far as to say that test files themselves shouldn't necessarily even be published to NPM. And that suggestion multiplies the amount of work you'd be putting on authors.

Can you try answering the simple question I posed before? What makes 'test' better than 'tests' to such a degree that it belongs as a scored validation metric in this module? If you can't at least answer that satisfactorily, the rest of the discussion is meaningless.

[konobi]

"test" is more common place on the registry than any other name for a testing folder.

[myndzi]

That doesn't make it better, just more common.

[konobi]

How does anything other than "test" make it better?