velvet

Smooth, functional testing in the Elvish shell

velvet is a minimalist - yet sophisticated - test framework and runner, enabling its users to run tests organized in hierarchical structures leveraging the functional programming elegance of the Elvish shell.

Why Velvet?

I am personally fond of the expressive, Gherkin-like syntax supported by testing infrastructures like Jest, Vitest and ScalaTest, but each of them is based on a specific language - which usually doesn't feel as natural as a shell when dealing with system aspects like files, networks or inter-process communication.

Given my passion for the Elvish shell, I've designed this testing architecture by combining my favorite aspects of such frameworks, while applying my own perspective - especially focused on cross-technology integration scenarios.

Installation

The library can be installed via epm - in particular:

use epm

epm:install github.com/giancosta86/velvet

Even better, if you have epm-plus, you can run:

epm:install github.com/giancosta86/velvet@v3

or any other specific Git reference.

Setup

In rc.elv, it is recommended to add the following lines:

use github.com/giancosta86/velvet/velvet

var velvet~ = $velvet:velvet~

This will make the velvet command globally available at the command prompt.

Writing tests

Tests are defined in test scripts - by convention, files having .test.elv extension: they are standard Elvish scripts with a fundamental plus - they can also transparently invoke a handful of additional builtin functions, injected by Velvet and described in this section.

Structuring the tests

The >> function is the basic building block for defining the test tree - adopting a Gherkin-like descriptive notation:

>> 'First component' {
  >> 'division operation' {
    >> 'when divisor is not 0' {
      >> 'should return value' {
        # Test code goes here
      }
    }

    >> 'when divisor is 0' {
      >> 'should crash' {
        # Test code goes here
      }
    }
  }
}

>> 'Second component' {
  >> 'other operation' {
    >> 'should work' {
      # Test code goes here
    }
  }
}

The innermost calls of >> define tests, which are the leaves in a tree of sections; in the example above, the tests are:

• should return value

• should crash

• should work

whereas all the other occurrences of >> define sections - and each section can include tests and sub-sections.

Please, note: tests can also reside in the root of the script - of course, when they are not contained in another >> block.

Please, note: >> can also be followed just by the test title, like in:

>> 'This is a test draft'

This will define a test draft, whose body simply calls fail-test - thus equivalent to:

>> 'This is a test draft' {
  fail-test
}

Please, note: the >> function - called at the beginning of the line, has no ambiguity with the >> redirection operator, which always appears after a command.

Test outcome

A test can only have one of two outcomes:

• ✅passed if its block ends with no exceptions

• ❌failed if an exception was thrown; in particular, you can fail a test via Elvish's fail function, or via one of Velvet's assertions.

In the default console reporter, the test output - on both stdout and stderr - is displayed, together with the exception log, only if the test fails.

Assertions

• should-be [&strict] <expected>: if the value passed via pipe (|) is not equal to the <expected> argument:

  1. Displays both values

  2. If the diff command is available on the system, also shows their differences

  3. Fails.

  To test a single variable, you can prepend the put function:

  put $my-variable |
    should-be 90

  although, most frequently, you'll use a direct pipe:

  my-command <arguments> |
    should-be <expected value>

  Equality

  In Velvet, equality is defined as follows:

  • the default behavior consists in comparing via eq the minimized representations of both operands.

    # EQUAL!
    put 90 |
      should-be (num 90)

    DEFINITION: the minimized representation for any value is defined as follows:

    • if the value is a number, it is the string denoting the number

    • if the value is a list, it is a list whose items are the minimized representations of its items

    • if the value is a map, it is a map whose keys and values are the minimized representations of the keys and values

    • otherwise, it is the value itself

  • if &strict is requested, the eq function is applied to the pair of values.

    # NOT EQUAL!
    put 90 |
      should-be &strict (num 90)

• should-not-be [&strict] <unexpected>: if the value passed via pipe (|) is equal to the <unexpected> argument:

  1. Displays such value

  2. Fails.

  some-command <arguments> |
    should-not-be 90

  The input mechanism and the equality logic are the same as those described for should-be.

• should-emit: ensures that the values passed via pipe (|) are exactly the values in the expected list; emission order matters, unless the order-key option is set; on the other hand, the strict option works according to the equality rules described within the context of should-be.

  The overall command is equivalent to:

  put [(
    all |
      order &key=$order-key | #Only if &order-key is set
  )] |
    should-be &strict=$strict $expected

  Example

  {
    put Hello
    put 90
  } |
    should-emit [
      Hello
      90
    ]
  
  {
    echo Hello
    echo World
  } |
    should-emit [
      Hello
      World
    ]

  Please, note: for more granular tests focused on the string output of a command, please refer to create-output-tester.

• should-not-emit: ensures that the values passed via pipe (|) do not include any of the values in the unexpected-values list; the strict option works according to the equality rules described within the context of should-be.

  Example

  {
    put Hello
    put 90
  } |
    should-not-emit [
      World
      4
      SomeOtherValue
    ]

  Please, note: for more granular tests focused on the string output of a command, please refer to create-output-tester.

• fails <block>: requires block to throw a fail exception - via fail - outputting the content of such failure and failing if no fail was actually thrown; if another type of exception is thrown by block - for example, a syntax error - it simply passes through. This assertion is preferable to throws, which is more general-purpose.

  Example

  fails {
    fail Dodo
  } |
    should-be Dodo

  Please, note: for more granular tests focused on the output of a command, please refer to create-output-tester.

• should-contain: receives a container via pipe (|) and a value as argument, then:

  • if the container is a string, ensures that value is a substring

  • if the container is a list, ensures that value is contained in the list

  • if the container is a map, ensures that value is a key of the map

  • if the container is a set from Ethereal, ensures that value belongs to the set

  In all cases, the strict flag is supported - enabling strict equality, as described in the related section above.

  Example

  # String
  put 'Greetings, magic world!' |
    should-contain magic
  
  # List
  put [alpha beta gamma] |
    should-contain beta
  
  # Map
  put [
    &a=90
    &b=92
    &c=95
  ] |
    should-contain b
  
  # Set
  use github.com/giancosta86/ethereal/v1/set
  
  all [alpha beta gamma] |
    set:of |
    should-contain beta

• should-not-contain: the negation of should-contain - please, see its documentation for aspects such as the supported container types.

  Example

  # String
  put 'Hello, everybody!' |
    should-not-contain world
  
  # List
  put [alpha beta gamma] |
    should-not-contain ro
  
  # Map
  put [
    &a=90
    &b=92
    &c=95
  ] |
    should-not-contain omega
  
  # Set
  use github.com/giancosta86/ethereal/v1/set
  
  set:of alpha beta gamma |
    should-not-contain ro

• throws <block>: most general way to assert that block throws an exception of any kind - failing if the block completed successfully. The function emits:

  • by default:

    • the exception itself, as a value

    • the bytes emitted by the block, redirected to stderr

  • if the swallow flag is enabled: the actual output of the block - both bytes and values

  Please, note: In general, you should use the fails assertion, as it focuses on fail-based exceptions.

  Example

  # This works fine
  throws {
    fail DODO
  }
  
  # This will fail, saying a failure was expected!
  throws {
    # This block throws nothing
  }

  As a plus, the exception itself is output as a value, so it can be further inspected via the exception module provided by Ethereal.

  Example

  throws {
    fail DODO
  } |
    exception:is-return |
    should-be $false

• fail-test takes no arguments and always fails - with a predefined message: it's perfect for quickly sketching out a new test in test iterations.

Using assertions in shared .elv modules

Assertions are already injected by Velvet into the default namespace when running .test.elv test script, but what if one needs to call them from within a .elv utility module shared by multiple test scripts?

In such a scenario, one can access all the assertions provided by Velvet simply by importing the assertions module from Velvet's package:

use github.com/giancosta86/velvet/<VERSION>/assertions

then, instead of calling - for example - should-be, one can pass through the module namespace: assertions:should-be.

Tools

Tools are utility constructs that can be used in advanced test scripts; just like assertions, they are all available in the default namespace of any test scripts - whereas other Elvish scripts can access them via the following module import:

use github.com/giancosta86/velvet/<VERSION>/tools

Output tester

Enables bulk tests - more precisely, multiple should-contain and should-not-contain assertions on the same command output in the form of a string containing both bytes and values, all converted via to-lines.

In particular, the create-output-tester function creates an object with the following methods:

• should-contain-all: takes a list of strings and ensures, via should-contain, that the buffered command output contains each and every given string.

• should-contain-none: takes a list of strings and ensures, via should-not-contain, that the buffered command output does not contain each and every given string.

• should-contain-snippet: takes a list of strings, joins them into a single string via the newline character, and ensures that such string is contained into the command output text.

• should-not-contain-snippet: takes a list of strings, joins them into a single string via the newline character, and ensures that such string is not contained into the command output text.

As a plus, the text field contains the buffered output as a string.

Please, note: create-output-tester takes an optional unstyled flag, which removes any style-related control sequence from the command output text - via the string:unstyled function provided by Ethereal.

Example

1. Create the output tester, by piping a command output into create-output-tester:

   var tester = (
    {
      put Alpha
      echo Beta
      put Gamma
      echo Delta
    } |
      output-tester:create

2. Invoke the assertions

   $tester[should-contain-all] [
     Alpha
     Beta
     Gamma
     Delta
   ]
   
   $tester[should-contain-none] [
     INEXISTENT
     MISSING
     'SOMETHING ELSE'
   ]

Example test script

The following script could be saved as a .test.elv file - ready to be run via the velvet command.

use str

>> 'In arithmetic' {
  >> 'addition' {
    >> 'should work' {
      + 89 1 |
        should-be 90
    }
  }

  >> 'multiplication' {
    >> 'should return just the expected value' {
      var result = (* 15 6)

      put $result |
        should-be 90

      put $result |
        should-not-be 12
    }
  }

  >> 'division' {
    >> 'when dividing by 0' {
      >> 'should fail' {
        throws {
          / 92 0
        } |
          to-string (all)[reason] |
          str:contains (all) divisor |
          should-be $true
      }
    }
  }

  >> 'custom fail' {
    >> 'should be handled and inspectable' {
      fails {
        if (== (% 8 2) 0) {
          fail '8 is even!'
        }
      } |
        should-be '8 is even!'
    }
  }
}

Running tests

To run all the tests within a directory containing one or more test scripts in its file system tree, just run this command in the Elvish shell:

velvet

The command can be customized via a few optional parameters:

• &must-pass: if at least one test fails, or if at least one test script fails, the command throws an exception. Default: disabled.

• &reporters: a list of functions to report the test summary; each reporter receives a summary map object - processing it as needed.

  Default: the reporter writing just failed tests to the console with colors and emojis.

• &put: outputs the summary to the value channel. In this case, you'll probably want to set &reporters=[] or to a list containing reporters not writing to the console - or simply pipe to only-values.

  Default: disabled.

• num-workers: the (actually maximum) number of parallel Elvish shells executing the test scripts. Default: 8.

The requested script paths can also be passed as variadic arguments to the velvet command:

velvet <script 1> <script 2> ... <script N>

otherwise, all the test scripts located in the directory tree below the current working directory will be run.

Execution flow

• Every test script runs its tests sequentially - in a (virtually) dedicated shell: consequently, the current working directory and other global variables can be changed with no fear of interference, as long as the script sets their initial values as required.

• Multiple test scripts are usually run in parallel - and all the results are merged to a final summary, ready to be sent to the requested reporters and/or emitted to Elvish's value channel.

  Please, note:

  • separate scripts can have sections with the same titles - and the test results will be merged as expected

  • on the other hand, each test must have a unique path in the test tree - that is, the sequence of its section titles combined with its own test title; otherwise, all the duplicate occurrences will be merged into a single DUPLICATE! failing test result

Predefined reporters

Console

There are 2 console reporters, writing the overall summary to the console and sharing these presentation rules:

• In each section, test results come before sub-sections

• Test results are shown in alphabetical order

• Sections are listed in alphabetical order, too

• The stats are displayed at the end

As for the differences between such reporters:

• the terse reporter - at reporting/console/terse:report~- only displays failed tests. It is the default one when running the velvet command

• the full reporter - reporting/console/full:report~- lists all tests - each one with its outcome

JSON

Writes the passed summary object to a given JSON file.

The reporter must be created via a factory function - reporting/json/report, which takes in input a file path and emits the actual reporter function.

For example:

var json-reporter = (json:report $json-report-path)

velvet:velvet &reporters=[$json-reporter]

Reporter spy

A reporter spy is an object providing both a reporter function and a getter emitting the latest summary passed to the former function - which can be especially useful when testing custom reporters.

It is provided by the reporting/spy module, exporting the create factory function:

var spy = (spy:create)

which instantiates an object having the following methods:

• reporter: the actual function to be passed to the velvet command, via its reporters list option

• get-summary: returns the latest value passed to the associated reporter

velvet:velvet &reporters=[$spy]

var summary = ($spy[get-summary])

Ethereal namespaces

The Ethereal library provides a rich set of general-purpose modules for the Elvish shell; Velvet automatically imports some of such modules, to simplify the development of test scripts. For example:

# fs is a module from Ethereal
# that is automatically provided by Velvet.
fs:with-temp-file { |temp-path|
  echo alpha >> $temp-path
  echo beta >> $temp-path

  slurp < $temp-path |
    should-be "alpha\nbeta\n"
}

The available namespaces at present are:

• command
• edit
• exception
• fs
• lang
• map
• resources
• seq
• set
• string

Please, refer to Ethereal's documentation for further details.

Frequently asked questions

Are there setup and teardown?

Nope, because velvet is designed to be minimalist - and you can achieve the same results via the Elvish language:

• setup can be replaced by a factory function - often with closures

• teardown can be replaced by a dedicated higher-order function with a block argument - executing it within a try/finally structure or in a defer-based context

echo, print, pprint and other functions have no output: why?

In the default console reporter, successful tests do not output their stdout/stderr, by design.

Should you need further inspection, you can make the test fail just by adding a call to fail-test - which takes no arguments.

Is there a recommended style to structure the tests?

The user is absolutely free: the system only provides the >> function to create nodes in the test tree: all the rest depends on the specific expressive requirements - which could change from one test script to another or even within the same script.

Why are the tests not in the same order as in my script?

Because they are alphabetically sorted by the default console reporter - a choice stemming from the fact that, in Velvet, sections can span multiple test scripts, as required by the testing style or by the isolation needs in terms of global variables like the current working directory.

Credits

Logo image generated by ChatGPT and manually enhanced with Google Fonts and GIMP.

See also

• Ethereal - Elegant utilities for the Elvish shell

• epm-plus - Package versioning for epm in Elvish

• Elvish