Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion MANIFEST.in
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ include setup.py
include AUTHORS
include LICENSE
include MANIFEST.in
include README.md
include README.rst
include CHANGELOG.md
recursive-include distributed_nose *.py
prune *.pyc
151 changes: 96 additions & 55 deletions README.md → README.rst
Original file line number Diff line number Diff line change
@@ -1,49 +1,65 @@
# distributed-nose Distribute your tests across multiple nodes with no hassle
distributed-nose Distribute your tests across multiple nodes with no hassle
===========================================================================

## TLDR
TLDR
----

### Before
Before
~~~~~~

Machine 1:

$ nosetests long_test_suite
...
Ran 1312 tests in 401.109s
::

$ nosetests long_test_suite
...
Ran 1312 tests in 401.109s

Machine 2:

$ echo "I'm bored :("
I'm bored :(
$ uptime | awk -F'load average:' '{ print "Load: " $2 }'
Load: 0.00, 0.00, 0.00
::

$ echo "I'm bored :("
I'm bored :(
$ uptime | awk -F'load average:' '{ print "Load: " $2 }'
Load: 0.00, 0.00, 0.00

Developer:

$ google-chrome http://news.ycombinator.com
::

$ google-chrome http://news.ycombinator.com

### After
After
~~~~~

Machine 1:

$ export NOSE_NODES=2;
$ export NOSE_NODE_NUMBER=1;
$ nosetests long_test_suite
...
Ran 660 tests in 220.502s
::

$ export NOSE_NODES=2;
$ export NOSE_NODE_NUMBER=1;
$ nosetests long_test_suite
...
Ran 660 tests in 220.502s

Machine 2:

$ echo "I feel loved"
I feel loved
$ export NOSE_NODES=2;
$ export NOSE_NODE_NUMBER=2;
$ nosetests long_test_suite
...
Ran 652 tests in 214.007s
::

$ echo "I feel loved"
I feel loved
$ export NOSE_NODES=2;
$ export NOSE_NODE_NUMBER=2;
$ nosetests long_test_suite
...
Ran 652 tests in 214.007s

## The Saga of Your Test Suite
The Saga of Your Test Suite
---------------------------

### Oppression of Untested Code
Oppression of Untested Code
~~~~~~~~~~~~~~~~~~~~~~~~~~~

The story of Your Test Suite has its heroes and its villains,
its triumphs and its missteps.
Expand Down Expand Up @@ -80,9 +96,11 @@ against the oppression of untested code.
She says,
understanding that working code is the best possible argument.

### A Nose-fueled Peace
A Nose-fueled Peace
~~~~~~~~~~~~~~~~~~~

Methodically wielding and teaching [Nose](https://github.com/nose-devs/nose),
Methodically wielding and teaching
`Nose <https://github.com/nose-devs/nose>`__,
she recruits from within to form a merry band of like-minded developers.
They are the fellowship of TDD
and they are determined to succeed.
Expand All @@ -100,7 +118,8 @@ Making little attempt to justify the practice,
word of time spent writing tests gets to management.
Managerial Oversight moves to squash our fledgling fellowship.

### Managerial Push-back
Managerial Push-back
~~~~~~~~~~~~~~~~~~~~

"If you have time to write tests,
I must have not given you enough to do!"
Expand All @@ -121,7 +140,8 @@ Soon, test failures go unnoticed.
Developers aren't waiting for tests to finish!
The Test Suite falls in to decay.

### Multiprocess Arrives
Multiprocess Arrives
~~~~~~~~~~~~~~~~~~~~

Developers grumble and management freely lobs I-Told-You-Sos.
The darkness of the age before tests threatens to return.
Expand All @@ -134,7 +154,7 @@ vowing to strike at the root cause of decay.

Pouring over ancient texts,
she quickly discovers
[Multiprocess](http://nose.readthedocs.org/en/latest/plugins/multiprocess.html).
`Multiprocess <http://nose.readthedocs.org/en/latest/plugins/multiprocess.html>`__.

"This will release us from our single-core shackles."

Expand All @@ -157,7 +177,8 @@ Fast forward several score tests,
and the once-speedy test suite
regularly brings the beefy, multi-core CI server to its knees.

### Distributed Despair
Distributed Despair
~~~~~~~~~~~~~~~~~~~

Remembering the ease of implementing Multiprocess,
our hero knows the drill.
Expand All @@ -182,7 +203,8 @@ with a new focus on speed!

Factions form and the party threatens to splinter entirely!

### Distributed-Nose: The Scalable Solution
Distributed-Nose: The Scalable Solution
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

"Wait a minute."

Expand All @@ -196,7 +218,8 @@ and achieve it with ease!"
Tickled by the flickering of an idea,
she sets to work.
She is steadfast, pouring over tomes on
[Consistent Hashing](http://en.wikipedia.org/wiki/Consistent_hashing).
`Consistent
Hashing <http://en.wikipedia.org/wiki/Consistent_hashing>`__.
Once the solution is clear in her mind,
she happens upon a fellow adventurer with sound advice.

Expand Down Expand Up @@ -230,56 +253,74 @@ our heroes go on to face many other adventures and trials.
But never again would something come so close to erasing their very core
as the oppression of untested code.

## Why distributed-nose?
Why distributed-nose?
---------------------

Scale your tests horizontally across unlimited machines with two test flags.
Scale your tests horizontally across unlimited machines with two test
flags.

## Installation
Installation
------------

1. Get the project source and install it
#. Get the project source and install it

$ pip install distributed-nose
$ pip install distributed-nose

## Usage
Usage
-----

To run half your tests on one machine and the other half on the other:

Machine 1:

$ nosetests --nodes 2 --node-number 1 long_test_suite
::

$ nosetests --nodes 2 --node-number 1 long_test_suite

Machine 2:

$ nosetests --nodes 2 --node-number 2 long_test_suite
::

$ nosetests --nodes 2 --node-number 2 long_test_suite

Alternatively, you can use the environment variables:
* `NOSE_NODES`
* `NOSE_NODE_NUMBER`

### Temporarily disabling test distribution
- ``NOSE_NODES``
- ``NOSE_NODE_NUMBER``

Temporarily disabling test distribution
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

In the case that you're using environment variables
to control test distribution,
you sometimes still might want to run a one-off test.
Instead of fiddling with environment variables,
you can just use the `--distributed-disabled` flag.
you can just use the ``--distributed-disabled`` flag.

$ export NOSE_NODES=2;
$ export NOSE_NODE_NUMBER=1;
$ nosetests --distributed-disabled long_test_suite
::

## Distribution algorithm
$ export NOSE_NODES=2;
$ export NOSE_NODE_NUMBER=1;
$ nosetests --distributed-disabled long_test_suite

Distribution algorithm
----------------------

To determine which node runs which test,
distributed-nose relies on the [hash_ring](https://github.com/Doist/hash_ring)
distributed-nose relies on the
`hash\_ring <https://github.com/Doist/hash_ring>`__
library's consistent hashing implementation.

## Running the test suite
Running the test suite
----------------------

The test suite requires nose, and can be run via ``setup.py``:

The test suite requires nose, and can be run via `setup.py`:
::

# python setup.py nosetests
# python setup.py nosetests

## Is it Awesome?
Is it Awesome?
--------------

Yes. Increasingly so.
2 changes: 1 addition & 1 deletion setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@

from setuptools import setup

long_description = codecs.open("README.md", "r", "utf-8").read()
long_description = codecs.open("README.rst", "r", "utf-8").read()

CLASSIFIERS = [
'Development Status :: 4 - Beta',
Expand Down