Music21 works only because the code has been extensively tested so
that new changes are confirmed not to break existing code. Time spent
testing always slows down coding in the short run, but long experience
has shown that it saves coding time in the long run. Contributions to
music21 without tests will always be rejected, so as developers,
it’s important that code be well tested.
All functions/methods/etc. need documentation and at least one test. Oh, and that test must pass.
music21 module should have exactly one class called
which should run the tests for the module. (If the module is getting
very big, it is possible to have this class reside in a different
module, so long as it is imported into the module that it tests).
Make the Test class be a subclass of
unittest and each of the test
methods within it must begin with
test, for instance
import unittest class Test(unittest.TestCase): def testNotesHaveStemDirection(self): from music21 import note n = note.Note('C#') self.assertTrue(hasattr(n, 'stemDirection')) n.stemDirection = 'down' self.assertEqual(n.stemDirection, 'down')
Test class must not produce any output if the tests succeed.
They must not create new windows, lose the focus from the current
window, play music, etc. They must not rely on modules that are not part
music21 ecosystem (i.e., scipy).
If you need to create a second test that generates external output,
create a second
should run quickly: what “quickly” means depends on the complexity of
what is being contributed; a major new module can add a few seconds to
the test suite. A tiny addition, should add milliseconds at most. If
your test adds 10 seconds to running, that’s 10 seconds that everyone
will have to wait when running test. If you need to have a test that
takes a lot of time, create a second Test class called
End your file so that if the module is run directly, it calls these tests.
if __name__ == "__main__": import music21 music21.mainTest(Test)
. ---------------------------------------------------------------------- Ran 1 test in 0.001s OK
music21.mainTest(Test) you can write
music21.mainTest(Test, verbose=True) or while debugging
music21.mainTest(Test, runTest='testNotesHaveStemDirection') to run
a single test. Or to run a single test, you can pass it from the command
python3 note.py testNotesHaveStemDirection
You can also call the tests for the module with:
python3 -m music21.note
One great feature Python has is the ability to write tests within the
documentation itself (“
doctest”). We use a lot of these to
demonstrate expected behavior in calling a method or function and also
to test that the expected behavior works. However, doctests do take up
space in the documentation, so do not write anything that can’t be read
(use the module-level testing instead).
Doc tests are written like so:
def flipStemDirection(n): ''' Takes in a note with a `.stemDirection` of 'up' or 'down' and transforms it to the other. Does nothing if the `stemDirection` is not defined. Notes without stem direction of `up` or `down` are left alone: >>> n = note.Note() >>> n.stemDirection 'unspecified' >>> flipStemDirection(n) >>> n.stemDirection 'unspecified' Here we flip an `up` to `down` and back again. >>> n.stemDirection = 'up' >>> flipStemDirection(n) >>> n.stemDirection 'down' >>> flipStemDirection(n) >>> n.stemDirection 'up' Objects without `.stemDirection` raise a Music21Exception >>> r = note.Rest() >>> flipStemDirection(r) Traceback (most recent call last): music21.exceptions21.Music21Exception: flipStemDirection only works on Notes ''' try: sd = n.stemDirection except AttributeError: raise exceptions21.Music21Exception("flipStemDirection only works on Notes") if sd == 'up': n.stemDirection = 'down' elif sd == 'down': n.stemDirection = 'up'
Generally in a doctest, the full path to the function (assuming
from music21 import *) must be given, so if this function were in a
stem.py file, it would need to be called as
stem.flipStemDirection. The reason for this is that we’re writing
tests and docs at the same time, so we want to document usage as our
users (who are suggested to run
from music21 import *) will need to
run the routine.
All doctests start with a clean slate of variables, so variables created in a prior doctest will not be available in another.
Running all tests¶
Once you’ve made sure your contributions to a module work, you’ll want to run all the tests in the system.
The simplest way of running all the tests for
music21 is to run:
This will use
(n - 1) of your n-processor system to run all of the
tests and will print a lot of output along the way. The tests run in
reverse-alphabetical order (with some slow tests boosted to the
beginning). All failed tests will appear near the bottom of the output.
A few modules are not tested in
multiprocessTest because they do not
play well with multiprocessing or are in obscure subdirectories. They do
not need to be tested on a general basis, but before a major release
they should be tested with:
Before a release, this should be run with every version of Python that is supported (and preferably on both Mac and PC). This can be a pain, so fortunately we have…
Continuous Integration Tests¶
Upon any commit or pull request, continuous integration testing at Travis-CI runs. See:
and hopefully it’s running Green at least for the master branch.
Travis-CI runs the
travisMain() function of test/singleCoreAll which
is the same as the
main() function except it exits with 0 or -1 or
whatever the UNIX convention for failure is (I always forget–that’s why
I coded it). Thanks to the great folks at Travis-CI, the risks of
accepting pull requests is lower (though not zero, especially in cases
where there are changes to external output or to parsing files outside
the corpus). Although the amount of time to run varies depending on the
load of travis-ci, keep a look out for changes that significantly slow
down the system – they may indicate an inefficient algorithm.
All pull requests must pass Travis-CI to be accepted. It doesn’t matter that “that bug was probably there before” – it’s time to fix it or to get help fixing it.
Code Coverage Tests¶
One of the systems running on Travis will always run slower than the
rest. This system is running
coverage to measure how much of the
music21 code is being run. For the results see:
While 100% code coverage is impossible, increasing the coverage percentage with each new contribution is essential. We’re at 90% code coverage and get the green badge of pride on our README.md file, and we like that!
If something should be impossible to trigger, but the code is being
maintained for some reason, you can add “
# pragma: no cover” to the
line or to the fork, but this is generally a no-no.
Testing the User’s Guide¶
Most new contributions will not change the documentation, but
occasionally it will. To make sure that the documentation is correct,
testDocumentation.py (under Python 3 only) in the
documentation directory (outside the
music21 source code
directory). If there are any changes, check to make sure they make sense
(maybe there are some new files in the corpus that is changing the count
somewhere?) and if they do, rerun the notebook with that documentation
(line by line!) and make sure that the new output makes better sense
than the previous.
This can take some time, so it does not need to be done for every commit.
music21 project relies on great tests, both to ensure everything
is working properly and also to save time for those who will support
your code in the future. One of the main things we can always use
contributors to is writing tests for code that is under tested, so
please feel free to grab some code from Coveralls and contribute tests
for it. It may sound like a small contribution, but it’s one that will
be greatly appreciated.