.. _usersGuide_11_corpusSearching: .. WARNING: DO NOT EDIT THIS FILE: AUTOMATICALLY GENERATED. PLEASE EDIT THE .py FILE DIRECTLY. User’s Guide, Chapter 11: Corpus Searching ========================================== One of music21’s important features is its capability to help users examine large bodies of musical works, or *corpora*. Music21 comes with a substantial corpus called the *core* corpus. When you download music21 you can immediately start working with the files in the corpus directory, including the complete chorales of Bach, many Haydn and Beethoven string quartets, three books of madrigals by Monteverdi, thousands of folk songs from the Essen and various ABC databases, and many more. To load a file from the corpus, simply call *corpus.parse* and assign that file to a variable: .. code:: ipython3 from music21 import * bach = corpus.parse('bach/bwv66.6') The ``music21`` core corpus comes with many thousands of works. All of them (or at least all the collections) are listed on the :ref:`Corpus Reference `. Users can also build their own corpora to index and quickly search their own collections on disk including multiple local corpora, for different projects, that can be accessed individually. This user’s guide will cover more about the corpus’s basic features. This chapter focuses on music21’s tools for extracting useful metadata - titles, locations, composers names, the key signatures used in each piece, total durations, ambitus (range) and so forth. This metadata is collected in *metadata bundles* for each corpus. The *corpus* module has tools to search these bundles and persist them on disk for later research. Types of corpora ---------------- Music21 works with three categories of *corpora*, made explicit via the ``corpus.Corpus`` abstract class. The first category is the *core* corpus, a large collection of musical works packaged with most music21 installations, including many works from the common practice era, and inumerable folk songs, in a variety of formats: .. code:: ipython3 coreCorpus = corpus.corpora.CoreCorpus() len(coreCorpus.getPaths()) .. parsed-literal:: :class: ipython-result 3193 .. note:: If you've installed a "no corpus" version of music21, you can still access the *core* corpus with a little work. Download the *core* corpus from music21's website, and install it on your system somewhere. Then, teach music21 where you installed it like this: >>> coreCorpus = corpus.corpora.CoreCorpus() >>> #_DOCS_SHOW coreCorpus.manualCoreCorpusPath = 'path/to/core/corpus' Local Corpus ------------ ``Music21`` also can have one or more *local* corpora–bodies of works provided and configured by individual music21 users for their own research. They will be covered in :ref:`Chapter 53`. Anyone wanting to use them can jump ahead immediately to that chapter, but for now we’ll continue with searching in the core corpus. .. code:: ipython3 localCorpus = corpus.corpora.LocalCorpus() You can add and remove paths from a *local* corpus with the ``addPath()`` and ``removePath()`` methods: .. code:: ipython3 localCorpus.addPath('~/Desktop') #_DOCS_SHOW localCorpus.directoryPaths ('/Users/myke/Desktop',) #_DOCS_HIDE .. parsed-literal:: :class: ipython-result ('/Users/myke/Desktop',) Currently, after adding paths to a corpus, you’ll need to rebuild the cache. .. code:: ipython3 #_DOCS_SHOW corpus.cacheMetadata() We hope that this won’t be necessary in the future. To remove a path, use the ``removePath()`` method. .. code:: ipython3 localCorpus.removePath('~/Desktop') .. parsed-literal:: :class: ipython-result /Users/cuthbert/git/music21base/music21/corpus/corpora.py: WARNING: local metadata cache: starting processing of paths: 0 /Users/cuthbert/git/music21base/music21/corpus/corpora.py: WARNING: cache: filename: /Users/cuthbert/music21temp/local.p.gz metadata.bundles: WARNING: MetadataBundle Modification Time: 1686436276.496933 metadata.bundles: WARNING: Skipped 0 sources already in cache. /Users/cuthbert/git/music21base/music21/corpus/corpora.py: WARNING: cache: writing time: 0.024 md items: 0 /Users/cuthbert/git/music21base/music21/corpus/corpora.py: WARNING: cache: filename: /Users/cuthbert/music21temp/local.p.gz By default, a call to ``corpus.parse`` or ``corpus.search`` will look for files in any corpus, core or local. Simple searches of the corpus ----------------------------- When you search the corpus, music21 examines each metadata object in the metadata bundle for the whole corpus and attempts to match your search string against the contents of the various search fields saved in that metadata object. You can use ``corpus.search()`` to search the metadata associated with all known corpora, *core*, *virtual* and even each *local* corpus: .. code:: ipython3 #_DOCS_SHOW sixEight = corpus.search('6/8') sixEight = corpus.corpora.CoreCorpus().search('6/8') #_DOCS_HIDE sixEight .. parsed-literal:: :class: ipython-result To work with all those pieces, you can parse treat the MetadataBundle like a list and call ``.parse()`` on any element: .. code:: ipython3 myPiece = sixEight[0].parse() myPiece.metadata.title .. parsed-literal:: :class: ipython-result "I'll Touzle your Kurchy." This will return a ``music21.stream.Score`` object which you can work with like any other stream. Or if you just want to see it, there’s a convenience ``.show()`` method you can call directly on a MetadataEntry. You can also search against a single ``Corpus`` instance, like this one which ignores anything in your local corpus: .. code:: ipython3 corpus.corpora.CoreCorpus().search('6/8') .. parsed-literal:: :class: ipython-result Because the result of every metadata search is also a metadata bundle, you can search your search results to do more complex searches. Remember that ``bachBundle`` is a collection of all works where the composer is Bach. Here we will limit to those pieces in 3/4 time: .. code:: ipython3 bachBundle = corpus.search('bach', 'composer') bachBundle .. parsed-literal:: :class: ipython-result .. code:: ipython3 bachBundle.search('3/4') .. parsed-literal:: :class: ipython-result Metadata search fields ---------------------- When you search metadata bundles, you can search either through every search field in every metadata instance, or through a single, specific search field. As we mentioned above, searching for “bach” as a composer renders different results from searching for the word “bach” in general: .. code:: ipython3 #_DOCS_SHOW corpus.search('bach', 'composer') corpus.corpora.CoreCorpus().search('bach', 'composer') #_DOCS_HIDE .. parsed-literal:: :class: ipython-result .. code:: ipython3 #_DOCS_SHOW corpus.search('bach', 'title') corpus.corpora.CoreCorpus().search('bach', 'title') #_DOCS_HIDE .. parsed-literal:: :class: ipython-result .. code:: ipython3 #_DOCS_SHOW corpus.search('bach') corpus.corpora.CoreCorpus().search('bach') #_DOCS_HIDE .. parsed-literal:: :class: ipython-result So what fields can we actually search through? You can find out like this (in v2, replace ``corpus.manager`` with ``corpus.corpora.Corpus``): .. code:: ipython3 for field in corpus.manager.listSearchFields(): print(field) .. parsed-literal:: :class: ipython-result abstract accessRights accompanyingMaterialWriter actNumber adapter afterwordAuthor alternativeTitle ambitus analyst annotator arranger associatedWork attributedComposer audience bibliographicCitation calligrapher collaborator collectionDesignation collotyper commentaryAuthor commission commissionedBy compiler composer composerAlias composerCorporate conceptor conductor conformsTo copyright corpusFilePath countryOfComposition date dateAccepted dateAvailable dateCopyrighted dateCreated dateFirstPublished dateIssued dateModified dateSubmitted dateValid dedicatedTo dedication description dialogAuthor distributor editor educationLevel electronicEditor electronicEncoder electronicPublisher electronicReleaseDate engraver etcher extent fileFormat fileNumber filePath firstPublisher format groupTitle hasFormat hasPart hasVersion identifier illuminator illustrator instructionalMethod instrumentalist introductionAuthor isFormatOf isPartOf isReferencedBy isReplacedBy isRequiredBy isVersionOf keySignatureFirst keySignatures language librettist license lithographer localeOfComposition lyricist manuscriptAccessAcknowledgement manuscriptLocation manuscriptSourceName medium metalEngraver movementName movementNumber musician noteCount number numberOfParts opusNumber orchestrator originalDocumentOwner originalEditor otherContributor otherDate parentTitle pitchHighest pitchLowest placeFirstPublished platemaker popularTitle printmaker producer proofreader provenance publicationTitle publisher publishersCatalogNumber quarterLength quotationsAuthor references relation replaces requires responsibleParty rightsHolder sceneNumber scholarlyCatalogAbbreviation scholarlyCatalogName scribe singer software source sourcePath subject suspectedComposer tableOfContents tempoFirst tempos textLanguage textOriginalLanguage timeSignatureFirst timeSignatures title transcriber translator type volume volumeNumber woodCutter woodEngraver This field has grown now that the development team is seeing how useful this searching method can be! Now that we know what all the search fields are, we can search through some of the more obscure corners of the *core* corpus: .. code:: ipython3 #_DOCS_SHOW corpus.search('taiwan', 'locale') corpus.corpora.CoreCorpus().search('taiwan', 'locale') #_DOCS_HIDE .. parsed-literal:: :class: ipython-result What if you are not searching for an exact match? If you’re searching for short pieces, you probably don’t want to find pieces with exactly 1 note then union that set with pieces with exactly 2 notes, etc. Or for pieces from the 19th century, you won’t want to search for 1801, 1802, etc. What you can do is set up a “predicate callable” which is a function (either a full python ``def`` statement or a short ``lambda`` function) to filter the results. Each piece will be checked against your predicate and only those that return true. Here we’ll search for pieces with between 400 and 500 notes, only in the ``core`` corpus: .. code:: ipython3 predicate = lambda x: 400 < x < 500 corpus.corpora.CoreCorpus().search(predicate, 'noteCount') .. parsed-literal:: :class: ipython-result You can also pass in compiled regular expressions into the search. In this case we will use a regular expression likely to find Handel and Haydn and perhaps not much else: .. code:: ipython3 import re haydnOrHandel = re.compile(r'ha.d.*', re.IGNORECASE) #_DOCS_SHOW corpus.search(haydnOrHandel) corpus.corpora.CoreCorpus().search(haydnOrHandel) #_DOCS_HIDE .. parsed-literal:: :class: ipython-result Unfortunately this really wasn’t a good search, since we also got folk songs with the title of “Shandy”. Best to use a ‘\*^\*’ search to match at the beginning of the word only: .. code:: ipython3 haydnOrHandel = re.compile(r'^ha.d.*', re.IGNORECASE) #_DOCS_SHOW corpus.search(haydnOrHandel) corpus.corpora.CoreCorpus().search(haydnOrHandel) #_DOCS_HIDE .. parsed-literal:: :class: ipython-result We’ve now gone fairly high level in our searching. We will return to the lowest level in :ref:`Chapter 12: The Music21Object `