You cannot select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
2200 lines
97 KiB
Plaintext
2200 lines
97 KiB
Plaintext
.. Copyright (C) 2001-2020 NLTK Project
|
|
.. For license information, see LICENSE.TXT
|
|
|
|
================
|
|
Corpus Readers
|
|
================
|
|
|
|
The `nltk.corpus` package defines a collection of *corpus reader*
|
|
classes, which can be used to access the contents of a diverse set of
|
|
corpora. The list of available corpora is given at:
|
|
|
|
http://www.nltk.org/nltk_data/
|
|
|
|
Each corpus reader class is specialized to handle a specific
|
|
corpus format. In addition, the `nltk.corpus` package automatically
|
|
creates a set of corpus reader instances that can be used to access
|
|
the corpora in the NLTK data package.
|
|
Section `Corpus Reader Objects`_ ("Corpus Reader Objects") describes
|
|
the corpus reader instances that can be used to read the corpora in
|
|
the NLTK data package. Section `Corpus Reader Classes`_ ("Corpus
|
|
Reader Classes") describes the corpus reader classes themselves, and
|
|
discusses the issues involved in creating new corpus reader objects
|
|
and new corpus reader classes. Section `Regression Tests`_
|
|
("Regression Tests") contains regression tests for the corpus readers
|
|
and associated functions and classes.
|
|
|
|
.. contents:: **Table of Contents**
|
|
:depth: 2
|
|
:backlinks: none
|
|
|
|
---------------------
|
|
Corpus Reader Objects
|
|
---------------------
|
|
|
|
Overview
|
|
========
|
|
|
|
NLTK includes a diverse set of corpora which can be
|
|
read using the ``nltk.corpus`` package. Each corpus is accessed by
|
|
means of a "corpus reader" object from ``nltk.corpus``:
|
|
|
|
>>> import nltk.corpus
|
|
>>> # The Brown corpus:
|
|
>>> print(str(nltk.corpus.brown).replace('\\\\','/'))
|
|
<CategorizedTaggedCorpusReader in '.../corpora/brown'...>
|
|
>>> # The Penn Treebank Corpus:
|
|
>>> print(str(nltk.corpus.treebank).replace('\\\\','/'))
|
|
<BracketParseCorpusReader in '.../corpora/treebank/combined'...>
|
|
>>> # The Name Genders Corpus:
|
|
>>> print(str(nltk.corpus.names).replace('\\\\','/'))
|
|
<WordListCorpusReader in '.../corpora/names'...>
|
|
>>> # The Inaugural Address Corpus:
|
|
>>> print(str(nltk.corpus.inaugural).replace('\\\\','/'))
|
|
<PlaintextCorpusReader in '.../corpora/inaugural'...>
|
|
|
|
Most corpora consist of a set of files, each containing a document (or
|
|
other pieces of text). A list of identifiers for these files is
|
|
accessed via the ``fileids()`` method of the corpus reader:
|
|
|
|
>>> nltk.corpus.treebank.fileids() # doctest: +ELLIPSIS
|
|
['wsj_0001.mrg', 'wsj_0002.mrg', 'wsj_0003.mrg', 'wsj_0004.mrg', ...]
|
|
>>> nltk.corpus.inaugural.fileids() # doctest: +ELLIPSIS
|
|
['1789-Washington.txt', '1793-Washington.txt', '1797-Adams.txt', ...]
|
|
|
|
Each corpus reader provides a variety of methods to read data from the
|
|
corpus, depending on the format of the corpus. For example, plaintext
|
|
corpora support methods to read the corpus as raw text, a list of
|
|
words, a list of sentences, or a list of paragraphs.
|
|
|
|
>>> from nltk.corpus import inaugural
|
|
>>> inaugural.raw('1789-Washington.txt') # doctest: +ELLIPSIS
|
|
'Fellow-Citizens of the Senate ...'
|
|
>>> inaugural.words('1789-Washington.txt')
|
|
['Fellow', '-', 'Citizens', 'of', 'the', ...]
|
|
>>> inaugural.sents('1789-Washington.txt') # doctest: +ELLIPSIS
|
|
[['Fellow', '-', 'Citizens'...], ['Among', 'the', 'vicissitudes'...]...]
|
|
>>> inaugural.paras('1789-Washington.txt') # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
|
|
[[['Fellow', '-', 'Citizens'...]],
|
|
[['Among', 'the', 'vicissitudes'...],
|
|
['On', 'the', 'one', 'hand', ',', 'I'...]...]...]
|
|
|
|
Each of these reader methods may be given a single document's item
|
|
name or a list of document item names. When given a list of document
|
|
item names, the reader methods will concatenate together the contents
|
|
of the individual documents.
|
|
|
|
>>> l1 = len(inaugural.words('1789-Washington.txt'))
|
|
>>> l2 = len(inaugural.words('1793-Washington.txt'))
|
|
>>> l3 = len(inaugural.words(['1789-Washington.txt', '1793-Washington.txt']))
|
|
>>> print('%s+%s == %s' % (l1, l2, l3))
|
|
1538+147 == 1685
|
|
|
|
If the reader methods are called without any arguments, they will
|
|
typically load all documents in the corpus.
|
|
|
|
>>> len(inaugural.words())
|
|
149797
|
|
|
|
If a corpus contains a README file, it can be accessed with a ``readme()`` method:
|
|
|
|
>>> inaugural.readme()[:32]
|
|
'C-Span Inaugural Address Corpus\n'
|
|
|
|
Plaintext Corpora
|
|
=================
|
|
|
|
Here are the first few words from each of NLTK's plaintext corpora:
|
|
|
|
>>> nltk.corpus.abc.words()
|
|
['PM', 'denies', 'knowledge', 'of', 'AWB', ...]
|
|
>>> nltk.corpus.genesis.words()
|
|
['In', 'the', 'beginning', 'God', 'created', ...]
|
|
>>> nltk.corpus.gutenberg.words(fileids='austen-emma.txt')
|
|
['[', 'Emma', 'by', 'Jane', 'Austen', '1816', ...]
|
|
>>> nltk.corpus.inaugural.words()
|
|
['Fellow', '-', 'Citizens', 'of', 'the', ...]
|
|
>>> nltk.corpus.state_union.words()
|
|
['PRESIDENT', 'HARRY', 'S', '.', 'TRUMAN', "'", ...]
|
|
>>> nltk.corpus.webtext.words()
|
|
['Cookie', 'Manager', ':', '"', 'Don', "'", 't', ...]
|
|
|
|
Tagged Corpora
|
|
==============
|
|
|
|
In addition to the plaintext corpora, NLTK's data package also
|
|
contains a wide variety of annotated corpora. For example, the Brown
|
|
Corpus is annotated with part-of-speech tags, and defines additional
|
|
methods ``tagged_*()`` which words as `(word,tag)` tuples, rather
|
|
than just bare word strings.
|
|
|
|
>>> from nltk.corpus import brown
|
|
>>> print(brown.words())
|
|
['The', 'Fulton', 'County', 'Grand', 'Jury', ...]
|
|
>>> print(brown.tagged_words())
|
|
[('The', 'AT'), ('Fulton', 'NP-TL'), ...]
|
|
>>> print(brown.sents()) # doctest: +ELLIPSIS
|
|
[['The', 'Fulton', 'County'...], ['The', 'jury', 'further'...], ...]
|
|
>>> print(brown.tagged_sents()) # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
|
|
[[('The', 'AT'), ('Fulton', 'NP-TL')...],
|
|
[('The', 'AT'), ('jury', 'NN'), ('further', 'RBR')...]...]
|
|
>>> print(brown.paras(categories='reviews')) # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
|
|
[[['It', 'is', 'not', 'news', 'that', 'Nathan', 'Milstein'...],
|
|
['Certainly', 'not', 'in', 'Orchestra', 'Hall', 'where'...]],
|
|
[['There', 'was', 'about', 'that', 'song', 'something', ...],
|
|
['Not', 'the', 'noblest', 'performance', 'we', 'have', ...], ...], ...]
|
|
>>> print(brown.tagged_paras(categories='reviews')) # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
|
|
[[[('It', 'PPS'), ('is', 'BEZ'), ('not', '*'), ...],
|
|
[('Certainly', 'RB'), ('not', '*'), ('in', 'IN'), ...]],
|
|
[[('There', 'EX'), ('was', 'BEDZ'), ('about', 'IN'), ...],
|
|
[('Not', '*'), ('the', 'AT'), ('noblest', 'JJT'), ...], ...], ...]
|
|
|
|
Similarly, the Indian Language POS-Tagged Corpus includes samples of
|
|
Indian text annotated with part-of-speech tags:
|
|
|
|
>>> from nltk.corpus import indian
|
|
>>> print(indian.words()) # doctest: +SKIP
|
|
['\xe0\xa6\xae\xe0\xa6\xb9\xe0\xa6\xbf\...',
|
|
'\xe0\xa6\xb8\xe0\xa6\xa8\xe0\xa7\x8d\xe0...', ...]
|
|
>>> print(indian.tagged_words()) # doctest: +SKIP
|
|
[('\xe0\xa6\xae\xe0\xa6\xb9\xe0\xa6\xbf...', 'NN'),
|
|
('\xe0\xa6\xb8\xe0\xa6\xa8\xe0\xa7\x8d\xe0...', 'NN'), ...]
|
|
|
|
Several tagged corpora support access to a simplified, universal tagset, e.g. where all nouns
|
|
tags are collapsed to a single category ``NOUN``:
|
|
|
|
>>> print(brown.tagged_sents(tagset='universal')) # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
|
|
[[('The', 'DET'), ('Fulton', 'NOUN'), ('County', 'NOUN'), ('Grand', 'ADJ'), ('Jury', 'NOUN'), ...],
|
|
[('The', 'DET'), ('jury', 'NOUN'), ('further', 'ADV'), ('said', 'VERB'), ('in', 'ADP'), ...]...]
|
|
>>> from nltk.corpus import conll2000, switchboard
|
|
>>> print(conll2000.tagged_words(tagset='universal')) # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
|
|
[('Confidence', 'NOUN'), ('in', 'ADP'), ...]
|
|
|
|
Use ``nltk.app.pos_concordance()`` to access a GUI for searching tagged corpora.
|
|
|
|
Chunked Corpora
|
|
===============
|
|
|
|
The CoNLL corpora also provide chunk structures, which are encoded as
|
|
flat trees. The CoNLL 2000 Corpus includes phrasal chunks; and the
|
|
CoNLL 2002 Corpus includes named entity chunks.
|
|
|
|
>>> from nltk.corpus import conll2000, conll2002
|
|
>>> print(conll2000.sents()) # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
|
|
[['Confidence', 'in', 'the', 'pound', 'is', 'widely', ...],
|
|
['Chancellor', 'of', 'the', 'Exchequer', ...], ...]
|
|
>>> for tree in conll2000.chunked_sents()[:2]:
|
|
... print(tree) # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
|
|
(S
|
|
(NP Confidence/NN)
|
|
(PP in/IN)
|
|
(NP the/DT pound/NN)
|
|
(VP is/VBZ widely/RB expected/VBN to/TO take/VB)
|
|
(NP another/DT sharp/JJ dive/NN)
|
|
if/IN
|
|
...)
|
|
(S
|
|
Chancellor/NNP
|
|
(PP of/IN)
|
|
(NP the/DT Exchequer/NNP)
|
|
...)
|
|
>>> print(conll2002.sents()) # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
|
|
[['Sao', 'Paulo', '(', 'Brasil', ')', ',', ...], ['-'], ...]
|
|
>>> for tree in conll2002.chunked_sents()[:2]:
|
|
... print(tree) # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
|
|
(S
|
|
(LOC Sao/NC Paulo/VMI)
|
|
(/Fpa
|
|
(LOC Brasil/NC)
|
|
)/Fpt
|
|
...)
|
|
(S -/Fg)
|
|
|
|
.. note:: Since the CONLL corpora do not contain paragraph break
|
|
information, these readers do not support the ``para()`` method.)
|
|
|
|
.. warning:: if you call the conll corpora reader methods without any
|
|
arguments, they will return the contents of the entire corpus,
|
|
*including* the 'test' portions of the corpus.)
|
|
|
|
SemCor is a subset of the Brown corpus tagged with WordNet senses and
|
|
named entities. Both kinds of lexical items include multiword units,
|
|
which are encoded as chunks (senses and part-of-speech tags pertain
|
|
to the entire chunk).
|
|
|
|
>>> from nltk.corpus import semcor
|
|
>>> semcor.words()
|
|
['The', 'Fulton', 'County', 'Grand', 'Jury', ...]
|
|
>>> semcor.chunks()
|
|
[['The'], ['Fulton', 'County', 'Grand', 'Jury'], ...]
|
|
>>> semcor.sents() # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
|
|
[['The', 'Fulton', 'County', 'Grand', 'Jury', 'said', ...],
|
|
['The', 'jury', 'further', 'said', ...], ...]
|
|
>>> semcor.chunk_sents() # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
|
|
[[['The'], ['Fulton', 'County', 'Grand', 'Jury'], ['said'], ...
|
|
['.']], [['The'], ['jury'], ['further'], ['said'], ... ['.']], ...]
|
|
>>> list(map(str, semcor.tagged_chunks(tag='both')[:3]))
|
|
['(DT The)', "(Lemma('group.n.01.group') (NE (NNP Fulton County Grand Jury)))", "(Lemma('state.v.01.say') (VB said))"]
|
|
>>> [[str(c) for c in s] for s in semcor.tagged_sents(tag='both')[:2]]
|
|
[['(DT The)', "(Lemma('group.n.01.group') (NE (NNP Fulton County Grand Jury)))", ...
|
|
'(None .)'], ['(DT The)', ... '(None .)']]
|
|
|
|
|
|
The IEER corpus is another chunked corpus. This corpus is unusual in
|
|
that each corpus item contains multiple documents. (This reflects the
|
|
fact that each corpus file contains multiple documents.) The IEER
|
|
corpus defines the `parsed_docs` method, which returns the documents
|
|
in a given item as `IEERDocument` objects:
|
|
|
|
>>> from nltk.corpus import ieer
|
|
>>> ieer.fileids() # doctest: +NORMALIZE_WHITESPACE
|
|
['APW_19980314', 'APW_19980424', 'APW_19980429',
|
|
'NYT_19980315', 'NYT_19980403', 'NYT_19980407']
|
|
>>> docs = ieer.parsed_docs('APW_19980314')
|
|
>>> print(docs[0])
|
|
<IEERDocument APW19980314.0391: 'Kenyans protest tax hikes'>
|
|
>>> print(docs[0].docno)
|
|
APW19980314.0391
|
|
>>> print(docs[0].doctype)
|
|
NEWS STORY
|
|
>>> print(docs[0].date_time)
|
|
03/14/1998 10:36:00
|
|
>>> print(docs[0].headline)
|
|
(DOCUMENT Kenyans protest tax hikes)
|
|
>>> print(docs[0].text) # doctest: +ELLIPSIS
|
|
(DOCUMENT
|
|
(LOCATION NAIROBI)
|
|
,
|
|
(LOCATION Kenya)
|
|
(
|
|
(ORGANIZATION AP)
|
|
)
|
|
_
|
|
(CARDINAL Thousands)
|
|
of
|
|
laborers,
|
|
...
|
|
on
|
|
(DATE Saturday)
|
|
...)
|
|
|
|
Parsed Corpora
|
|
==============
|
|
|
|
The Treebank corpora provide a syntactic parse for each sentence. The
|
|
NLTK data package includes a 10% sample of the Penn Treebank (in
|
|
``treebank``), as well as the Sinica Treebank (in ``sinica_treebank``).
|
|
|
|
Reading the Penn Treebank (Wall Street Journal sample):
|
|
|
|
>>> from nltk.corpus import treebank
|
|
>>> print(treebank.fileids()) # doctest: +ELLIPSIS
|
|
['wsj_0001.mrg', 'wsj_0002.mrg', 'wsj_0003.mrg', 'wsj_0004.mrg', ...]
|
|
>>> print(treebank.words('wsj_0003.mrg'))
|
|
['A', 'form', 'of', 'asbestos', 'once', 'used', ...]
|
|
>>> print(treebank.tagged_words('wsj_0003.mrg'))
|
|
[('A', 'DT'), ('form', 'NN'), ('of', 'IN'), ...]
|
|
>>> print(treebank.parsed_sents('wsj_0003.mrg')[0]) # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
|
|
(S
|
|
(S-TPC-1
|
|
(NP-SBJ
|
|
(NP (NP (DT A) (NN form)) (PP (IN of) (NP (NN asbestos))))
|
|
(RRC ...)...)...)
|
|
...
|
|
(VP (VBD reported) (SBAR (-NONE- 0) (S (-NONE- *T*-1))))
|
|
(. .))
|
|
|
|
If you have access to a full installation of the Penn Treebank, NLTK
|
|
can be configured to load it as well. Download the ``ptb`` package,
|
|
and in the directory ``nltk_data/corpora/ptb`` place the ``BROWN``
|
|
and ``WSJ`` directories of the Treebank installation (symlinks work
|
|
as well). Then use the ``ptb`` module instead of ``treebank``:
|
|
|
|
>>> from nltk.corpus import ptb
|
|
>>> print(ptb.fileids()) # doctest: +SKIP
|
|
['BROWN/CF/CF01.MRG', 'BROWN/CF/CF02.MRG', 'BROWN/CF/CF03.MRG', 'BROWN/CF/CF04.MRG', ...]
|
|
>>> print(ptb.words('WSJ/00/WSJ_0003.MRG')) # doctest: +SKIP
|
|
['A', 'form', 'of', 'asbestos', 'once', 'used', '*', ...]
|
|
>>> print(ptb.tagged_words('WSJ/00/WSJ_0003.MRG')) # doctest: +SKIP
|
|
[('A', 'DT'), ('form', 'NN'), ('of', 'IN'), ...]
|
|
|
|
...and so forth, like ``treebank`` but with extended fileids. Categories
|
|
specified in ``allcats.txt`` can be used to filter by genre; they consist
|
|
of ``news`` (for WSJ articles) and names of the Brown subcategories
|
|
(``fiction``, ``humor``, ``romance``, etc.):
|
|
|
|
>>> ptb.categories() # doctest: +SKIP
|
|
['adventure', 'belles_lettres', 'fiction', 'humor', 'lore', 'mystery', 'news', 'romance', 'science_fiction']
|
|
>>> print(ptb.fileids('news')) # doctest: +SKIP
|
|
['WSJ/00/WSJ_0001.MRG', 'WSJ/00/WSJ_0002.MRG', 'WSJ/00/WSJ_0003.MRG', ...]
|
|
>>> print(ptb.words(categories=['humor','fiction'])) # doctest: +SKIP
|
|
['Thirty-three', 'Scotty', 'did', 'not', 'go', 'back', ...]
|
|
|
|
As PropBank and NomBank depend on the (WSJ portion of the) Penn Treebank,
|
|
the modules ``propbank_ptb`` and ``nombank_ptb`` are provided for access
|
|
to a full PTB installation.
|
|
|
|
Reading the Sinica Treebank:
|
|
|
|
>>> from nltk.corpus import sinica_treebank
|
|
>>> print(sinica_treebank.sents()) # doctest: +SKIP
|
|
[['\xe4\xb8\x80'], ['\xe5\x8f\x8b\xe6\x83\x85'], ...]
|
|
>>> sinica_treebank.parsed_sents()[25] # doctest: +SKIP
|
|
Tree('S',
|
|
[Tree('NP',
|
|
[Tree('Nba', ['\xe5\x98\x89\xe7\x8f\x8d'])]),
|
|
Tree('V\xe2\x80\xa7\xe5\x9c\xb0',
|
|
[Tree('VA11', ['\xe4\xb8\x8d\xe5\x81\x9c']),
|
|
Tree('DE', ['\xe7\x9a\x84'])]),
|
|
Tree('VA4', ['\xe5\x93\xad\xe6\xb3\xa3'])])
|
|
|
|
Reading the CoNLL 2007 Dependency Treebanks:
|
|
|
|
>>> from nltk.corpus import conll2007
|
|
>>> conll2007.sents('esp.train')[0] # doctest: +SKIP
|
|
['El', 'aumento', 'del', 'índice', 'de', 'desempleo', ...]
|
|
>>> conll2007.parsed_sents('esp.train')[0] # doctest: +SKIP
|
|
<DependencyGraph with 38 nodes>
|
|
>>> print(conll2007.parsed_sents('esp.train')[0].tree()) # doctest: +SKIP
|
|
(fortaleció
|
|
(aumento El (del (índice (de (desempleo estadounidense)))))
|
|
hoy
|
|
considerablemente
|
|
(al
|
|
(euro
|
|
(cotizaba
|
|
,
|
|
que
|
|
(a (15.35 las GMT))
|
|
se
|
|
(en (mercado el (de divisas) (de Fráncfort)))
|
|
(a 0,9452_dólares)
|
|
(frente_a , (0,9349_dólares los (de (mañana esta)))))))
|
|
.)
|
|
|
|
Word Lists and Lexicons
|
|
=======================
|
|
|
|
The NLTK data package also includes a number of lexicons and word
|
|
lists. These are accessed just like text corpora. The following
|
|
examples illustrate the use of the wordlist corpora:
|
|
|
|
>>> from nltk.corpus import names, stopwords, words
|
|
>>> words.fileids()
|
|
['en', 'en-basic']
|
|
>>> words.words('en') # doctest: +ELLIPSIS
|
|
['A', 'a', 'aa', 'aal', 'aalii', 'aam', 'Aani', 'aardvark', 'aardwolf', ...]
|
|
|
|
>>> stopwords.fileids() # doctest: +ELLIPSIS
|
|
['arabic', 'azerbaijani', 'danish', 'dutch', 'english', 'finnish', 'french', ...]
|
|
>>> sorted(stopwords.words('portuguese')) # doctest: +ELLIPSIS
|
|
['a', 'ao', 'aos', 'aquela', 'aquelas', 'aquele', 'aqueles', ...]
|
|
>>> names.fileids()
|
|
['female.txt', 'male.txt']
|
|
>>> names.words('male.txt') # doctest: +ELLIPSIS
|
|
['Aamir', 'Aaron', 'Abbey', 'Abbie', 'Abbot', 'Abbott', ...]
|
|
>>> names.words('female.txt') # doctest: +ELLIPSIS
|
|
['Abagael', 'Abagail', 'Abbe', 'Abbey', 'Abbi', 'Abbie', ...]
|
|
|
|
The CMU Pronunciation Dictionary corpus contains pronounciation
|
|
transcriptions for over 100,000 words. It can be accessed as a list
|
|
of entries (where each entry consists of a word, an identifier, and a
|
|
transcription) or as a dictionary from words to lists of
|
|
transcriptions. Transcriptions are encoded as tuples of phoneme
|
|
strings.
|
|
|
|
>>> from nltk.corpus import cmudict
|
|
>>> print(cmudict.entries()[653:659]) # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
|
|
[('acetate', ['AE1', 'S', 'AH0', 'T', 'EY2', 'T']),
|
|
('acetic', ['AH0', 'S', 'EH1', 'T', 'IH0', 'K']),
|
|
('acetic', ['AH0', 'S', 'IY1', 'T', 'IH0', 'K']),
|
|
('aceto', ['AA0', 'S', 'EH1', 'T', 'OW0']),
|
|
('acetochlor', ['AA0', 'S', 'EH1', 'T', 'OW0', 'K', 'L', 'AO2', 'R']),
|
|
('acetone', ['AE1', 'S', 'AH0', 'T', 'OW2', 'N'])]
|
|
>>> # Load the entire cmudict corpus into a Python dictionary:
|
|
>>> transcr = cmudict.dict()
|
|
>>> print([transcr[w][0] for w in 'Natural Language Tool Kit'.lower().split()]) # doctest: +NORMALIZE_WHITESPACE
|
|
[['N', 'AE1', 'CH', 'ER0', 'AH0', 'L'],
|
|
['L', 'AE1', 'NG', 'G', 'W', 'AH0', 'JH'],
|
|
['T', 'UW1', 'L'],
|
|
['K', 'IH1', 'T']]
|
|
|
|
|
|
WordNet
|
|
=======
|
|
|
|
Please see the separate WordNet howto.
|
|
|
|
FrameNet
|
|
========
|
|
|
|
Please see the separate FrameNet howto.
|
|
|
|
PropBank
|
|
========
|
|
|
|
Please see the separate PropBank howto.
|
|
|
|
SentiWordNet
|
|
============
|
|
|
|
Please see the separate SentiWordNet howto.
|
|
|
|
Categorized Corpora
|
|
===================
|
|
|
|
Several corpora included with NLTK contain documents that have been categorized for
|
|
topic, genre, polarity, etc. In addition to the standard corpus interface, these
|
|
corpora provide access to the list of categories and the mapping between the documents
|
|
and their categories (in both directions). Access the categories using the ``categories()``
|
|
method, e.g.:
|
|
|
|
>>> from nltk.corpus import brown, movie_reviews, reuters
|
|
>>> brown.categories() # doctest: +NORMALIZE_WHITESPACE
|
|
['adventure', 'belles_lettres', 'editorial', 'fiction', 'government', 'hobbies', 'humor',
|
|
'learned', 'lore', 'mystery', 'news', 'religion', 'reviews', 'romance', 'science_fiction']
|
|
>>> movie_reviews.categories()
|
|
['neg', 'pos']
|
|
>>> reuters.categories() # doctest: +NORMALIZE_WHITESPACE +ELLIPSIS
|
|
['acq', 'alum', 'barley', 'bop', 'carcass', 'castor-oil', 'cocoa',
|
|
'coconut', 'coconut-oil', 'coffee', 'copper', 'copra-cake', 'corn',
|
|
'cotton', 'cotton-oil', 'cpi', 'cpu', 'crude', 'dfl', 'dlr', ...]
|
|
|
|
This method has an optional argument that specifies a document or a list
|
|
of documents, allowing us to map from (one or more) documents to (one or more) categories:
|
|
|
|
>>> brown.categories('ca01')
|
|
['news']
|
|
>>> brown.categories(['ca01','cb01'])
|
|
['editorial', 'news']
|
|
>>> reuters.categories('training/9865')
|
|
['barley', 'corn', 'grain', 'wheat']
|
|
>>> reuters.categories(['training/9865', 'training/9880'])
|
|
['barley', 'corn', 'grain', 'money-fx', 'wheat']
|
|
|
|
We can go back the other way using the optional argument of the ``fileids()`` method:
|
|
|
|
>>> reuters.fileids('barley') # doctest: +ELLIPSIS
|
|
['test/15618', 'test/15649', 'test/15676', 'test/15728', 'test/15871', ...]
|
|
|
|
Both the ``categories()`` and ``fileids()`` methods return a sorted list containing
|
|
no duplicates.
|
|
|
|
In addition to mapping between categories and documents, these corpora permit
|
|
direct access to their contents via the categories. Instead of accessing a subset
|
|
of a corpus by specifying one or more fileids, we can identify one or more categories, e.g.:
|
|
|
|
>>> brown.tagged_words(categories='news')
|
|
[('The', 'AT'), ('Fulton', 'NP-TL'), ...]
|
|
>>> brown.sents(categories=['editorial','reviews']) # doctest: +NORMALIZE_WHITESPACE
|
|
[['Assembly', 'session', 'brought', 'much', 'good'], ['The', 'General',
|
|
'Assembly', ',', 'which', 'adjourns', 'today', ',', 'has', 'performed',
|
|
'in', 'an', 'atmosphere', 'of', 'crisis', 'and', 'struggle', 'from',
|
|
'the', 'day', 'it', 'convened', '.'], ...]
|
|
|
|
Note that it is an error to specify both documents and categories.
|
|
|
|
In the context of a text categorization system, we can easily test if the
|
|
category assigned to a document is correct as follows:
|
|
|
|
>>> def classify(doc): return 'news' # Trivial classifier
|
|
>>> doc = 'ca01'
|
|
>>> classify(doc) in brown.categories(doc)
|
|
True
|
|
|
|
|
|
Other Corpora
|
|
=============
|
|
|
|
comparative_sentences
|
|
---------------------
|
|
A list of sentences from various sources, especially reviews and articles. Each
|
|
line contains one sentence; sentences were separated by using a sentence tokenizer.
|
|
Comparative sentences have been annotated with their type, entities, features and
|
|
keywords.
|
|
|
|
>>> from nltk.corpus import comparative_sentences
|
|
>>> comparison = comparative_sentences.comparisons()[0]
|
|
>>> comparison.text
|
|
['its', 'fast-forward', 'and', 'rewind', 'work', 'much', 'more', 'smoothly',
|
|
'and', 'consistently', 'than', 'those', 'of', 'other', 'models', 'i', "'ve",
|
|
'had', '.']
|
|
>>> comparison.entity_2
|
|
'models'
|
|
>>> (comparison.feature, comparison.keyword)
|
|
('rewind', 'more')
|
|
>>> len(comparative_sentences.comparisons())
|
|
853
|
|
|
|
opinion_lexicon
|
|
---------------
|
|
A list of positive and negative opinion words or sentiment words for English.
|
|
|
|
>>> from nltk.corpus import opinion_lexicon
|
|
>>> opinion_lexicon.words()[:4]
|
|
['2-faced', '2-faces', 'abnormal', 'abolish']
|
|
|
|
The OpinionLexiconCorpusReader also provides shortcuts to retrieve positive/negative
|
|
words:
|
|
|
|
>>> opinion_lexicon.negative()[:4]
|
|
['2-faced', '2-faces', 'abnormal', 'abolish']
|
|
|
|
Note that words from `words()` method in opinion_lexicon are sorted by file id,
|
|
not alphabetically:
|
|
|
|
>>> opinion_lexicon.words()[0:10]
|
|
['2-faced', '2-faces', 'abnormal', 'abolish', 'abominable', 'abominably',
|
|
'abominate', 'abomination', 'abort', 'aborted']
|
|
>>> sorted(opinion_lexicon.words())[0:10]
|
|
['2-faced', '2-faces', 'a+', 'abnormal', 'abolish', 'abominable', 'abominably',
|
|
'abominate', 'abomination', 'abort']
|
|
|
|
ppattach
|
|
--------
|
|
The Prepositional Phrase Attachment corpus is a corpus of
|
|
prepositional phrase attachment decisions. Each instance in the
|
|
corpus is encoded as a ``PPAttachment`` object:
|
|
|
|
>>> from nltk.corpus import ppattach
|
|
>>> ppattach.attachments('training') # doctest: +NORMALIZE_WHITESPACE
|
|
[PPAttachment(sent='0', verb='join', noun1='board',
|
|
prep='as', noun2='director', attachment='V'),
|
|
PPAttachment(sent='1', verb='is', noun1='chairman',
|
|
prep='of', noun2='N.V.', attachment='N'),
|
|
...]
|
|
>>> inst = ppattach.attachments('training')[0]
|
|
>>> (inst.sent, inst.verb, inst.noun1, inst.prep, inst.noun2)
|
|
('0', 'join', 'board', 'as', 'director')
|
|
>>> inst.attachment
|
|
'V'
|
|
|
|
product_reviews_1 and product_reviews_2
|
|
---------------------------------------
|
|
These two datasets respectively contain annotated customer reviews of 5 and 9
|
|
products from amazon.com.
|
|
|
|
>>> from nltk.corpus import product_reviews_1
|
|
>>> camera_reviews = product_reviews_1.reviews('Canon_G3.txt')
|
|
>>> review = camera_reviews[0]
|
|
>>> review.sents()[0]
|
|
['i', 'recently', 'purchased', 'the', 'canon', 'powershot', 'g3', 'and', 'am',
|
|
'extremely', 'satisfied', 'with', 'the', 'purchase', '.']
|
|
>>> review.features()
|
|
[('canon powershot g3', '+3'), ('use', '+2'), ('picture', '+2'),
|
|
('picture quality', '+1'), ('picture quality', '+1'), ('camera', '+2'),
|
|
('use', '+2'), ('feature', '+1'), ('picture quality', '+3'), ('use', '+1'),
|
|
('option', '+1')]
|
|
|
|
It is also possible to reach the same information directly from the stream:
|
|
|
|
>>> product_reviews_1.features('Canon_G3.txt')
|
|
[('canon powershot g3', '+3'), ('use', '+2'), ...]
|
|
|
|
We can compute stats for specific product features:
|
|
|
|
>>> n_reviews = len([(feat,score) for (feat,score) in product_reviews_1.features('Canon_G3.txt') if feat=='picture'])
|
|
>>> tot = sum([int(score) for (feat,score) in product_reviews_1.features('Canon_G3.txt') if feat=='picture'])
|
|
>>> mean = tot / n_reviews
|
|
>>> print(n_reviews, tot, mean)
|
|
15 24 1.6
|
|
|
|
pros_cons
|
|
---------
|
|
A list of pros/cons sentences for determining context (aspect) dependent
|
|
sentiment words, which are then applied to sentiment analysis of comparative
|
|
sentences.
|
|
|
|
>>> from nltk.corpus import pros_cons
|
|
>>> pros_cons.sents(categories='Cons')
|
|
[['East', 'batteries', '!', 'On', '-', 'off', 'switch', 'too', 'easy',
|
|
'to', 'maneuver', '.'], ['Eats', '...', 'no', ',', 'GULPS', 'batteries'],
|
|
...]
|
|
>>> pros_cons.words('IntegratedPros.txt')
|
|
['Easy', 'to', 'use', ',', 'economical', '!', ...]
|
|
|
|
semcor
|
|
------
|
|
The Brown Corpus, annotated with WordNet senses.
|
|
|
|
>>> from nltk.corpus import semcor
|
|
>>> semcor.words('brown2/tagfiles/br-n12.xml') # doctest: +ELLIPSIS
|
|
['When', 'several', 'minutes', 'had', 'passed', ...]
|
|
>>> sent = semcor.xml('brown2/tagfiles/br-n12.xml').findall('context/p/s')[0]
|
|
>>> for wordform in sent.getchildren():
|
|
... print(wordform.text, end=' ')
|
|
... for key in sorted(wordform.keys()):
|
|
... print(key + '=' + wordform.get(key), end=' ')
|
|
... print()
|
|
...
|
|
When cmd=ignore pos=WRB
|
|
several cmd=done lemma=several lexsn=5:00:00:some(a):00 pos=JJ wnsn=1
|
|
minutes cmd=done lemma=minute lexsn=1:28:00:: pos=NN wnsn=1
|
|
had cmd=done ot=notag pos=VBD
|
|
passed cmd=done lemma=pass lexsn=2:38:03:: pos=VB wnsn=4
|
|
and cmd=ignore pos=CC
|
|
Curt cmd=done lemma=person lexsn=1:03:00:: pn=person pos=NNP rdf=person wnsn=1
|
|
had cmd=done ot=notag pos=VBD
|
|
n't cmd=done lemma=n't lexsn=4:02:00:: pos=RB wnsn=0
|
|
emerged cmd=done lemma=emerge lexsn=2:30:00:: pos=VB wnsn=1
|
|
from cmd=ignore pos=IN
|
|
the cmd=ignore pos=DT
|
|
livery_stable cmd=done lemma=livery_stable lexsn=1:06:00:: pos=NN wnsn=1
|
|
,
|
|
Brenner cmd=done lemma=person lexsn=1:03:00:: pn=person pos=NNP rdf=person wnsn=1
|
|
re-entered cmd=done lemma=re-enter lexsn=2:38:00:: pos=VB wnsn=1
|
|
the cmd=ignore pos=DT
|
|
hotel cmd=done lemma=hotel lexsn=1:06:00:: pos=NN wnsn=1
|
|
and cmd=ignore pos=CC
|
|
faced cmd=done lemma=face lexsn=2:42:02:: pos=VB wnsn=4
|
|
Summers cmd=done lemma=person lexsn=1:03:00:: pn=person pos=NNP rdf=person wnsn=1
|
|
across cmd=ignore pos=IN
|
|
the cmd=ignore pos=DT
|
|
counter cmd=done lemma=counter lexsn=1:06:00:: pos=NN wnsn=1
|
|
.
|
|
|
|
senseval
|
|
--------
|
|
The Senseval 2 corpus is a word sense disambiguation corpus. Each
|
|
item in the corpus corresponds to a single ambiguous word. For each
|
|
of these words, the corpus contains a list of instances, corresponding
|
|
to occurrences of that word. Each instance provides the word; a list
|
|
of word senses that apply to the word occurrence; and the word's
|
|
context.
|
|
|
|
>>> from nltk.corpus import senseval
|
|
>>> senseval.fileids()
|
|
['hard.pos', 'interest.pos', 'line.pos', 'serve.pos']
|
|
>>> senseval.instances('hard.pos')
|
|
... # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
|
|
[SensevalInstance(word='hard-a',
|
|
position=20,
|
|
context=[('``', '``'), ('he', 'PRP'), ...('hard', 'JJ'), ...],
|
|
senses=('HARD1',)),
|
|
SensevalInstance(word='hard-a',
|
|
position=10,
|
|
context=[('clever', 'NNP'), ...('hard', 'JJ'), ('time', 'NN'), ...],
|
|
senses=('HARD1',)), ...]
|
|
|
|
The following code looks at instances of the word 'interest', and
|
|
displays their local context (2 words on each side) and word sense(s):
|
|
|
|
>>> for inst in senseval.instances('interest.pos')[:10]:
|
|
... p = inst.position
|
|
... left = ' '.join(w for (w,t) in inst.context[p-2:p])
|
|
... word = ' '.join(w for (w,t) in inst.context[p:p+1])
|
|
... right = ' '.join(w for (w,t) in inst.context[p+1:p+3])
|
|
... senses = ' '.join(inst.senses)
|
|
... print('%20s |%10s | %-15s -> %s' % (left, word, right, senses))
|
|
declines in | interest | rates . -> interest_6
|
|
indicate declining | interest | rates because -> interest_6
|
|
in short-term | interest | rates . -> interest_6
|
|
4 % | interest | in this -> interest_5
|
|
company with | interests | in the -> interest_5
|
|
, plus | interest | . -> interest_6
|
|
set the | interest | rate on -> interest_6
|
|
's own | interest | , prompted -> interest_4
|
|
principal and | interest | is the -> interest_6
|
|
increase its | interest | to 70 -> interest_5
|
|
|
|
sentence_polarity
|
|
-----------------
|
|
The Sentence Polarity dataset contains 5331 positive and 5331 negative processed
|
|
sentences.
|
|
|
|
>>> from nltk.corpus import sentence_polarity
|
|
>>> sentence_polarity.sents()
|
|
[['simplistic', ',', 'silly', 'and', 'tedious', '.'], ["it's", 'so', 'laddish',
|
|
'and', 'juvenile', ',', 'only', 'teenage', 'boys', 'could', 'possibly', 'find',
|
|
'it', 'funny', '.'], ...]
|
|
>>> sentence_polarity.categories()
|
|
['neg', 'pos']
|
|
>>> sentence_polarity.sents()[1]
|
|
["it's", 'so', 'laddish', 'and', 'juvenile', ',', 'only', 'teenage', 'boys',
|
|
'could', 'possibly', 'find', 'it', 'funny', '.']
|
|
|
|
shakespeare
|
|
-----------
|
|
The Shakespeare corpus contains a set of Shakespeare plays, formatted
|
|
as XML files. These corpora are returned as ElementTree objects:
|
|
|
|
>>> from nltk.corpus import shakespeare
|
|
>>> from xml.etree import ElementTree
|
|
>>> shakespeare.fileids() # doctest: +ELLIPSIS
|
|
['a_and_c.xml', 'dream.xml', 'hamlet.xml', 'j_caesar.xml', ...]
|
|
>>> play = shakespeare.xml('dream.xml')
|
|
>>> print(play) # doctest: +ELLIPSIS
|
|
<Element 'PLAY' at ...>
|
|
>>> print('%s: %s' % (play[0].tag, play[0].text))
|
|
TITLE: A Midsummer Night's Dream
|
|
>>> personae = [persona.text for persona in
|
|
... play.findall('PERSONAE/PERSONA')]
|
|
>>> print(personae) # doctest: +ELLIPSIS
|
|
['THESEUS, Duke of Athens.', 'EGEUS, father to Hermia.', ...]
|
|
>>> # Find and print speakers not listed as personae
|
|
>>> names = [persona.split(',')[0] for persona in personae]
|
|
>>> speakers = set(speaker.text for speaker in
|
|
... play.findall('*/*/*/SPEAKER'))
|
|
>>> print(sorted(speakers.difference(names))) # doctest: +NORMALIZE_WHITESPACE
|
|
['ALL', 'COBWEB', 'DEMETRIUS', 'Fairy', 'HERNIA', 'LYSANDER',
|
|
'Lion', 'MOTH', 'MUSTARDSEED', 'Moonshine', 'PEASEBLOSSOM',
|
|
'Prologue', 'Pyramus', 'Thisbe', 'Wall']
|
|
|
|
subjectivity
|
|
-----------
|
|
The Subjectivity Dataset contains 5000 subjective and 5000 objective processed
|
|
sentences.
|
|
|
|
>>> from nltk.corpus import subjectivity
|
|
>>> subjectivity.categories()
|
|
['obj', 'subj']
|
|
>>> subjectivity.sents()[23]
|
|
['television', 'made', 'him', 'famous', ',', 'but', 'his', 'biggest', 'hits',
|
|
'happened', 'off', 'screen', '.']
|
|
>>> subjectivity.words(categories='subj')
|
|
['smart', 'and', 'alert', ',', 'thirteen', ...]
|
|
|
|
toolbox
|
|
-------
|
|
The Toolbox corpus distributed with NLTK contains a sample lexicon and
|
|
several sample texts from the Rotokas language. The Toolbox corpus
|
|
reader returns Toolbox files as XML ElementTree objects. The
|
|
following example loads the Rotokas dictionary, and figures out the
|
|
distribution of part-of-speech tags for reduplicated words.
|
|
|
|
.. doctest: +SKIP
|
|
|
|
>>> from nltk.corpus import toolbox
|
|
>>> from nltk.probability import FreqDist
|
|
>>> from xml.etree import ElementTree
|
|
>>> import re
|
|
>>> rotokas = toolbox.xml('rotokas.dic')
|
|
>>> redup_pos_freqdist = FreqDist()
|
|
>>> # Note: we skip over the first record, which is actually
|
|
>>> # the header.
|
|
>>> for record in rotokas[1:]:
|
|
... lexeme = record.find('lx').text
|
|
... if re.match(r'(.*)\1$', lexeme):
|
|
... redup_pos_freqdist[record.find('ps').text] += 1
|
|
>>> for item, count in redup_pos_freqdist.most_common():
|
|
... print(item, count)
|
|
V 41
|
|
N 14
|
|
??? 4
|
|
|
|
This example displays some records from a Rotokas text:
|
|
|
|
.. doctest: +SKIP
|
|
|
|
>>> river = toolbox.xml('rotokas/river.txt', key='ref')
|
|
>>> for record in river.findall('record')[:3]:
|
|
... for piece in record:
|
|
... if len(piece.text) > 60:
|
|
... print('%-6s %s...' % (piece.tag, piece.text[:57]))
|
|
... else:
|
|
... print('%-6s %s' % (piece.tag, piece.text))
|
|
ref Paragraph 1
|
|
t ``Viapau oisio ra ovaupasi ...
|
|
m viapau oisio ra ovau -pa -si ...
|
|
g NEG this way/like this and forget -PROG -2/3.DL...
|
|
p NEG ??? CONJ V.I -SUFF.V.3 -SUFF.V...
|
|
f ``No ken lus tingting wanema samting papa i bin tok,'' Na...
|
|
fe ``Don't forget what Dad said,'' yelled Naomi.
|
|
ref 2
|
|
t Osa Ira ora Reviti viapau uvupasiva.
|
|
m osa Ira ora Reviti viapau uvu -pa -si ...
|
|
g as/like name and name NEG hear/smell -PROG -2/3...
|
|
p CONJ N.PN CONJ N.PN NEG V.T -SUFF.V.3 -SUF...
|
|
f Tasol Ila na David no bin harim toktok.
|
|
fe But Ila and David took no notice.
|
|
ref 3
|
|
t Ikaupaoro rokosiva ...
|
|
m ikau -pa -oro roko -si -va ...
|
|
g run/hurry -PROG -SIM go down -2/3.DL.M -RP ...
|
|
p V.T -SUFF.V.3 -SUFF.V.4 ADV -SUFF.V.4 -SUFF.VT....
|
|
f Tupela i bin hariap i go long wara .
|
|
fe They raced to the river.
|
|
|
|
timit
|
|
-----
|
|
The NLTK data package includes a fragment of the TIMIT
|
|
Acoustic-Phonetic Continuous Speech Corpus. This corpus is broken
|
|
down into small speech samples, each of which is available as a wave
|
|
file, a phonetic transcription, and a tokenized word list.
|
|
|
|
>>> from nltk.corpus import timit
|
|
>>> print(timit.utteranceids()) # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
|
|
['dr1-fvmh0/sa1', 'dr1-fvmh0/sa2', 'dr1-fvmh0/si1466',
|
|
'dr1-fvmh0/si2096', 'dr1-fvmh0/si836', 'dr1-fvmh0/sx116',
|
|
'dr1-fvmh0/sx206', 'dr1-fvmh0/sx26', 'dr1-fvmh0/sx296', ...]
|
|
|
|
>>> item = timit.utteranceids()[5]
|
|
>>> print(timit.phones(item)) # doctest: +NORMALIZE_WHITESPACE
|
|
['h#', 'k', 'l', 'ae', 's', 'pcl', 'p', 'dh', 'ax',
|
|
's', 'kcl', 'k', 'r', 'ux', 'ix', 'nx', 'y', 'ax',
|
|
'l', 'eh', 'f', 'tcl', 't', 'hh', 'ae', 'n', 'dcl',
|
|
'd', 'h#']
|
|
>>> print(timit.words(item))
|
|
['clasp', 'the', 'screw', 'in', 'your', 'left', 'hand']
|
|
>>> timit.play(item) # doctest: +SKIP
|
|
|
|
The corpus reader can combine the word segmentation information with
|
|
the phonemes to produce a single tree structure:
|
|
|
|
>>> for tree in timit.phone_trees(item):
|
|
... print(tree)
|
|
(S
|
|
h#
|
|
(clasp k l ae s pcl p)
|
|
(the dh ax)
|
|
(screw s kcl k r ux)
|
|
(in ix nx)
|
|
(your y ax)
|
|
(left l eh f tcl t)
|
|
(hand hh ae n dcl d)
|
|
h#)
|
|
|
|
The start time and stop time of each phoneme, word, and sentence are
|
|
also available:
|
|
|
|
>>> print(timit.phone_times(item)) # doctest: +ELLIPSIS
|
|
[('h#', 0, 2190), ('k', 2190, 3430), ('l', 3430, 4326), ...]
|
|
>>> print(timit.word_times(item)) # doctest: +ELLIPSIS
|
|
[('clasp', 2190, 8804), ('the', 8804, 9734), ...]
|
|
>>> print(timit.sent_times(item))
|
|
[('Clasp the screw in your left hand.', 0, 32154)]
|
|
|
|
We can use these times to play selected pieces of a speech sample:
|
|
|
|
>>> timit.play(item, 2190, 8804) # 'clasp' # doctest: +SKIP
|
|
|
|
The corpus reader can also be queried for information about the
|
|
speaker and sentence identifier for a given speech sample:
|
|
|
|
>>> print(timit.spkrid(item))
|
|
dr1-fvmh0
|
|
>>> print(timit.sentid(item))
|
|
sx116
|
|
>>> print(timit.spkrinfo(timit.spkrid(item))) # doctest: +NORMALIZE_WHITESPACE
|
|
SpeakerInfo(id='VMH0',
|
|
sex='F',
|
|
dr='1',
|
|
use='TRN',
|
|
recdate='03/11/86',
|
|
birthdate='01/08/60',
|
|
ht='5\'05"',
|
|
race='WHT',
|
|
edu='BS',
|
|
comments='BEST NEW ENGLAND ACCENT SO FAR')
|
|
|
|
>>> # List the speech samples from the same speaker:
|
|
>>> timit.utteranceids(spkrid=timit.spkrid(item)) # doctest: +ELLIPSIS
|
|
['dr1-fvmh0/sa1', 'dr1-fvmh0/sa2', 'dr1-fvmh0/si1466', ...]
|
|
|
|
twitter_samples
|
|
---------------
|
|
|
|
Twitter is well-known microblog service that allows public data to be
|
|
collected via APIs. NLTK's twitter corpus currently contains a sample of 20k Tweets
|
|
retrieved from the Twitter Streaming API.
|
|
|
|
>>> from nltk.corpus import twitter_samples
|
|
>>> twitter_samples.fileids()
|
|
['negative_tweets.json', 'positive_tweets.json', 'tweets.20150430-223406.json']
|
|
|
|
We follow standard practice in storing full Tweets as line-separated
|
|
JSON. These data structures can be accessed via `tweets.docs()`. However, in general it
|
|
is more practical to focus just on the text field of the Tweets, which
|
|
are accessed via the `strings()` method.
|
|
|
|
>>> twitter_samples.strings('tweets.20150430-223406.json')
|
|
['RT @KirkKus: Indirect cost of the UK being in the EU is estimated to be costing Britain \xa3170 billion per year! #BetterOffOut #UKIP', ...]
|
|
|
|
The default tokenizer for Tweets is specialised for 'casual' text, and
|
|
the `tokenized()` method returns a list of lists of tokens.
|
|
|
|
>>> twitter_samples.tokenized('tweets.20150430-223406.json')
|
|
[['RT', '@KirkKus', ':', 'Indirect', 'cost', 'of', 'the', 'UK', 'being', 'in', ...],
|
|
['VIDEO', ':', 'Sturgeon', 'on', 'post-election', 'deals', 'http://t.co/BTJwrpbmOY'], ...]
|
|
|
|
rte
|
|
---
|
|
The RTE (Recognizing Textual Entailment) corpus was derived from the
|
|
RTE1, RTE2 and RTE3 datasets (dev and test data), and consists of a
|
|
list of XML-formatted 'text'/'hypothesis' pairs.
|
|
|
|
>>> from nltk.corpus import rte
|
|
>>> print(rte.fileids()) # doctest: +ELLIPSIS
|
|
['rte1_dev.xml', 'rte1_test.xml', 'rte2_dev.xml', ..., 'rte3_test.xml']
|
|
>>> rtepairs = rte.pairs(['rte2_test.xml', 'rte3_test.xml'])
|
|
>>> print(rtepairs) # doctest: +ELLIPSIS
|
|
[<RTEPair: gid=2-8>, <RTEPair: gid=2-9>, <RTEPair: gid=2-15>, ...]
|
|
|
|
In the gold standard test sets, each pair is labeled according to
|
|
whether or not the text 'entails' the hypothesis; the
|
|
entailment value is mapped to an integer 1 (True) or 0 (False).
|
|
|
|
>>> rtepairs[5]
|
|
<RTEPair: gid=2-23>
|
|
>>> rtepairs[5].text # doctest: +NORMALIZE_WHITESPACE
|
|
'His wife Strida won a seat in parliament after forging an alliance
|
|
with the main anti-Syrian coalition in the recent election.'
|
|
>>> rtepairs[5].hyp
|
|
'Strida elected to parliament.'
|
|
>>> rtepairs[5].value
|
|
1
|
|
|
|
The RTE corpus also supports an ``xml()`` method which produces ElementTrees.
|
|
|
|
>>> xmltree = rte.xml('rte3_dev.xml')
|
|
>>> xmltree # doctest: +SKIP
|
|
<Element entailment-corpus at ...>
|
|
>>> xmltree[7].findtext('t') # doctest: +NORMALIZE_WHITESPACE
|
|
"Mrs. Bush's approval ratings have remained very high, above 80%,
|
|
even as her husband's have recently dropped below 50%."
|
|
|
|
verbnet
|
|
-------
|
|
The VerbNet corpus is a lexicon that divides verbs into classes, based
|
|
on their syntax-semantics linking behavior. The basic elements in the
|
|
lexicon are verb lemmas, such as 'abandon' and 'accept', and verb
|
|
classes, which have identifiers such as 'remove-10.1' and
|
|
'admire-31.2-1'. These class identifiers consist of a representative
|
|
verb selected from the class, followed by a numerical identifier. The
|
|
list of verb lemmas, and the list of class identifiers, can be
|
|
retrieved with the following methods:
|
|
|
|
>>> from nltk.corpus import verbnet
|
|
>>> verbnet.lemmas()[20:25]
|
|
['accelerate', 'accept', 'acclaim', 'accompany', 'accrue']
|
|
>>> verbnet.classids()[:5]
|
|
['accompany-51.7', 'admire-31.2', 'admire-31.2-1', 'admit-65', 'adopt-93']
|
|
|
|
The `classids()` method may also be used to retrieve the classes that
|
|
a given lemma belongs to:
|
|
|
|
>>> verbnet.classids('accept')
|
|
['approve-77', 'characterize-29.2-1-1', 'obtain-13.5.2']
|
|
|
|
The `classids()` method may additionally be used to retrieve all classes
|
|
within verbnet if nothing is passed:
|
|
|
|
>>> verbnet.classids()
|
|
['accompany-51.7', 'admire-31.2', 'admire-31.2-1', 'admit-65', 'adopt-93', 'advise-37.9', 'advise-37.9-1', 'allow-64', 'amalgamate-22.2', 'amalgamate-22.2-1', 'amalgamate-22.2-1-1', 'amalgamate-22.2-2', 'amalgamate-22.2-2-1', 'amalgamate-22.2-3', 'amalgamate-22.2-3-1', 'amalgamate-22.2-3-1-1', 'amalgamate-22.2-3-2', 'amuse-31.1', 'animal_sounds-38', 'appeal-31.4', 'appeal-31.4-1', 'appeal-31.4-2', 'appeal-31.4-3', 'appear-48.1.1', 'appoint-29.1', 'approve-77', 'assessment-34', 'assuming_position-50', 'avoid-52', 'banish-10.2', 'battle-36.4', 'battle-36.4-1', 'begin-55.1', 'begin-55.1-1', 'being_dressed-41.3.3', 'bend-45.2', 'berry-13.7', 'bill-54.5', 'body_internal_motion-49', 'body_internal_states-40.6', 'braid-41.2.2', 'break-45.1', 'breathe-40.1.2', 'breathe-40.1.2-1', 'bring-11.3', 'bring-11.3-1', 'build-26.1', 'build-26.1-1', 'bulge-47.5.3', 'bump-18.4', 'bump-18.4-1', 'butter-9.9', 'calibratable_cos-45.6', 'calibratable_cos-45.6-1', 'calve-28', 'captain-29.8', 'captain-29.8-1', 'captain-29.8-1-1', 'care-88', 'care-88-1', 'carry-11.4', 'carry-11.4-1', 'carry-11.4-1-1', 'carve-21.2', 'carve-21.2-1', 'carve-21.2-2', 'change_bodily_state-40.8.4', 'characterize-29.2', 'characterize-29.2-1', 'characterize-29.2-1-1', 'characterize-29.2-1-2', 'chase-51.6', 'cheat-10.6', 'cheat-10.6-1', 'cheat-10.6-1-1', 'chew-39.2', 'chew-39.2-1', 'chew-39.2-2', 'chit_chat-37.6', 'clear-10.3', 'clear-10.3-1', 'cling-22.5', 'coil-9.6', 'coil-9.6-1', 'coloring-24', 'complain-37.8', 'complete-55.2', 'concealment-16', 'concealment-16-1', 'confess-37.10', 'confine-92', 'confine-92-1', 'conjecture-29.5', 'conjecture-29.5-1', 'conjecture-29.5-2', 'consider-29.9', 'consider-29.9-1', 'consider-29.9-1-1', 'consider-29.9-1-1-1', 'consider-29.9-2', 'conspire-71', 'consume-66', 'consume-66-1', 'contiguous_location-47.8', 'contiguous_location-47.8-1', 'contiguous_location-47.8-2', 'continue-55.3', 'contribute-13.2', 'contribute-13.2-1', 'contribute-13.2-1-1', 'contribute-13.2-1-1-1', 'contribute-13.2-2', 'contribute-13.2-2-1', 'convert-26.6.2', 'convert-26.6.2-1', 'cooking-45.3', 'cooperate-73', 'cooperate-73-1', 'cooperate-73-2', 'cooperate-73-3', 'cope-83', 'cope-83-1', 'cope-83-1-1', 'correlate-86', 'correspond-36.1', 'correspond-36.1-1', 'correspond-36.1-1-1', 'cost-54.2', 'crane-40.3.2', 'create-26.4', 'create-26.4-1', 'curtsey-40.3.3', 'cut-21.1', 'cut-21.1-1', 'debone-10.8', 'declare-29.4', 'declare-29.4-1', 'declare-29.4-1-1', 'declare-29.4-1-1-1', 'declare-29.4-1-1-2', 'declare-29.4-1-1-3', 'declare-29.4-2', 'dedicate-79', 'defend-85', 'destroy-44', 'devour-39.4', 'devour-39.4-1', 'devour-39.4-2', 'differ-23.4', 'dine-39.5', 'disappearance-48.2', 'disassemble-23.3', 'discover-84', 'discover-84-1', 'discover-84-1-1', 'dress-41.1.1', 'dressing_well-41.3.2', 'drive-11.5', 'drive-11.5-1', 'dub-29.3', 'dub-29.3-1', 'eat-39.1', 'eat-39.1-1', 'eat-39.1-2', 'enforce-63', 'engender-27', 'entity_specific_cos-45.5', 'entity_specific_modes_being-47.2', 'equip-13.4.2', 'equip-13.4.2-1', 'equip-13.4.2-1-1', 'escape-51.1', 'escape-51.1-1', 'escape-51.1-2', 'escape-51.1-2-1', 'exceed-90', 'exchange-13.6', 'exchange-13.6-1', 'exchange-13.6-1-1', 'exhale-40.1.3', 'exhale-40.1.3-1', 'exhale-40.1.3-2', 'exist-47.1', 'exist-47.1-1', 'exist-47.1-1-1', 'feeding-39.7', 'ferret-35.6', 'fill-9.8', 'fill-9.8-1', 'fit-54.3', 'flinch-40.5', 'floss-41.2.1', 'focus-87', 'forbid-67', 'force-59', 'force-59-1', 'free-80', 'free-80-1', 'fulfilling-13.4.1', 'fulfilling-13.4.1-1', 'fulfilling-13.4.1-2', 'funnel-9.3', 'funnel-9.3-1', 'funnel-9.3-2', 'funnel-9.3-2-1', 'future_having-13.3', 'get-13.5.1', 'get-13.5.1-1', 'give-13.1', 'give-13.1-1', 'gobble-39.3', 'gobble-39.3-1', 'gobble-39.3-2', 'gorge-39.6', 'groom-41.1.2', 'grow-26.2', 'help-72', 'help-72-1', 'herd-47.5.2', 'hiccup-40.1.1', 'hit-18.1', 'hit-18.1-1', 'hold-15.1', 'hold-15.1-1', 'hunt-35.1', 'hurt-40.8.3', 'hurt-40.8.3-1', 'hurt-40.8.3-1-1', 'hurt-40.8.3-2', 'illustrate-25.3', 'image_impression-25.1', 'indicate-78', 'indicate-78-1', 'indicate-78-1-1', 'inquire-37.1.2', 'instr_communication-37.4', 'investigate-35.4', 'judgement-33', 'keep-15.2', 'knead-26.5', 'learn-14', 'learn-14-1', 'learn-14-2', 'learn-14-2-1', 'leave-51.2', 'leave-51.2-1', 'lecture-37.11', 'lecture-37.11-1', 'lecture-37.11-1-1', 'lecture-37.11-2', 'light_emission-43.1', 'limit-76', 'linger-53.1', 'linger-53.1-1', 'lodge-46', 'long-32.2', 'long-32.2-1', 'long-32.2-2', 'manner_speaking-37.3', 'marry-36.2', 'marvel-31.3', 'marvel-31.3-1', 'marvel-31.3-2', 'marvel-31.3-3', 'marvel-31.3-4', 'marvel-31.3-5', 'marvel-31.3-6', 'marvel-31.3-7', 'marvel-31.3-8', 'marvel-31.3-9', 'masquerade-29.6', 'masquerade-29.6-1', 'masquerade-29.6-2', 'matter-91', 'meander-47.7', 'meet-36.3', 'meet-36.3-1', 'meet-36.3-2', 'mine-10.9', 'mix-22.1', 'mix-22.1-1', 'mix-22.1-1-1', 'mix-22.1-2', 'mix-22.1-2-1', 'modes_of_being_with_motion-47.3', 'murder-42.1', 'murder-42.1-1', 'neglect-75', 'neglect-75-1', 'neglect-75-1-1', 'neglect-75-2', 'nonvehicle-51.4.2', 'nonverbal_expression-40.2', 'obtain-13.5.2', 'obtain-13.5.2-1', 'occurrence-48.3', 'order-60', 'order-60-1', 'orphan-29.7', 'other_cos-45.4', 'pain-40.8.1', 'pay-68', 'peer-30.3', 'pelt-17.2', 'performance-26.7', 'performance-26.7-1', 'performance-26.7-1-1', 'performance-26.7-2', 'performance-26.7-2-1', 'pit-10.7', 'pocket-9.10', 'pocket-9.10-1', 'poison-42.2', 'poke-19', 'pour-9.5', 'preparing-26.3', 'preparing-26.3-1', 'preparing-26.3-2', 'price-54.4', 'push-12', 'push-12-1', 'push-12-1-1', 'put-9.1', 'put-9.1-1', 'put-9.1-2', 'put_direction-9.4', 'put_spatial-9.2', 'put_spatial-9.2-1', 'reach-51.8', 'reflexive_appearance-48.1.2', 'refrain-69', 'register-54.1', 'rely-70', 'remove-10.1', 'risk-94', 'risk-94-1', 'roll-51.3.1', 'rummage-35.5', 'run-51.3.2', 'rush-53.2', 'say-37.7', 'say-37.7-1', 'say-37.7-1-1', 'say-37.7-2', 'scribble-25.2', 'search-35.2', 'see-30.1', 'see-30.1-1', 'see-30.1-1-1', 'send-11.1', 'send-11.1-1', 'separate-23.1', 'separate-23.1-1', 'separate-23.1-2', 'settle-89', 'shake-22.3', 'shake-22.3-1', 'shake-22.3-1-1', 'shake-22.3-2', 'shake-22.3-2-1', 'sight-30.2', 'simple_dressing-41.3.1', 'slide-11.2', 'slide-11.2-1-1', 'smell_emission-43.3', 'snooze-40.4', 'sound_emission-43.2', 'sound_existence-47.4', 'spank-18.3', 'spatial_configuration-47.6', 'split-23.2', 'spray-9.7', 'spray-9.7-1', 'spray-9.7-1-1', 'spray-9.7-2', 'stalk-35.3', 'steal-10.5', 'stimulus_subject-30.4', 'stop-55.4', 'stop-55.4-1', 'substance_emission-43.4', 'succeed-74', 'succeed-74-1', 'succeed-74-1-1', 'succeed-74-2', 'suffocate-40.7', 'suspect-81', 'swarm-47.5.1', 'swarm-47.5.1-1', 'swarm-47.5.1-2', 'swarm-47.5.1-2-1', 'swat-18.2', 'talk-37.5', 'tape-22.4', 'tape-22.4-1', 'tell-37.2', 'throw-17.1', 'throw-17.1-1', 'throw-17.1-1-1', 'tingle-40.8.2', 'touch-20', 'touch-20-1', 'transcribe-25.4', 'transfer_mesg-37.1.1', 'transfer_mesg-37.1.1-1', 'transfer_mesg-37.1.1-1-1', 'try-61', 'turn-26.6.1', 'turn-26.6.1-1', 'urge-58', 'vehicle-51.4.1', 'vehicle-51.4.1-1', 'waltz-51.5', 'want-32.1', 'want-32.1-1', 'want-32.1-1-1', 'weather-57', 'weekend-56', 'wink-40.3.1', 'wink-40.3.1-1', 'wipe_instr-10.4.2', 'wipe_instr-10.4.2-1', 'wipe_manner-10.4.1', 'wipe_manner-10.4.1-1', 'wish-62', 'withdraw-82', 'withdraw-82-1', 'withdraw-82-2', 'withdraw-82-3']
|
|
|
|
The primary object in the lexicon is a class record, which is stored
|
|
as an ElementTree xml object. The class record for a given class
|
|
identifier is returned by the `vnclass()` method:
|
|
|
|
>>> verbnet.vnclass('remove-10.1') # doctest: +ELLIPSIS
|
|
<Element 'VNCLASS' at ...>
|
|
|
|
The `vnclass()` method also accepts "short" identifiers, such as '10.1':
|
|
|
|
>>> verbnet.vnclass('10.1') # doctest: +ELLIPSIS
|
|
<Element 'VNCLASS' at ...>
|
|
|
|
See the Verbnet documentation, or the Verbnet files, for information
|
|
about the structure of this xml. As an example, we can retrieve a
|
|
list of thematic roles for a given Verbnet class:
|
|
|
|
>>> vn_31_2 = verbnet.vnclass('admire-31.2')
|
|
>>> for themrole in vn_31_2.findall('THEMROLES/THEMROLE'):
|
|
... print(themrole.attrib['type'], end=' ')
|
|
... for selrestr in themrole.findall('SELRESTRS/SELRESTR'):
|
|
... print('[%(Value)s%(type)s]' % selrestr.attrib, end=' ')
|
|
... print()
|
|
Theme
|
|
Experiencer [+animate]
|
|
Predicate
|
|
|
|
The Verbnet corpus also provides a variety of pretty printing
|
|
functions that can be used to display the xml contents in a more
|
|
concise form. The simplest such method is `pprint()`:
|
|
|
|
>>> print(verbnet.pprint('57'))
|
|
weather-57
|
|
Subclasses: (none)
|
|
Members: blow clear drizzle fog freeze gust hail howl lightning mist
|
|
mizzle pelt pour precipitate rain roar shower sleet snow spit spot
|
|
sprinkle storm swelter teem thaw thunder
|
|
Thematic roles:
|
|
* Theme[+concrete +force]
|
|
Frames:
|
|
Intransitive (Expletive Subject)
|
|
Example: It's raining.
|
|
Syntax: LEX[it] LEX[[+be]] VERB
|
|
Semantics:
|
|
* weather(during(E), Weather_type, ?Theme)
|
|
NP (Expletive Subject, Theme Object)
|
|
Example: It's raining cats and dogs.
|
|
Syntax: LEX[it] LEX[[+be]] VERB NP[Theme]
|
|
Semantics:
|
|
* weather(during(E), Weather_type, Theme)
|
|
PP (Expletive Subject, Theme-PP)
|
|
Example: It was pelting with rain.
|
|
Syntax: LEX[it[+be]] VERB PREP[with] NP[Theme]
|
|
Semantics:
|
|
* weather(during(E), Weather_type, Theme)
|
|
|
|
Verbnet gives us frames that link the syntax and semantics using an example.
|
|
These frames are part of the corpus and we can use `frames()` to get a frame
|
|
for a given verbnet class.
|
|
|
|
>>> frame = verbnet.frames('57')
|
|
>>> frame == [{'semantics': [{'arguments': [{'value': 'during(E)', 'type': 'Event'}, {'value': 'Weather_type', 'type': 'VerbSpecific'}, {'value': '?Theme', 'type': 'ThemRole'}], 'predicate_value': 'weather'}], 'example': "It's raining.", 'syntax': [{'pos_tag': 'LEX', 'modifiers': {'value': 'it', 'synrestrs': [], 'selrestrs': []}}, {'pos_tag': 'LEX', 'modifiers': {'value': '[+be]', 'synrestrs': [], 'selrestrs': []}}, {'pos_tag': 'VERB', 'modifiers': {'value': '', 'synrestrs': [], 'selrestrs': []}}], 'description': {'primary': 'Intransitive', 'secondary': 'Expletive Subject'}}, {'semantics': [{'arguments': [{'value': 'during(E)', 'type': 'Event'}, {'value': 'Weather_type', 'type': 'VerbSpecific'}, {'value': 'Theme', 'type': 'ThemRole'}], 'predicate_value': 'weather'}], 'example': "It's raining cats and dogs.", 'syntax': [{'pos_tag': 'LEX', 'modifiers': {'value': 'it', 'synrestrs': [], 'selrestrs': []}}, {'pos_tag': 'LEX', 'modifiers': {'value': '[+be]', 'synrestrs': [], 'selrestrs': []}}, {'pos_tag': 'VERB', 'modifiers': {'value': '', 'synrestrs': [], 'selrestrs': []}}, {'pos_tag': 'NP', 'modifiers': {'value': 'Theme', 'synrestrs': [], 'selrestrs': []}}], 'description': {'primary': 'NP', 'secondary': 'Expletive Subject, Theme Object'}}, {'semantics': [{'arguments': [{'value': 'during(E)', 'type': 'Event'}, {'value': 'Weather_type', 'type': 'VerbSpecific'}, {'value': 'Theme', 'type': 'ThemRole'}], 'predicate_value': 'weather'}], 'example': 'It was pelting with rain.', 'syntax': [{'pos_tag': 'LEX', 'modifiers': {'value': 'it[+be]', 'synrestrs': [], 'selrestrs': []}}, {'pos_tag': 'VERB', 'modifiers': {'value': '', 'synrestrs': [], 'selrestrs': []}}, {'pos_tag': 'PREP', 'modifiers': {'value': 'with', 'synrestrs': [], 'selrestrs': []}}, {'pos_tag': 'NP', 'modifiers': {'value': 'Theme', 'synrestrs': [], 'selrestrs': []}}], 'description': {'primary': 'PP', 'secondary': 'Expletive Subject, Theme-PP'}}]
|
|
True
|
|
|
|
Verbnet corpus lets us access thematic roles individually using `themroles()`.
|
|
|
|
>>> themroles = verbnet.themroles('57')
|
|
>>> themroles == [{'modifiers': [{'type': 'concrete', 'value': '+'}, {'type': 'force', 'value': '+'}], 'type': 'Theme'}]
|
|
True
|
|
|
|
Verbnet classes may also have subclasses sharing similar syntactic and semantic properties
|
|
while having differences with the superclass. The Verbnet corpus allows us to access these
|
|
subclasses using `subclasses()`.
|
|
|
|
>>> print(verbnet.subclasses('9.1')) #Testing for 9.1 since '57' does not have subclasses
|
|
['put-9.1-1', 'put-9.1-2']
|
|
|
|
|
|
nps_chat
|
|
--------
|
|
|
|
The NPS Chat Corpus, Release 1.0 consists of over 10,000 posts in age-specific
|
|
chat rooms, which have been anonymized, POS-tagged and dialogue-act tagged.
|
|
|
|
>>> print(nltk.corpus.nps_chat.words())
|
|
['now', 'im', 'left', 'with', 'this', 'gay', ...]
|
|
>>> print(nltk.corpus.nps_chat.tagged_words())
|
|
[('now', 'RB'), ('im', 'PRP'), ('left', 'VBD'), ...]
|
|
>>> print(nltk.corpus.nps_chat.tagged_posts()) # doctest: +NORMALIZE_WHITESPACE
|
|
[[('now', 'RB'), ('im', 'PRP'), ('left', 'VBD'), ('with', 'IN'),
|
|
('this', 'DT'), ('gay', 'JJ'), ('name', 'NN')], [(':P', 'UH')], ...]
|
|
|
|
We can access the XML elements corresponding to individual posts. These elements
|
|
have ``class`` and ``user`` attributes that we can access using ``p.attrib['class']``
|
|
and ``p.attrib['user']``. They also have text content, accessed using ``p.text``.
|
|
|
|
>>> print(nltk.corpus.nps_chat.xml_posts()) # doctest: +ELLIPSIS
|
|
[<Element 'Post' at 0...>, <Element 'Post' at 0...>, ...]
|
|
>>> posts = nltk.corpus.nps_chat.xml_posts()
|
|
>>> sorted(nltk.FreqDist(p.attrib['class'] for p in posts).keys())
|
|
['Accept', 'Bye', 'Clarify', 'Continuer', 'Emotion', 'Emphasis',
|
|
'Greet', 'Other', 'Reject', 'Statement', 'System', 'nAnswer',
|
|
'whQuestion', 'yAnswer', 'ynQuestion']
|
|
>>> posts[0].text
|
|
'now im left with this gay name'
|
|
|
|
In addition to the above methods for accessing tagged text, we can navigate
|
|
the XML structure directly, as follows:
|
|
|
|
>>> tokens = posts[0].findall('terminals/t')
|
|
>>> [t.attrib['pos'] + "/" + t.attrib['word'] for t in tokens]
|
|
['RB/now', 'PRP/im', 'VBD/left', 'IN/with', 'DT/this', 'JJ/gay', 'NN/name']
|
|
|
|
multext_east
|
|
------------
|
|
|
|
The Multext-East Corpus consists of POS-tagged versions of George Orwell's book
|
|
1984 in 12 languages: English, Czech, Hungarian, Macedonian, Slovenian, Serbian,
|
|
Slovak, Romanian, Estonian, Farsi, Bulgarian and Polish.
|
|
The corpus can be accessed using the usual methods for tagged corpora. The tagset
|
|
can be transformed from the Multext-East specific MSD tags to the Universal tagset
|
|
using the "tagset" parameter of all functions returning tagged parts of the corpus.
|
|
|
|
>>> print(nltk.corpus.multext_east.words("oana-en.xml"))
|
|
['It', 'was', 'a', 'bright', ...]
|
|
>>> print(nltk.corpus.multext_east.tagged_words("oana-en.xml"))
|
|
[('It', '#Pp3ns'), ('was', '#Vmis3s'), ('a', '#Di'), ...]
|
|
>>> print(nltk.corpus.multext_east.tagged_sents("oana-en.xml", "universal"))
|
|
[[('It', 'PRON'), ('was', 'VERB'), ('a', 'DET'), ...]
|
|
|
|
|
|
|
|
---------------------
|
|
Corpus Reader Classes
|
|
---------------------
|
|
|
|
NLTK's *corpus reader* classes are used to access the contents of a
|
|
diverse set of corpora. Each corpus reader class is specialized to
|
|
handle a specific corpus format. Examples include the
|
|
`PlaintextCorpusReader`, which handles corpora that consist of a set
|
|
of unannotated text files, and the `BracketParseCorpusReader`, which
|
|
handles corpora that consist of files containing
|
|
parenthesis-delineated parse trees.
|
|
|
|
Automatically Created Corpus Reader Instances
|
|
=============================================
|
|
|
|
When the `nltk.corpus` module is imported, it automatically creates a
|
|
set of corpus reader instances that can be used to access the corpora
|
|
in the NLTK data distribution. Here is a small sample of those
|
|
corpus reader instances:
|
|
|
|
>>> import nltk
|
|
>>> nltk.corpus.brown # doctest: +ELLIPSIS
|
|
<CategorizedTaggedCorpusReader ...>
|
|
>>> nltk.corpus.treebank # doctest: +ELLIPSIS
|
|
<BracketParseCorpusReader ...>
|
|
>>> nltk.corpus.names # doctest: +ELLIPSIS
|
|
<WordListCorpusReader ...>
|
|
>>> nltk.corpus.genesis # doctest: +ELLIPSIS
|
|
<PlaintextCorpusReader ...>
|
|
>>> nltk.corpus.inaugural # doctest: +ELLIPSIS
|
|
<PlaintextCorpusReader ...>
|
|
|
|
This sample illustrates that different corpus reader classes are used
|
|
to read different corpora; but that the same corpus reader class may
|
|
be used for more than one corpus (e.g., ``genesis`` and ``inaugural``).
|
|
|
|
Creating New Corpus Reader Instances
|
|
====================================
|
|
|
|
Although the `nltk.corpus` module automatically creates corpus reader
|
|
instances for the corpora in the NLTK data distribution, you may
|
|
sometimes need to create your own corpus reader. In particular, you
|
|
would need to create your own corpus reader if you want...
|
|
|
|
- To access a corpus that is not included in the NLTK data
|
|
distribution.
|
|
|
|
- To access a full copy of a corpus for which the NLTK data
|
|
distribution only provides a sample.
|
|
|
|
- To access a corpus using a customized corpus reader (e.g., with
|
|
a customized tokenizer).
|
|
|
|
To create a new corpus reader, you will first need to look up the
|
|
signature for that corpus reader's constructor. Different corpus
|
|
readers have different constructor signatures, but most of the
|
|
constructor signatures have the basic form::
|
|
|
|
SomeCorpusReader(root, files, ...options...)
|
|
|
|
Where ``root`` is an absolute path to the directory containing the
|
|
corpus data files; ``files`` is either a list of file names (relative
|
|
to ``root``) or a regexp specifying which files should be included;
|
|
and ``options`` are additional reader-specific options. For example,
|
|
we can create a customized corpus reader for the genesis corpus that
|
|
uses a different sentence tokenizer as follows:
|
|
|
|
>>> # Find the directory where the corpus lives.
|
|
>>> genesis_dir = nltk.data.find('corpora/genesis')
|
|
>>> # Create our custom sentence tokenizer.
|
|
>>> my_sent_tokenizer = nltk.RegexpTokenizer('[^.!?]+')
|
|
>>> # Create the new corpus reader object.
|
|
>>> my_genesis = nltk.corpus.PlaintextCorpusReader(
|
|
... genesis_dir, '.*\.txt', sent_tokenizer=my_sent_tokenizer)
|
|
>>> # Use the new corpus reader object.
|
|
>>> print(my_genesis.sents('english-kjv.txt')[0]) # doctest: +NORMALIZE_WHITESPACE
|
|
['In', 'the', 'beginning', 'God', 'created', 'the', 'heaven',
|
|
'and', 'the', 'earth']
|
|
|
|
If you wish to read your own plaintext corpus, which is stored in the
|
|
directory '/usr/share/some-corpus', then you can create a corpus
|
|
reader for it with::
|
|
|
|
>>> my_corpus = nltk.corpus.PlaintextCorpusReader(
|
|
... '/usr/share/some-corpus', '.*\.txt') # doctest: +SKIP
|
|
|
|
For a complete list of corpus reader subclasses, see the API
|
|
documentation for `nltk.corpus.reader`.
|
|
|
|
Corpus Types
|
|
============
|
|
|
|
Corpora vary widely in the types of content they include. This is
|
|
reflected in the fact that the base class `CorpusReader` only defines
|
|
a few general-purpose methods for listing and accessing the files that
|
|
make up a corpus. It is up to the subclasses to define *data access
|
|
methods* that provide access to the information in the corpus.
|
|
However, corpus reader subclasses should be consistent in their
|
|
definitions of these data access methods wherever possible.
|
|
|
|
At a high level, corpora can be divided into three basic types:
|
|
|
|
- A *token corpus* contains information about specific occurrences of
|
|
language use (or linguistic tokens), such as dialogues or written
|
|
texts. Examples of token corpora are collections of written text
|
|
and collections of speech.
|
|
|
|
- A *type corpus*, or *lexicon*, contains information about a coherent
|
|
set of lexical items (or linguistic types). Examples of lexicons
|
|
are dictionaries and word lists.
|
|
|
|
- A *language description corpus* contains information about a set of
|
|
non-lexical linguistic constructs, such as grammar rules.
|
|
|
|
However, many individual corpora blur the distinctions between these
|
|
types. For example, corpora that are primarily lexicons may include
|
|
token data in the form of example sentences; and corpora that are
|
|
primarily token corpora may be accompanied by one or more word lists
|
|
or other lexical data sets.
|
|
|
|
Because corpora vary so widely in their information content, we have
|
|
decided that it would not be wise to use separate corpus reader base
|
|
classes for different corpus types. Instead, we simply try to make
|
|
the corpus readers consistent wherever possible, but let them differ
|
|
where the underlying data itself differs.
|
|
|
|
Common Corpus Reader Methods
|
|
============================
|
|
|
|
As mentioned above, there are only a handful of methods that all
|
|
corpus readers are guaranteed to implement. These methods provide
|
|
access to the files that contain the corpus data. Every corpus is
|
|
assumed to consist of one or more files, all located in a common root
|
|
directory (or in subdirectories of that root directory). The absolute
|
|
path to the root directory is stored in the ``root`` property:
|
|
|
|
>>> import os
|
|
>>> str(nltk.corpus.genesis.root).replace(os.path.sep,'/') # doctest: +ELLIPSIS
|
|
'.../nltk_data/corpora/genesis'
|
|
|
|
Each file within the corpus is identified by a platform-independent
|
|
identifier, which is basically a path string that uses ``/`` as the
|
|
path separator. I.e., this identifier can be converted to a relative
|
|
path as follows:
|
|
|
|
>>> some_corpus_file_id = nltk.corpus.reuters.fileids()[0]
|
|
>>> import os.path
|
|
>>> os.path.normpath(some_corpus_file_id).replace(os.path.sep,'/')
|
|
'test/14826'
|
|
|
|
To get a list of all data files that make up a corpus, use the
|
|
``fileids()`` method. In some corpora, these files will not all contain
|
|
the same type of data; for example, for the ``nltk.corpus.timit``
|
|
corpus, ``fileids()`` will return a list including text files, word
|
|
segmentation files, phonetic transcription files, sound files, and
|
|
metadata files. For corpora with diverse file types, the ``fileids()``
|
|
method will often take one or more optional arguments, which can be
|
|
used to get a list of the files with a specific file type:
|
|
|
|
>>> nltk.corpus.timit.fileids() # doctest: +ELLIPSIS
|
|
['dr1-fvmh0/sa1.phn', 'dr1-fvmh0/sa1.txt', 'dr1-fvmh0/sa1.wav', ...]
|
|
>>> nltk.corpus.timit.fileids('phn') # doctest: +ELLIPSIS
|
|
['dr1-fvmh0/sa1.phn', 'dr1-fvmh0/sa2.phn', 'dr1-fvmh0/si1466.phn', ...]
|
|
|
|
In some corpora, the files are divided into distinct categories. For
|
|
these corpora, the ``fileids()`` method takes an optional argument,
|
|
which can be used to get a list of the files within a specific category:
|
|
|
|
>>> nltk.corpus.brown.fileids('hobbies') # doctest: +ELLIPSIS
|
|
['ce01', 'ce02', 'ce03', 'ce04', 'ce05', 'ce06', 'ce07', ...]
|
|
|
|
The ``abspath()`` method can be used to find the absolute path to a
|
|
corpus file, given its file identifier:
|
|
|
|
>>> str(nltk.corpus.brown.abspath('ce06')).replace(os.path.sep,'/') # doctest: +ELLIPSIS
|
|
'.../corpora/brown/ce06'
|
|
|
|
The ``abspaths()`` method can be used to find the absolute paths for
|
|
one corpus file, a list of corpus files, or (if no fileids are specified),
|
|
all corpus files.
|
|
|
|
This method is mainly useful as a helper method when defining corpus
|
|
data access methods, since data access methods can usually be called
|
|
with a string argument (to get a view for a specific file), with a
|
|
list argument (to get a view for a specific list of files), or with no
|
|
argument (to get a view for the whole corpus).
|
|
|
|
Data Access Methods
|
|
===================
|
|
|
|
Individual corpus reader subclasses typically extend this basic set of
|
|
file-access methods with one or more *data access methods*, which provide
|
|
easy access to the data contained in the corpus. The signatures for
|
|
data access methods often have the basic form::
|
|
|
|
corpus_reader.some_data access(fileids=None, ...options...)
|
|
|
|
Where ``fileids`` can be a single file identifier string (to get a view
|
|
for a specific file); a list of file identifier strings (to get a view
|
|
for a specific list of files); or None (to get a view for the entire
|
|
corpus). Some of the common data access methods, and their return
|
|
types, are:
|
|
|
|
- I{corpus}.words(): list of str
|
|
- I{corpus}.sents(): list of (list of str)
|
|
- I{corpus}.paras(): list of (list of (list of str))
|
|
- I{corpus}.tagged_words(): list of (str,str) tuple
|
|
- I{corpus}.tagged_sents(): list of (list of (str,str))
|
|
- I{corpus}.tagged_paras(): list of (list of (list of (str,str)))
|
|
- I{corpus}.chunked_sents(): list of (Tree w/ (str,str) leaves)
|
|
- I{corpus}.parsed_sents(): list of (Tree with str leaves)
|
|
- I{corpus}.parsed_paras(): list of (list of (Tree with str leaves))
|
|
- I{corpus}.xml(): A single xml ElementTree
|
|
- I{corpus}.raw(): str (unprocessed corpus contents)
|
|
|
|
For example, the `words()` method is supported by many different
|
|
corpora, and returns a flat list of word strings:
|
|
|
|
>>> nltk.corpus.brown.words()
|
|
['The', 'Fulton', 'County', 'Grand', 'Jury', ...]
|
|
>>> nltk.corpus.treebank.words()
|
|
['Pierre', 'Vinken', ',', '61', 'years', 'old', ...]
|
|
>>> nltk.corpus.conll2002.words()
|
|
['Sao', 'Paulo', '(', 'Brasil', ')', ',', '23', ...]
|
|
>>> nltk.corpus.genesis.words()
|
|
['In', 'the', 'beginning', 'God', 'created', ...]
|
|
|
|
On the other hand, the `tagged_words()` method is only supported by
|
|
corpora that include part-of-speech annotations:
|
|
|
|
>>> nltk.corpus.brown.tagged_words()
|
|
[('The', 'AT'), ('Fulton', 'NP-TL'), ...]
|
|
>>> nltk.corpus.treebank.tagged_words()
|
|
[('Pierre', 'NNP'), ('Vinken', 'NNP'), ...]
|
|
>>> nltk.corpus.conll2002.tagged_words()
|
|
[('Sao', 'NC'), ('Paulo', 'VMI'), ('(', 'Fpa'), ...]
|
|
>>> nltk.corpus.genesis.tagged_words()
|
|
Traceback (most recent call last):
|
|
...
|
|
AttributeError: 'PlaintextCorpusReader' object has no attribute 'tagged_words'
|
|
|
|
Although most corpus readers use file identifiers to index their
|
|
content, some corpora use different identifiers instead. For example,
|
|
the data access methods for the ``timit`` corpus uses *utterance
|
|
identifiers* to select which corpus items should be returned:
|
|
|
|
>>> nltk.corpus.timit.utteranceids() # doctest: +ELLIPSIS
|
|
['dr1-fvmh0/sa1', 'dr1-fvmh0/sa2', 'dr1-fvmh0/si1466', ...]
|
|
>>> nltk.corpus.timit.words('dr1-fvmh0/sa2')
|
|
["don't", 'ask', 'me', 'to', 'carry', 'an', 'oily', 'rag', 'like', 'that']
|
|
|
|
Attempting to call ``timit``\ 's data access methods with a file
|
|
identifier will result in an exception:
|
|
|
|
>>> nltk.corpus.timit.fileids() # doctest: +ELLIPSIS
|
|
['dr1-fvmh0/sa1.phn', 'dr1-fvmh0/sa1.txt', 'dr1-fvmh0/sa1.wav', ...]
|
|
>>> nltk.corpus.timit.words('dr1-fvmh0/sa1.txt') # doctest: +SKIP
|
|
Traceback (most recent call last):
|
|
...
|
|
IOError: No such file or directory: '.../dr1-fvmh0/sa1.txt.wrd'
|
|
|
|
As another example, the ``propbank`` corpus defines the ``roleset()``
|
|
method, which expects a roleset identifier, not a file identifier:
|
|
|
|
>>> roleset = nltk.corpus.propbank.roleset('eat.01')
|
|
>>> from xml.etree import ElementTree as ET
|
|
>>> print(ET.tostring(roleset).decode('utf8')) # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
|
|
<roleset id="eat.01" name="consume" vncls="39.1">
|
|
<roles>
|
|
<role descr="consumer, eater" n="0">...</role>...
|
|
</roles>...
|
|
</roleset>...
|
|
|
|
Stream Backed Corpus Views
|
|
==========================
|
|
An important feature of NLTK's corpus readers is that many of them
|
|
access the underlying data files using "corpus views." A *corpus
|
|
view* is an object that acts like a simple data structure (such as a
|
|
list), but does not store the data elements in memory; instead, data
|
|
elements are read from the underlying data files on an as-needed
|
|
basis.
|
|
|
|
By only loading items from the file on an as-needed basis, corpus
|
|
views maintain both memory efficiency and responsiveness. The memory
|
|
efficiency of corpus readers is important because some corpora contain
|
|
very large amounts of data, and storing the entire data set in memory
|
|
could overwhelm many machines. The responsiveness is important when
|
|
experimenting with corpora in interactive sessions and in in-class
|
|
demonstrations.
|
|
|
|
The most common corpus view is the `StreamBackedCorpusView`, which
|
|
acts as a read-only list of tokens. Two additional corpus view
|
|
classes, `ConcatenatedCorpusView` and `LazySubsequence`, make it
|
|
possible to create concatenations and take slices of
|
|
`StreamBackedCorpusView` objects without actually storing the
|
|
resulting list-like object's elements in memory.
|
|
|
|
In the future, we may add additional corpus views that act like other
|
|
basic data structures, such as dictionaries.
|
|
|
|
Writing New Corpus Readers
|
|
==========================
|
|
|
|
In order to add support for new corpus formats, it is necessary to
|
|
define new corpus reader classes. For many corpus formats, writing
|
|
new corpus readers is relatively straight-forward. In this section,
|
|
we'll describe what's involved in creating a new corpus reader. If
|
|
you do create a new corpus reader, we encourage you to contribute it
|
|
back to the NLTK project.
|
|
|
|
Don't Reinvent the Wheel
|
|
------------------------
|
|
Before you start writing a new corpus reader, you should check to be
|
|
sure that the desired format can't be read using an existing corpus
|
|
reader with appropriate constructor arguments. For example, although
|
|
the `TaggedCorpusReader` assumes that words and tags are separated by
|
|
``/`` characters by default, an alternative tag-separation character
|
|
can be specified via the ``sep`` constructor argument. You should
|
|
also check whether the new corpus format can be handled by subclassing
|
|
an existing corpus reader, and tweaking a few methods or variables.
|
|
|
|
Design
|
|
------
|
|
If you decide to write a new corpus reader from scratch, then you
|
|
should first decide which data access methods you want the reader to
|
|
provide, and what their signatures should be. You should look at
|
|
existing corpus readers that process corpora with similar data
|
|
contents, and try to be consistent with those corpus readers whenever
|
|
possible.
|
|
|
|
You should also consider what sets of identifiers are appropriate for
|
|
the corpus format. Where it's practical, file identifiers should be
|
|
used. However, for some corpora, it may make sense to use additional
|
|
sets of identifiers. Each set of identifiers should have a distinct
|
|
name (e.g., fileids, utteranceids, rolesets); and you should be consistent
|
|
in using that name to refer to that identifier. Do not use parameter
|
|
names like ``id``, which leave it unclear what type of identifier is
|
|
required.
|
|
|
|
Once you've decided what data access methods and identifiers are
|
|
appropriate for your corpus, you should decide if there are any
|
|
customizable parameters that you'd like the corpus reader to handle.
|
|
These parameters make it possible to use a single corpus reader to
|
|
handle a wider variety of corpora. The ``sep`` argument for
|
|
`TaggedCorpusReader`, mentioned above, is an example of a customizable
|
|
corpus reader parameter.
|
|
|
|
Implementation
|
|
--------------
|
|
|
|
Constructor
|
|
~~~~~~~~~~~
|
|
If your corpus reader implements any customizable parameters, then
|
|
you'll need to override the constructor. Typically, the new
|
|
constructor will first call its base class's constructor, and then
|
|
store the customizable parameters. For example, the
|
|
`ConllChunkCorpusReader`\ 's constructor is defined as follows:
|
|
|
|
def __init__(self, root, fileids, chunk_types, encoding='utf8',
|
|
tagset=None, separator=None):
|
|
ConllCorpusReader.__init__(
|
|
self, root, fileids, ('words', 'pos', 'chunk'),
|
|
chunk_types=chunk_types, encoding=encoding,
|
|
tagset=tagset, separator=separator)
|
|
|
|
If your corpus reader does not implement any customization parameters,
|
|
then you can often just inherit the base class's constructor.
|
|
|
|
Data Access Methods
|
|
~~~~~~~~~~~~~~~~~~~
|
|
|
|
The most common type of data access method takes an argument
|
|
identifying which files to access, and returns a view covering those
|
|
files. This argument may be a single file identifier string (to get a
|
|
view for a specific file); a list of file identifier strings (to get a
|
|
view for a specific list of files); or None (to get a view for the
|
|
entire corpus). The method's implementation converts this argument to
|
|
a list of path names using the `abspaths()` method, which handles all
|
|
three value types (string, list, and None):
|
|
|
|
>>> print(str(nltk.corpus.brown.abspaths()).replace('\\\\','/')) # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
|
|
[FileSystemPathPointer('.../corpora/brown/ca01'),
|
|
FileSystemPathPointer('.../corpora/brown/ca02'), ...]
|
|
>>> print(str(nltk.corpus.brown.abspaths('ce06')).replace('\\\\','/')) # doctest: +ELLIPSIS
|
|
[FileSystemPathPointer('.../corpora/brown/ce06')]
|
|
>>> print(str(nltk.corpus.brown.abspaths(['ce06', 'ce07'])).replace('\\\\','/')) # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
|
|
[FileSystemPathPointer('.../corpora/brown/ce06'),
|
|
FileSystemPathPointer('.../corpora/brown/ce07')]
|
|
|
|
An example of this type of method is the `words()` method, defined by
|
|
the `PlaintextCorpusReader` as follows:
|
|
|
|
>>> def words(self, fileids=None):
|
|
... return concat([self.CorpusView(fileid, self._read_word_block)
|
|
... for fileid in self.abspaths(fileids)])
|
|
|
|
This method first uses `abspaths()` to convert ``fileids`` to a list of
|
|
absolute paths. It then creates a corpus view for each file, using
|
|
the `PlaintextCorpusReader._read_word_block()` method to read elements
|
|
from the data file (see the discussion of corpus views below).
|
|
Finally, it combines these corpus views using the
|
|
`nltk.corpus.reader.util.concat()` function.
|
|
|
|
When writing a corpus reader for a corpus that is never expected to be
|
|
very large, it can sometimes be appropriate to read the files
|
|
directly, rather than using a corpus view. For example, the
|
|
`WordListCorpusView` class defines its `words()` method as follows:
|
|
|
|
>>> def words(self, fileids=None):
|
|
... return concat([[w for w in open(fileid).read().split('\n') if w]
|
|
... for fileid in self.abspaths(fileids)])
|
|
|
|
(This is usually more appropriate for lexicons than for token corpora.)
|
|
|
|
If the type of data returned by a data access method is one for which
|
|
NLTK has a conventional representation (e.g., words, tagged words, and
|
|
parse trees), then you should use that representation. Otherwise, you
|
|
may find it necessary to define your own representation. For data
|
|
structures that are relatively corpus-specific, it's usually best to
|
|
define new classes for these elements. For example, the ``propbank``
|
|
corpus defines the `PropbankInstance` class to store the semantic role
|
|
labeling instances described by the corpus; and the ``ppattach``
|
|
corpus defines the `PPAttachment` class to store the prepositional
|
|
attachment instances described by the corpus.
|
|
|
|
Corpus Views
|
|
~~~~~~~~~~~~
|
|
.. (Much of the content for this section is taken from the
|
|
StreamBackedCorpusView docstring.)
|
|
|
|
The heart of a `StreamBackedCorpusView` is its *block reader*
|
|
function, which reads zero or more tokens from a stream, and returns
|
|
them as a list. A very simple example of a block reader is:
|
|
|
|
>>> def simple_block_reader(stream):
|
|
... return stream.readline().split()
|
|
|
|
This simple block reader reads a single line at a time, and returns a
|
|
single token (consisting of a string) for each whitespace-separated
|
|
substring on the line. A `StreamBackedCorpusView` built from this
|
|
block reader will act like a read-only list of all the
|
|
whitespace-separated tokens in an underlying file.
|
|
|
|
When deciding how to define the block reader for a given corpus,
|
|
careful consideration should be given to the size of blocks handled by
|
|
the block reader. Smaller block sizes will increase the memory
|
|
requirements of the corpus view's internal data structures (by 2
|
|
integers per block). On the other hand, larger block sizes may
|
|
decrease performance for random access to the corpus. (But note that
|
|
larger block sizes will *not* decrease performance for iteration.)
|
|
|
|
Internally, the `StreamBackedCorpusView` class maintains a partial
|
|
mapping from token index to file position, with one entry per block.
|
|
When a token with a given index *i* is requested, the corpus view
|
|
constructs it as follows:
|
|
|
|
1. First, it searches the toknum/filepos mapping for the token index
|
|
closest to (but less than or equal to) *i*.
|
|
|
|
2. Then, starting at the file position corresponding to that index, it
|
|
reads one block at a time using the block reader until it reaches
|
|
the requested token.
|
|
|
|
The toknum/filepos mapping is created lazily: it is initially empty,
|
|
but every time a new block is read, the block's initial token is added
|
|
to the mapping. (Thus, the toknum/filepos map has one entry per
|
|
block.)
|
|
|
|
You can create your own corpus view in one of two ways:
|
|
|
|
1. Call the `StreamBackedCorpusView` constructor, and provide your
|
|
block reader function via the ``block_reader`` argument.
|
|
|
|
2. Subclass `StreamBackedCorpusView`, and override the
|
|
`read_block()` method.
|
|
|
|
The first option is usually easier, but the second option can allow
|
|
you to write a single `read_block` method whose behavior can be
|
|
customized by different parameters to the subclass's constructor. For
|
|
an example of this design pattern, see the `TaggedCorpusView` class,
|
|
which is used by `TaggedCorpusView`.
|
|
|
|
----------------
|
|
Regression Tests
|
|
----------------
|
|
|
|
The following helper functions are used to create and then delete
|
|
testing corpora that are stored in temporary directories. These
|
|
testing corpora are used to make sure the readers work correctly.
|
|
|
|
>>> import tempfile, os.path, textwrap
|
|
>>> def make_testcorpus(ext='', **fileids):
|
|
... root = tempfile.mkdtemp()
|
|
... for fileid, contents in fileids.items():
|
|
... fileid += ext
|
|
... f = open(os.path.join(root, fileid), 'w')
|
|
... f.write(textwrap.dedent(contents))
|
|
... f.close()
|
|
... return root
|
|
>>> def del_testcorpus(root):
|
|
... for fileid in os.listdir(root):
|
|
... os.remove(os.path.join(root, fileid))
|
|
... os.rmdir(root)
|
|
|
|
Plaintext Corpus Reader
|
|
=======================
|
|
The plaintext corpus reader is used to access corpora that consist of
|
|
unprocessed plaintext data. It assumes that paragraph breaks are
|
|
indicated by blank lines. Sentences and words can be tokenized using
|
|
the default tokenizers, or by custom tokenizers specified as
|
|
parameters to the constructor.
|
|
|
|
>>> root = make_testcorpus(ext='.txt',
|
|
... a="""\
|
|
... This is the first sentence. Here is another
|
|
... sentence! And here's a third sentence.
|
|
...
|
|
... This is the second paragraph. Tokenization is currently
|
|
... fairly simple, so the period in Mr. gets tokenized.
|
|
... """,
|
|
... b="""This is the second file.""")
|
|
|
|
>>> from nltk.corpus.reader.plaintext import PlaintextCorpusReader
|
|
|
|
The list of documents can be specified explicitly, or implicitly (using a
|
|
regexp). The ``ext`` argument specifies a file extension.
|
|
|
|
>>> corpus = PlaintextCorpusReader(root, ['a.txt', 'b.txt'])
|
|
>>> corpus.fileids()
|
|
['a.txt', 'b.txt']
|
|
>>> corpus = PlaintextCorpusReader(root, '.*\.txt')
|
|
>>> corpus.fileids()
|
|
['a.txt', 'b.txt']
|
|
|
|
The directory containing the corpus is corpus.root:
|
|
|
|
>>> str(corpus.root) == str(root)
|
|
True
|
|
|
|
We can get a list of words, or the raw string:
|
|
|
|
>>> corpus.words()
|
|
['This', 'is', 'the', 'first', 'sentence', '.', ...]
|
|
>>> corpus.raw()[:40]
|
|
'This is the first sentence. Here is ano'
|
|
|
|
Check that reading individual documents works, and reading all documents at
|
|
once works:
|
|
|
|
>>> len(corpus.words()), [len(corpus.words(d)) for d in corpus.fileids()]
|
|
(46, [40, 6])
|
|
>>> corpus.words('a.txt')
|
|
['This', 'is', 'the', 'first', 'sentence', '.', ...]
|
|
>>> corpus.words('b.txt')
|
|
['This', 'is', 'the', 'second', 'file', '.']
|
|
>>> corpus.words()[:4], corpus.words()[-4:]
|
|
(['This', 'is', 'the', 'first'], ['the', 'second', 'file', '.'])
|
|
|
|
We're done with the test corpus:
|
|
|
|
>>> del_testcorpus(root)
|
|
|
|
Test the plaintext corpora that come with nltk:
|
|
|
|
>>> from nltk.corpus import abc, genesis, inaugural
|
|
>>> from nltk.corpus import state_union, webtext
|
|
>>> for corpus in (abc, genesis, inaugural, state_union,
|
|
... webtext):
|
|
... print(str(corpus).replace('\\\\','/'))
|
|
... print(' ', repr(corpus.fileids())[:60])
|
|
... print(' ', repr(corpus.words()[:10])[:60])
|
|
<PlaintextCorpusReader in '.../nltk_data/corpora/ab...'>
|
|
['rural.txt', 'science.txt']
|
|
['PM', 'denies', 'knowledge', 'of', 'AWB', ...
|
|
<PlaintextCorpusReader in '.../nltk_data/corpora/genesi...'>
|
|
['english-kjv.txt', 'english-web.txt', 'finnish.txt', ...
|
|
['In', 'the', 'beginning', 'God', 'created', 'the', ...
|
|
<PlaintextCorpusReader in '.../nltk_data/corpora/inaugura...'>
|
|
['1789-Washington.txt', '1793-Washington.txt', ...
|
|
['Fellow', '-', 'Citizens', 'of', 'the', 'Senate', ...
|
|
<PlaintextCorpusReader in '.../nltk_data/corpora/state_unio...'>
|
|
['1945-Truman.txt', '1946-Truman.txt', ...
|
|
['PRESIDENT', 'HARRY', 'S', '.', 'TRUMAN', "'", ...
|
|
<PlaintextCorpusReader in '.../nltk_data/corpora/webtex...'>
|
|
['firefox.txt', 'grail.txt', 'overheard.txt', ...
|
|
['Cookie', 'Manager', ':', '"', 'Don', "'", 't', ...
|
|
|
|
|
|
Tagged Corpus Reader
|
|
====================
|
|
The Tagged Corpus reader can give us words, sentences, and paragraphs,
|
|
each tagged or untagged. All of the read methods can take one item
|
|
(in which case they return the contents of that file) or a list of
|
|
documents (in which case they concatenate the contents of those files).
|
|
By default, they apply to all documents in the corpus.
|
|
|
|
>>> root = make_testcorpus(
|
|
... a="""\
|
|
... This/det is/verb the/det first/adj sentence/noun ./punc
|
|
... Here/det is/verb another/adj sentence/noun ./punc
|
|
... Note/verb that/comp you/pron can/verb use/verb \
|
|
... any/noun tag/noun set/noun
|
|
...
|
|
... This/det is/verb the/det second/adj paragraph/noun ./punc
|
|
... word/n without/adj a/det tag/noun :/: hello ./punc
|
|
... """,
|
|
... b="""\
|
|
... This/det is/verb the/det second/adj file/noun ./punc
|
|
... """)
|
|
|
|
>>> from nltk.corpus.reader.tagged import TaggedCorpusReader
|
|
>>> corpus = TaggedCorpusReader(root, list('ab'))
|
|
>>> corpus.fileids()
|
|
['a', 'b']
|
|
>>> str(corpus.root) == str(root)
|
|
True
|
|
>>> corpus.words()
|
|
['This', 'is', 'the', 'first', 'sentence', '.', ...]
|
|
>>> corpus.sents() # doctest: +ELLIPSIS
|
|
[['This', 'is', 'the', 'first', ...], ['Here', 'is', 'another'...], ...]
|
|
>>> corpus.paras() # doctest: +ELLIPSIS
|
|
[[['This', ...], ['Here', ...], ...], [['This', ...], ...], ...]
|
|
>>> corpus.tagged_words() # doctest: +ELLIPSIS
|
|
[('This', 'DET'), ('is', 'VERB'), ('the', 'DET'), ...]
|
|
>>> corpus.tagged_sents() # doctest: +ELLIPSIS
|
|
[[('This', 'DET'), ('is', 'VERB'), ...], [('Here', 'DET'), ...], ...]
|
|
>>> corpus.tagged_paras() # doctest: +ELLIPSIS
|
|
[[[('This', 'DET'), ...], ...], [[('This', 'DET'), ...], ...], ...]
|
|
>>> corpus.raw()[:40]
|
|
'This/det is/verb the/det first/adj sente'
|
|
>>> len(corpus.words()), [len(corpus.words(d)) for d in corpus.fileids()]
|
|
(38, [32, 6])
|
|
>>> len(corpus.sents()), [len(corpus.sents(d)) for d in corpus.fileids()]
|
|
(6, [5, 1])
|
|
>>> len(corpus.paras()), [len(corpus.paras(d)) for d in corpus.fileids()]
|
|
(3, [2, 1])
|
|
>>> print(corpus.words('a'))
|
|
['This', 'is', 'the', 'first', 'sentence', '.', ...]
|
|
>>> print(corpus.words('b'))
|
|
['This', 'is', 'the', 'second', 'file', '.']
|
|
>>> del_testcorpus(root)
|
|
|
|
The Brown Corpus uses the tagged corpus reader:
|
|
|
|
>>> from nltk.corpus import brown
|
|
>>> brown.fileids() # doctest: +ELLIPSIS
|
|
['ca01', 'ca02', 'ca03', 'ca04', 'ca05', 'ca06', 'ca07', ...]
|
|
>>> brown.categories() # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
|
|
['adventure', 'belles_lettres', 'editorial', 'fiction', 'government', 'hobbies', 'humor',
|
|
'learned', 'lore', 'mystery', 'news', 'religion', 'reviews', 'romance', 'science_fiction']
|
|
>>> print(repr(brown.root).replace('\\\\','/')) # doctest: +ELLIPSIS
|
|
FileSystemPathPointer('.../corpora/brown')
|
|
>>> brown.words()
|
|
['The', 'Fulton', 'County', 'Grand', 'Jury', ...]
|
|
>>> brown.sents() # doctest: +ELLIPSIS
|
|
[['The', 'Fulton', 'County', 'Grand', ...], ...]
|
|
>>> brown.paras() # doctest: +ELLIPSIS
|
|
[[['The', 'Fulton', 'County', ...]], [['The', 'jury', ...]], ...]
|
|
>>> brown.tagged_words() # doctest: +ELLIPSIS
|
|
[('The', 'AT'), ('Fulton', 'NP-TL'), ...]
|
|
>>> brown.tagged_sents() # doctest: +ELLIPSIS
|
|
[[('The', 'AT'), ('Fulton', 'NP-TL'), ('County', 'NN-TL'), ...], ...]
|
|
>>> brown.tagged_paras() # doctest: +ELLIPSIS
|
|
[[[('The', 'AT'), ...]], [[('The', 'AT'), ...]], ...]
|
|
|
|
Verbnet Corpus Reader
|
|
=====================
|
|
|
|
Make sure we're picking up the right number of elements:
|
|
|
|
>>> from nltk.corpus import verbnet
|
|
>>> len(verbnet.lemmas())
|
|
3621
|
|
>>> len(verbnet.wordnetids())
|
|
4953
|
|
>>> len(verbnet.classids())
|
|
429
|
|
|
|
Selecting classids based on various selectors:
|
|
|
|
>>> verbnet.classids(lemma='take') # doctest: +NORMALIZE_WHITESPACE
|
|
['bring-11.3', 'characterize-29.2', 'convert-26.6.2', 'cost-54.2',
|
|
'fit-54.3', 'performance-26.7-2', 'steal-10.5']
|
|
>>> verbnet.classids(wordnetid='lead%2:38:01')
|
|
['accompany-51.7']
|
|
>>> verbnet.classids(fileid='approve-77.xml')
|
|
['approve-77']
|
|
>>> verbnet.classids(classid='admire-31.2') # subclasses
|
|
['admire-31.2-1']
|
|
|
|
vnclass() accepts filenames, long ids, and short ids:
|
|
|
|
>>> a = ElementTree.tostring(verbnet.vnclass('admire-31.2.xml'))
|
|
>>> b = ElementTree.tostring(verbnet.vnclass('admire-31.2'))
|
|
>>> c = ElementTree.tostring(verbnet.vnclass('31.2'))
|
|
>>> a == b == c
|
|
True
|
|
|
|
fileids() can be used to get files based on verbnet class ids:
|
|
|
|
>>> verbnet.fileids('admire-31.2')
|
|
['admire-31.2.xml']
|
|
>>> verbnet.fileids(['admire-31.2', 'obtain-13.5.2'])
|
|
['admire-31.2.xml', 'obtain-13.5.2.xml']
|
|
>>> verbnet.fileids('badidentifier')
|
|
Traceback (most recent call last):
|
|
. . .
|
|
ValueError: vnclass identifier 'badidentifier' not found
|
|
|
|
longid() and shortid() can be used to convert identifiers:
|
|
|
|
>>> verbnet.longid('31.2')
|
|
'admire-31.2'
|
|
>>> verbnet.longid('admire-31.2')
|
|
'admire-31.2'
|
|
>>> verbnet.shortid('31.2')
|
|
'31.2'
|
|
>>> verbnet.shortid('admire-31.2')
|
|
'31.2'
|
|
>>> verbnet.longid('badidentifier')
|
|
Traceback (most recent call last):
|
|
. . .
|
|
ValueError: vnclass identifier 'badidentifier' not found
|
|
>>> verbnet.shortid('badidentifier')
|
|
Traceback (most recent call last):
|
|
. . .
|
|
ValueError: vnclass identifier 'badidentifier' not found
|
|
|
|
Corpus View Regression Tests
|
|
============================
|
|
|
|
Select some corpus files to play with:
|
|
|
|
>>> import nltk.data
|
|
>>> # A very short file (160 chars):
|
|
>>> f1 = nltk.data.find('corpora/inaugural/README')
|
|
>>> # A relatively short file (791 chars):
|
|
>>> f2 = nltk.data.find('corpora/inaugural/1793-Washington.txt')
|
|
>>> # A longer file (32k chars):
|
|
>>> f3 = nltk.data.find('corpora/inaugural/1909-Taft.txt')
|
|
>>> fileids = [f1, f2, f3]
|
|
|
|
|
|
Concatenation
|
|
-------------
|
|
Check that concatenation works as intended.
|
|
|
|
>>> from nltk.corpus.reader.util import *
|
|
|
|
>>> c1 = StreamBackedCorpusView(f1, read_whitespace_block, encoding='utf-8')
|
|
>>> c2 = StreamBackedCorpusView(f2, read_whitespace_block, encoding='utf-8')
|
|
>>> c3 = StreamBackedCorpusView(f3, read_whitespace_block, encoding='utf-8')
|
|
>>> c123 = c1+c2+c3
|
|
>>> print(c123)
|
|
['C-Span', 'Inaugural', 'Address', 'Corpus', 'US', ...]
|
|
|
|
>>> l1 = f1.open(encoding='utf-8').read().split()
|
|
>>> l2 = f2.open(encoding='utf-8').read().split()
|
|
>>> l3 = f3.open(encoding='utf-8').read().split()
|
|
>>> l123 = l1+l2+l3
|
|
|
|
>>> list(c123) == l123
|
|
True
|
|
|
|
>>> (c1+c2+c3)[100] == l123[100]
|
|
True
|
|
|
|
Slicing
|
|
-------
|
|
First, do some tests with fairly small slices. These will all
|
|
generate tuple values.
|
|
|
|
>>> from nltk.util import LazySubsequence
|
|
>>> c1 = StreamBackedCorpusView(f1, read_whitespace_block, encoding='utf-8')
|
|
>>> l1 = f1.open(encoding='utf-8').read().split()
|
|
>>> print(len(c1))
|
|
21
|
|
>>> len(c1) < LazySubsequence.MIN_SIZE
|
|
True
|
|
|
|
Choose a list of indices, based on the length, that covers the
|
|
important corner cases:
|
|
|
|
>>> indices = [-60, -30, -22, -21, -20, -1,
|
|
... 0, 1, 10, 20, 21, 22, 30, 60]
|
|
|
|
Test slicing with explicit start & stop value:
|
|
|
|
>>> for s in indices:
|
|
... for e in indices:
|
|
... assert list(c1[s:e]) == l1[s:e]
|
|
|
|
Test slicing with stop=None:
|
|
|
|
>>> for s in indices:
|
|
... assert list(c1[s:]) == l1[s:]
|
|
|
|
Test slicing with start=None:
|
|
|
|
>>> for e in indices:
|
|
... assert list(c1[:e]) == l1[:e]
|
|
|
|
Test slicing with start=stop=None:
|
|
|
|
>>> list(c1[:]) == list(l1[:])
|
|
True
|
|
|
|
Next, we'll do some tests with much longer slices. These will
|
|
generate LazySubsequence objects.
|
|
|
|
>>> c3 = StreamBackedCorpusView(f3, read_whitespace_block, encoding='utf-8')
|
|
>>> l3 = f3.open(encoding='utf-8').read().split()
|
|
>>> print(len(c3))
|
|
5430
|
|
>>> len(c3) > LazySubsequence.MIN_SIZE*2
|
|
True
|
|
|
|
Choose a list of indices, based on the length, that covers the
|
|
important corner cases:
|
|
|
|
>>> indices = [-12000, -6000, -5431, -5430, -5429, -3000, -200, -1,
|
|
... 0, 1, 200, 3000, 5000, 5429, 5430, 5431, 6000, 12000]
|
|
|
|
Test slicing with explicit start & stop value:
|
|
|
|
>>> for s in indices:
|
|
... for e in indices:
|
|
... assert list(c3[s:e]) == l3[s:e]
|
|
|
|
Test slicing with stop=None:
|
|
|
|
>>> for s in indices:
|
|
... assert list(c3[s:]) == l3[s:]
|
|
|
|
Test slicing with start=None:
|
|
|
|
>>> for e in indices:
|
|
... assert list(c3[:e]) == l3[:e]
|
|
|
|
Test slicing with start=stop=None:
|
|
|
|
>>> list(c3[:]) == list(l3[:])
|
|
True
|
|
|
|
Multiple Iterators
|
|
------------------
|
|
If multiple iterators are created for the same corpus view, their
|
|
iteration can be interleaved:
|
|
|
|
>>> c3 = StreamBackedCorpusView(f3, read_whitespace_block)
|
|
>>> iterators = [c3.iterate_from(n) for n in [0,15,30,45]]
|
|
>>> for i in range(15):
|
|
... for iterator in iterators:
|
|
... print('%-15s' % next(iterator), end=' ')
|
|
... print()
|
|
My a duties in
|
|
fellow heavy of a
|
|
citizens: weight the proper
|
|
Anyone of office sense
|
|
who responsibility. upon of
|
|
has If which the
|
|
taken not, he obligation
|
|
the he is which
|
|
oath has about the
|
|
I no to oath
|
|
have conception enter, imposes.
|
|
just of or The
|
|
taken the he office
|
|
must powers is of
|
|
feel and lacking an
|
|
|
|
SeekableUnicodeStreamReader
|
|
===========================
|
|
|
|
The file-like objects provided by the ``codecs`` module unfortunately
|
|
suffer from a bug that prevents them from working correctly with
|
|
corpus view objects. In particular, although the expose ``seek()``
|
|
and ``tell()`` methods, those methods do not exhibit the expected
|
|
behavior, because they are not synchronized with the internal buffers
|
|
that are kept by the file-like objects. For example, the ``tell()``
|
|
method will return the file position at the end of the buffers (whose
|
|
contents have not yet been returned by the stream); and therefore this
|
|
file position can not be used to return to the 'current' location in
|
|
the stream (since ``seek()`` has no way to reconstruct the buffers).
|
|
|
|
To get around these problems, we define a new class,
|
|
`SeekableUnicodeStreamReader`, to act as a file-like interface to
|
|
files containing encoded unicode data. This class is loosely based on
|
|
the ``codecs.StreamReader`` class. To construct a new reader, we call
|
|
the constructor with an underlying stream and an encoding name:
|
|
|
|
>>> from io import StringIO, BytesIO
|
|
>>> from nltk.data import SeekableUnicodeStreamReader
|
|
>>> stream = BytesIO(b"""\
|
|
... This is a test file.
|
|
... It is encoded in ascii.
|
|
... """.decode('ascii').encode('ascii'))
|
|
>>> reader = SeekableUnicodeStreamReader(stream, 'ascii')
|
|
|
|
`SeekableUnicodeStreamReader`\ s support all of the normal operations
|
|
supplied by a read-only stream. Note that all of the read operations
|
|
return ``unicode`` objects (not ``str`` objects).
|
|
|
|
>>> reader.read() # read the entire file.
|
|
'This is a test file.\nIt is encoded in ascii.\n'
|
|
>>> reader.seek(0) # rewind to the start.
|
|
>>> reader.read(5) # read at most 5 bytes.
|
|
'This '
|
|
>>> reader.readline() # read to the end of the line.
|
|
'is a test file.\n'
|
|
>>> reader.seek(0) # rewind to the start.
|
|
>>> for line in reader:
|
|
... print(repr(line)) # iterate over lines
|
|
'This is a test file.\n'
|
|
'It is encoded in ascii.\n'
|
|
>>> reader.seek(0) # rewind to the start.
|
|
>>> reader.readlines() # read a list of line strings
|
|
['This is a test file.\n', 'It is encoded in ascii.\n']
|
|
>>> reader.close()
|
|
|
|
Size argument to ``read()``
|
|
---------------------------
|
|
The ``size`` argument to ``read()`` specifies the maximum number of
|
|
*bytes* to read, not the maximum number of *characters*. Thus, for
|
|
encodings that use multiple bytes per character, it may return fewer
|
|
characters than the ``size`` argument:
|
|
|
|
>>> stream = BytesIO(b"""\
|
|
... This is a test file.
|
|
... It is encoded in utf-16.
|
|
... """.decode('ascii').encode('utf-16'))
|
|
>>> reader = SeekableUnicodeStreamReader(stream, 'utf-16')
|
|
>>> reader.read(10)
|
|
'This '
|
|
|
|
If a read block ends in the middle of the byte string encoding a
|
|
single character, then that byte string is stored in an internal
|
|
buffer, and re-used on the next call to ``read()``. However, if the
|
|
size argument is too small to read even a single character, even
|
|
though at least one character is available, then the ``read()`` method
|
|
will read additional bytes until it can return a single character.
|
|
This ensures that the ``read()`` method does not return an empty
|
|
string, which could be mistaken for indicating the end of the file.
|
|
|
|
>>> reader.seek(0) # rewind to the start.
|
|
>>> reader.read(1) # we actually need to read 4 bytes
|
|
'T'
|
|
>>> int(reader.tell())
|
|
4
|
|
|
|
The ``readline()`` method may read more than a single line of text, in
|
|
which case it stores the text that it does not return in a buffer. If
|
|
this buffer is not empty, then its contents will be included in the
|
|
value returned by the next call to ``read()``, regardless of the
|
|
``size`` argument, since they are available without reading any new
|
|
bytes from the stream:
|
|
|
|
>>> reader.seek(0) # rewind to the start.
|
|
>>> reader.readline() # stores extra text in a buffer
|
|
'This is a test file.\n'
|
|
>>> print(reader.linebuffer) # examine the buffer contents
|
|
['It is encoded i']
|
|
>>> reader.read(0) # returns the contents of the buffer
|
|
'It is encoded i'
|
|
>>> print(reader.linebuffer) # examine the buffer contents
|
|
None
|
|
|
|
Seek and Tell
|
|
-------------
|
|
In addition to these basic read operations,
|
|
`SeekableUnicodeStreamReader` also supports the ``seek()`` and
|
|
``tell()`` operations. However, some care must still be taken when
|
|
using these operations. In particular, the only file offsets that
|
|
should be passed to ``seek()`` are ``0`` and any offset that has been
|
|
returned by ``tell``.
|
|
|
|
>>> stream = BytesIO(b"""\
|
|
... This is a test file.
|
|
... It is encoded in utf-16.
|
|
... """.decode('ascii').encode('utf-16'))
|
|
>>> reader = SeekableUnicodeStreamReader(stream, 'utf-16')
|
|
>>> reader.read(20)
|
|
'This is a '
|
|
>>> pos = reader.tell(); print(pos)
|
|
22
|
|
>>> reader.read(20)
|
|
'test file.'
|
|
>>> reader.seek(pos) # rewind to the position from tell.
|
|
>>> reader.read(20)
|
|
'test file.'
|
|
|
|
The ``seek()`` and ``tell()`` methods work property even when
|
|
``readline()`` is used.
|
|
|
|
>>> stream = BytesIO(b"""\
|
|
... This is a test file.
|
|
... It is encoded in utf-16.
|
|
... """.decode('ascii').encode('utf-16'))
|
|
>>> reader = SeekableUnicodeStreamReader(stream, 'utf-16')
|
|
>>> reader.readline()
|
|
'This is a test file.\n'
|
|
>>> pos = reader.tell(); print(pos)
|
|
44
|
|
>>> reader.readline()
|
|
'It is encoded in utf-16.\n'
|
|
>>> reader.seek(pos) # rewind to the position from tell.
|
|
>>> reader.readline()
|
|
'It is encoded in utf-16.\n'
|
|
|
|
|
|
Squashed Bugs
|
|
=============
|
|
|
|
svn 5276 fixed a bug in the comment-stripping behavior of
|
|
parse_sexpr_block.
|
|
|
|
>>> from io import StringIO
|
|
>>> from nltk.corpus.reader.util import read_sexpr_block
|
|
>>> f = StringIO(b"""
|
|
... (a b c)
|
|
... # This line is a comment.
|
|
... (d e f\ng h)""".decode('ascii'))
|
|
>>> print(read_sexpr_block(f, block_size=38, comment_char='#'))
|
|
['(a b c)']
|
|
>>> print(read_sexpr_block(f, block_size=38, comment_char='#'))
|
|
['(d e f\ng h)']
|
|
|
|
svn 5277 fixed a bug in parse_sexpr_block, which would cause it to
|
|
enter an infinite loop if a file ended mid-sexpr, or ended with a
|
|
token that was not followed by whitespace. A related bug caused
|
|
an infinite loop if the corpus ended in an unmatched close paren --
|
|
this was fixed in svn 5279
|
|
|
|
>>> f = StringIO(b"""
|
|
... This file ends mid-sexpr
|
|
... (hello (world""".decode('ascii'))
|
|
>>> for i in range(3): print(read_sexpr_block(f))
|
|
['This', 'file', 'ends', 'mid-sexpr']
|
|
['(hello (world']
|
|
[]
|
|
|
|
>>> f = StringIO(b"This file has no trailing whitespace.".decode('ascii'))
|
|
>>> for i in range(3): print(read_sexpr_block(f))
|
|
['This', 'file', 'has', 'no', 'trailing']
|
|
['whitespace.']
|
|
[]
|
|
|
|
>>> # Bug fixed in 5279:
|
|
>>> f = StringIO(b"a b c)".decode('ascii'))
|
|
>>> for i in range(3): print(read_sexpr_block(f))
|
|
['a', 'b']
|
|
['c)']
|
|
[]
|
|
|
|
|
|
svn 5624 & 5265 fixed a bug in ConcatenatedCorpusView, which caused it
|
|
to return the wrong items when indexed starting at any index beyond
|
|
the first file.
|
|
|
|
>>> import nltk
|
|
>>> sents = nltk.corpus.brown.sents()
|
|
>>> print(sents[6000])
|
|
['Cholesterol', 'and', 'thyroid']
|
|
>>> print(sents[6000])
|
|
['Cholesterol', 'and', 'thyroid']
|
|
|
|
svn 5728 fixed a bug in Categorized*CorpusReader, which caused them
|
|
to return words from *all* files when just one file was specified.
|
|
|
|
>>> from nltk.corpus import reuters
|
|
>>> reuters.words('training/13085')
|
|
['SNYDER', '&', 'lt', ';', 'SOI', '>', 'MAKES', ...]
|
|
>>> reuters.words('training/5082')
|
|
['SHEPPARD', 'RESOURCES', 'TO', 'MERGE', 'WITH', ...]
|
|
|
|
svn 7227 fixed a bug in the qc corpus reader, which prevented
|
|
access to its tuples() method
|
|
|
|
>>> from nltk.corpus import qc
|
|
>>> qc.tuples('test.txt')
|
|
[('NUM:dist', 'How far is it from Denver to Aspen ?'), ('LOC:city', 'What county is Modesto , California in ?'), ...]
|
|
|
|
|
|
|