NLTK - 获取和简化标签列表
NLTK - Get and Simplify List of Tags
我正在使用 Brown Corpus。我想要一些方法来打印出所有可能的标签及其名称（不仅仅是标签缩写）。还有很多标签，有没有办法'simplify'标签？我所说的简化是指将两个极其相似的标签合并为一个，然后用另一个标签重新标记合并后的词？
之前以某种方式讨论过：
Java Stanford NLP: Part of Speech labels?
Simplifying the French POS Tag Set with NLTK
https://linguistics.stackexchange.com/questions/2249/turn-penn-treebank-into-simpler-pos-tags
从 nltk.pos_tag 输出的 POS 标签是 PennTreeBank 标签集，https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html, see What are all possible pos tags of NLTK?
有多种方法，但最简单的方法可能是仅使用 POS 的前 2 个字符作为 POS 标签的主要集合。这是因为 POS 标签中的前两个字符代表 Penn Tree Bank 标签集中 POS 的广泛 类。
例如NNS表示复数名词，NNP表示专有名词，NN标签通过表示通用名词来包含所有这些名词。
这是一个代码示例：
>>> from nltk.corpus import brown
>>> from collections import Counter
>>> x = defaultdict(list)
>>> for word,pos in brown.tagged_words()[1:100]:
... x[pos].append(word)
...
>>> x
defaultdict(<type 'list'>, {u'DTI': [u'any'], u'BEN': [u'been'], u'VBD': [u'said', u'produced', u'took', u'said'], u'NP$': [u"Atlanta's"], u'NN-TL': [u'County', u'Jury', u'City', u'Committee', u'City', u'Court', u'Judge', u'Mayor-nominate'], u'VBN': [u'conducted', u'charged', u'won'], u"''": [u"''", u"''", u"''"], u'WDT': [u'which', u'which', u'which'], u'JJ': [u'recent', u'over-all', u'possible', u'hard-fought'], u'VBZ': [u'deserves'], u'NN': [u'investigation', u'primary', u'election', u'evidence', u'place', u'jury', u'term-end', u'charge', u'election', u'praise', u'manner', u'election', u'term', u'jury', u'primary'], u',': [u',', u','], u'.': [u'.', u'.'], u'TO': [u'to'], u'NP': [u'September-October', u'Durwood', u'Pye', u'Ivan'], u'BEDZ': [u'was', u'was'], u'NR': [u'Friday'], u'NNS': [u'irregularities', u'presentments', u'thanks', u'reports', u'irregularities'], u'``': [u'``', u'``', u'``'], u'CC': [u'and'], u'RBR': [u'further'], u'AT': [u'an', u'no', u'The', u'the', u'the', u'the', u'the', u'the', u'the', u'The', u'the'], u'IN': [u'of', u'in', u'of', u'of', u'for', u'in', u'by', u'of', u'in', u'by'], u'CS': [u'that', u'that'], u'NP-TL': [u'Fulton', u'Atlanta', u'Fulton'], u'HVD': [u'had', u'had'], u'IN-TL': [u'of'], u'VB': [u'investigate'], u'JJ-TL': [u'Grand', u'Executive', u'Superior']})
>>> len(x)
29
缩短版如下所示：
>>> x = defaultdict(list)
>>> for word,pos in brown.tagged_words()[1:100]:
... x[pos[:2]].append(word)
...
>>> x
defaultdict(<type 'list'>, {u'BE': [u'was', u'been', u'was'], u'VB': [u'said', u'produced', u'took', u'said', u'deserves', u'conducted', u'charged', u'investigate', u'won'], u'WD': [u'which', u'which', u'which'], u'RB': [u'further'], u'NN': [u'County', u'Jury', u'investigation', u'primary', u'election', u'evidence', u'irregularities', u'place', u'jury', u'term-end', u'presentments', u'City', u'Committee', u'charge', u'election', u'praise', u'thanks', u'City', u'manner', u'election', u'term', u'jury', u'Court', u'Judge', u'reports', u'irregularities', u'primary', u'Mayor-nominate'], u'TO': [u'to'], u'CC': [u'and'], u'HV': [u'had', u'had'], u'``': [u'``', u'``', u'``'], u',': [u',', u','], u'.': [u'.', u'.'], u"''": [u"''", u"''", u"''"], u'CS': [u'that', u'that'], u'AT': [u'an', u'no', u'The', u'the', u'the', u'the', u'the', u'the', u'the', u'The', u'the'], u'JJ': [u'Grand', u'recent', u'Executive', u'over-all', u'Superior', u'possible', u'hard-fought'], u'IN': [u'of', u'in', u'of', u'of', u'of', u'for', u'in', u'by', u'of', u'in', u'by'], u'NP': [u'Fulton', u"Atlanta's", u'Atlanta', u'September-October', u'Fulton', u'Durwood', u'Pye', u'Ivan'], u'NR': [u'Friday'], u'DT': [u'any']})
>>> len(x)
19
另一个解决方案是使用通用邮寄标签，参见http://www.nltk.org/book/ch05.html
>>> x = defaultdict(list)
>>> for word,pos in brown.tagged_words(tagset='universal')[1:100]:
... x[pos].append(word)
...
>>> x
defaultdict(<type 'list'>, {u'ADV': [u'further'], u'NOUN': [u'Fulton', u'County', u'Jury', u'Friday', u'investigation', u"Atlanta's", u'primary', u'election', u'evidence', u'irregularities', u'place', u'jury', u'term-end', u'presentments', u'City', u'Committee', u'charge', u'election', u'praise', u'thanks', u'City', u'Atlanta', u'manner', u'election', u'September-October', u'term', u'jury', u'Fulton', u'Court', u'Judge', u'Durwood', u'Pye', u'reports', u'irregularities', u'primary', u'Mayor-nominate', u'Ivan'], u'ADP': [u'of', u'that', u'in', u'that', u'of', u'of', u'of', u'for', u'in', u'by', u'of', u'in', u'by'], u'DET': [u'an', u'no', u'any', u'The', u'the', u'which', u'the', u'the', u'the', u'the', u'which', u'the', u'The', u'the', u'which'], u'.': [u'``', u"''", u'.', u',', u',', u'``', u"''", u'.', u'``', u"''"], u'PRT': [u'to'], u'VERB': [u'said', u'produced', u'took', u'said', u'had', u'deserves', u'was', u'conducted', u'had', u'been', u'charged', u'investigate', u'was', u'won'], u'CONJ': [u'and'], u'ADJ': [u'Grand', u'recent', u'Executive', u'over-all', u'Superior', u'possible', u'hard-fought']})
>>> len(x)
9
NLTK 语料库中的许多标签集都带有到简化的 "universal" 标签集的预定义映射。除了对许多用途更方便之外，简化的标签集允许不同语料库之间一定程度的兼容性，允许重新映射到通用标签集。
对于棕色语料库，您可以像这样简单地获取标记词或发送：
brown.tagged_words(tagset="universal")
例如：
>>> print(brown.tagged_words()[:10])
[('The', 'DET'), ('Fulton', 'NOUN'), ('County', 'NOUN'), ('Grand', 'ADJ'), ('Jury', 'NOUN'),
('said', 'VERB'), ('Friday', 'NOUN'), ('an', 'DET'), ('investigation', 'NOUN'),
('of', 'ADP')]
要查看 Brown 语料库中原始复杂标签的定义，请使用
nltk.help.upenn_tagset()（如 this 答案中所述，由 Alvas 链接）。您可以通过不带参数调用它来获取整个列表，或者传递参数（正则表达式）以仅获取匹配的标签。结果包括简要定义和示例。
>>> nltk.help.brown_tagset("DT.*")
DT: determiner/pronoun, singular
this each another that 'nother
DT$: determiner/pronoun, singular, genitive
another's
DT+BEZ: determiner/pronoun + verb 'to be', present tense, 3rd person singular
that's
DT+MD: determiner/pronoun + modal auxillary
that'll this'll
...
我正在使用 Brown Corpus。我想要一些方法来打印出所有可能的标签及其名称（不仅仅是标签缩写）。还有很多标签，有没有办法'simplify'标签？我所说的简化是指将两个极其相似的标签合并为一个，然后用另一个标签重新标记合并后的词？
之前以某种方式讨论过：
Java Stanford NLP: Part of Speech labels?
Simplifying the French POS Tag Set with NLTK
https://linguistics.stackexchange.com/questions/2249/turn-penn-treebank-into-simpler-pos-tags
从 nltk.pos_tag 输出的 POS 标签是 PennTreeBank 标签集，https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html, see What are all possible pos tags of NLTK?
有多种方法，但最简单的方法可能是仅使用 POS 的前 2 个字符作为 POS 标签的主要集合。这是因为 POS 标签中的前两个字符代表 Penn Tree Bank 标签集中 POS 的广泛 类。
例如NNS表示复数名词，NNP表示专有名词，NN标签通过表示通用名词来包含所有这些名词。
这是一个代码示例：
>>> from nltk.corpus import brown
>>> from collections import Counter
>>> x = defaultdict(list)
>>> for word,pos in brown.tagged_words()[1:100]:
... x[pos].append(word)
...
>>> x
defaultdict(<type 'list'>, {u'DTI': [u'any'], u'BEN': [u'been'], u'VBD': [u'said', u'produced', u'took', u'said'], u'NP$': [u"Atlanta's"], u'NN-TL': [u'County', u'Jury', u'City', u'Committee', u'City', u'Court', u'Judge', u'Mayor-nominate'], u'VBN': [u'conducted', u'charged', u'won'], u"''": [u"''", u"''", u"''"], u'WDT': [u'which', u'which', u'which'], u'JJ': [u'recent', u'over-all', u'possible', u'hard-fought'], u'VBZ': [u'deserves'], u'NN': [u'investigation', u'primary', u'election', u'evidence', u'place', u'jury', u'term-end', u'charge', u'election', u'praise', u'manner', u'election', u'term', u'jury', u'primary'], u',': [u',', u','], u'.': [u'.', u'.'], u'TO': [u'to'], u'NP': [u'September-October', u'Durwood', u'Pye', u'Ivan'], u'BEDZ': [u'was', u'was'], u'NR': [u'Friday'], u'NNS': [u'irregularities', u'presentments', u'thanks', u'reports', u'irregularities'], u'``': [u'``', u'``', u'``'], u'CC': [u'and'], u'RBR': [u'further'], u'AT': [u'an', u'no', u'The', u'the', u'the', u'the', u'the', u'the', u'the', u'The', u'the'], u'IN': [u'of', u'in', u'of', u'of', u'for', u'in', u'by', u'of', u'in', u'by'], u'CS': [u'that', u'that'], u'NP-TL': [u'Fulton', u'Atlanta', u'Fulton'], u'HVD': [u'had', u'had'], u'IN-TL': [u'of'], u'VB': [u'investigate'], u'JJ-TL': [u'Grand', u'Executive', u'Superior']})
>>> len(x)
29
缩短版如下所示：
>>> x = defaultdict(list)
>>> for word,pos in brown.tagged_words()[1:100]:
... x[pos[:2]].append(word)
...
>>> x
defaultdict(<type 'list'>, {u'BE': [u'was', u'been', u'was'], u'VB': [u'said', u'produced', u'took', u'said', u'deserves', u'conducted', u'charged', u'investigate', u'won'], u'WD': [u'which', u'which', u'which'], u'RB': [u'further'], u'NN': [u'County', u'Jury', u'investigation', u'primary', u'election', u'evidence', u'irregularities', u'place', u'jury', u'term-end', u'presentments', u'City', u'Committee', u'charge', u'election', u'praise', u'thanks', u'City', u'manner', u'election', u'term', u'jury', u'Court', u'Judge', u'reports', u'irregularities', u'primary', u'Mayor-nominate'], u'TO': [u'to'], u'CC': [u'and'], u'HV': [u'had', u'had'], u'``': [u'``', u'``', u'``'], u',': [u',', u','], u'.': [u'.', u'.'], u"''": [u"''", u"''", u"''"], u'CS': [u'that', u'that'], u'AT': [u'an', u'no', u'The', u'the', u'the', u'the', u'the', u'the', u'the', u'The', u'the'], u'JJ': [u'Grand', u'recent', u'Executive', u'over-all', u'Superior', u'possible', u'hard-fought'], u'IN': [u'of', u'in', u'of', u'of', u'of', u'for', u'in', u'by', u'of', u'in', u'by'], u'NP': [u'Fulton', u"Atlanta's", u'Atlanta', u'September-October', u'Fulton', u'Durwood', u'Pye', u'Ivan'], u'NR': [u'Friday'], u'DT': [u'any']})
>>> len(x)
19
另一个解决方案是使用通用邮寄标签，参见http://www.nltk.org/book/ch05.html
>>> x = defaultdict(list)
>>> for word,pos in brown.tagged_words(tagset='universal')[1:100]:
... x[pos].append(word)
...
>>> x
defaultdict(<type 'list'>, {u'ADV': [u'further'], u'NOUN': [u'Fulton', u'County', u'Jury', u'Friday', u'investigation', u"Atlanta's", u'primary', u'election', u'evidence', u'irregularities', u'place', u'jury', u'term-end', u'presentments', u'City', u'Committee', u'charge', u'election', u'praise', u'thanks', u'City', u'Atlanta', u'manner', u'election', u'September-October', u'term', u'jury', u'Fulton', u'Court', u'Judge', u'Durwood', u'Pye', u'reports', u'irregularities', u'primary', u'Mayor-nominate', u'Ivan'], u'ADP': [u'of', u'that', u'in', u'that', u'of', u'of', u'of', u'for', u'in', u'by', u'of', u'in', u'by'], u'DET': [u'an', u'no', u'any', u'The', u'the', u'which', u'the', u'the', u'the', u'the', u'which', u'the', u'The', u'the', u'which'], u'.': [u'``', u"''", u'.', u',', u',', u'``', u"''", u'.', u'``', u"''"], u'PRT': [u'to'], u'VERB': [u'said', u'produced', u'took', u'said', u'had', u'deserves', u'was', u'conducted', u'had', u'been', u'charged', u'investigate', u'was', u'won'], u'CONJ': [u'and'], u'ADJ': [u'Grand', u'recent', u'Executive', u'over-all', u'Superior', u'possible', u'hard-fought']})
>>> len(x)
9
NLTK 语料库中的许多标签集都带有到简化的 "universal" 标签集的预定义映射。除了对许多用途更方便之外，简化的标签集允许不同语料库之间一定程度的兼容性，允许重新映射到通用标签集。
对于棕色语料库，您可以像这样简单地获取标记词或发送：
brown.tagged_words(tagset="universal")
例如：
>>> print(brown.tagged_words()[:10])
[('The', 'DET'), ('Fulton', 'NOUN'), ('County', 'NOUN'), ('Grand', 'ADJ'), ('Jury', 'NOUN'),
('said', 'VERB'), ('Friday', 'NOUN'), ('an', 'DET'), ('investigation', 'NOUN'),
('of', 'ADP')]
要查看 Brown 语料库中原始复杂标签的定义，请使用
nltk.help.upenn_tagset()（如 this 答案中所述，由 Alvas 链接）。您可以通过不带参数调用它来获取整个列表，或者传递参数（正则表达式）以仅获取匹配的标签。结果包括简要定义和示例。
>>> nltk.help.brown_tagset("DT.*")
DT: determiner/pronoun, singular
this each another that 'nother
DT$: determiner/pronoun, singular, genitive
another's
DT+BEZ: determiner/pronoun + verb 'to be', present tense, 3rd person singular
that's
DT+MD: determiner/pronoun + modal auxillary
that'll this'll
...