语料库中每个文本的平均句子长度 (python3 & nltk)

Question

我正在分析 NLTK 包中的就职地址语料库，作为 python 编程课程介绍的一部分。我想找出语料库中每个文本的平均句子长度（以便我以后可以比较它们），但我似乎被困在这里。

我创建了这个函数：

def averageSentence(text):
    sents = inaugural.sents(fileids=['fileid_here.txt']  
    avg = sum(len(word) for word in sents) / len(sents)  
    print(avg)

哪个（如果我是正确的）应该给我单个文本的平均句子长度。现在，我知道我需要一个 for 循环 。难道我不能用我刚定义的这个函数做一个相对简单直接的 for 循环吗？这非常令人沮丧。

编辑：这是我的进展：

for fileid in inaugural.fileids():
    avg_sents = averageSentence(fileid)
    print = sum(avg_sents) / avg_sents

Answer 1

尝试：

>>> from __future__ import division
>>> from nltk.corpus import inaugural
>>> total_lens = 0
>>> for i, sent in enumerate(inaugural.sents()):
...     total_lens += len(sent)
... 
>>> total_lens 
145735
>>> i
4867
>>> avg_sent_len = total_lens / i
>>> avg_sent_len
29.943497020752
>>> avg_sent_len = total_lens / (i+1)
>>> avg_sent_len
29.9373459326212

请注意，当分母足够大时，+1 并不重要。

所有文本的微平均句子长度

以下代码是单行代码，但不鼓励这样做，因为您可能已经实现了两次生成器：

>>> sum(len(sent) for sent in inaugural.sents()) / len(inaugural.sents())
29.9373459326212

所有文本的宏观平均句子长度：

>>> sum(sum(len(sent) for sent in inaugural.sents(fileids=[fileid])) / len(inaugural.sents(fileids=[fileid])) for fileid in inaugural.fileids()) / len(inaugural.fileids())
32.84054349411484

每篇文章的平均句子长度:

>>> from __future__ import division
>>> from nltk.corpus import inaugural
>>> inaugural.fileids()
[u'1789-Washington.txt', u'1793-Washington.txt', u'1797-Adams.txt', u'1801-Jefferson.txt', u'1805-Jefferson.txt', u'1809-Madison.txt', u'1813-Madison.txt', u'1817-Monroe.txt', u'1821-Monroe.txt', u'1825-Adams.txt', u'1829-Jackson.txt', u'1833-Jackson.txt', u'1837-VanBuren.txt', u'1841-Harrison.txt', u'1845-Polk.txt', u'1849-Taylor.txt', u'1853-Pierce.txt', u'1857-Buchanan.txt', u'1861-Lincoln.txt', u'1865-Lincoln.txt', u'1869-Grant.txt', u'1873-Grant.txt', u'1877-Hayes.txt', u'1881-Garfield.txt', u'1885-Cleveland.txt', u'1889-Harrison.txt', u'1893-Cleveland.txt', u'1897-McKinley.txt', u'1901-McKinley.txt', u'1905-Roosevelt.txt', u'1909-Taft.txt', u'1913-Wilson.txt', u'1917-Wilson.txt', u'1921-Harding.txt', u'1925-Coolidge.txt', u'1929-Hoover.txt', u'1933-Roosevelt.txt', u'1937-Roosevelt.txt', u'1941-Roosevelt.txt', u'1945-Roosevelt.txt', u'1949-Truman.txt', u'1953-Eisenhower.txt', u'1957-Eisenhower.txt', u'1961-Kennedy.txt', u'1965-Johnson.txt', u'1969-Nixon.txt', u'1973-Nixon.txt', u'1977-Carter.txt', u'1981-Reagan.txt', u'1985-Reagan.txt', u'1989-Bush.txt', u'1993-Clinton.txt', u'1997-Clinton.txt', u'2001-Bush.txt', u'2005-Bush.txt', u'2009-Obama.txt']
>>> for fileid in inaugural.fileids():
...     avg = sum(len(sent) for sent in inaugural.sents(fileids=[fileid])) / len(inaugural.sents(fileids=[fileid]))
...     print fileid, avg
... 
1789-Washington.txt 64.0833333333
1793-Washington.txt 36.75
1797-Adams.txt 69.8648648649
1801-Jefferson.txt 46.0714285714
1805-Jefferson.txt 52.9777777778
1809-Madison.txt 60.2380952381
1813-Madison.txt 39.5151515152
1817-Monroe.txt 30.2704918033
1821-Monroe.txt 38.0542635659
1825-Adams.txt 42.5675675676
1829-Jackson.txt 48.32
1833-Jackson.txt 42.2333333333
1837-VanBuren.txt 43.9052631579
1841-Harrison.txt 43.6428571429
1845-Polk.txt 33.9607843137
1849-Taylor.txt 53.7272727273
1853-Pierce.txt 35.1634615385
1857-Buchanan.txt 34.808988764
1861-Lincoln.txt 29.0217391304
1865-Lincoln.txt 29.0740740741
1869-Grant.txt 30.2195121951
1873-Grant.txt 33.5909090909
1877-Hayes.txt 46.1694915254
1881-Garfield.txt 28.9196428571
1885-Cleveland.txt 41.5454545455
1889-Harrison.txt 30.2547770701
1893-Cleveland.txt 37.1206896552
1897-McKinley.txt 33.6230769231
1901-McKinley.txt 24.5
1905-Roosevelt.txt 33.0606060606
1909-Taft.txt 36.7672955975
1913-Wilson.txt 28.0147058824
1917-Wilson.txt 27.6
1921-Harding.txt 25.2080536913
1925-Coolidge.txt 22.5482233503
1929-Hoover.txt 24.6202531646
1933-Roosevelt.txt 24.2705882353
1937-Roosevelt.txt 21.03125
1941-Roosevelt.txt 22.5882352941
1945-Roosevelt.txt 24.5
1949-Truman.txt 21.7931034483
1953-Eisenhower.txt 22.5609756098
1957-Eisenhower.txt 20.8369565217
1961-Kennedy.txt 29.7307692308
1965-Johnson.txt 18.2446808511
1969-Nixon.txt 22.8773584906
1973-Nixon.txt 29.3913043478
1977-Carter.txt 26.0377358491
1981-Reagan.txt 22.0551181102
1985-Reagan.txt 23.380952381
1989-Bush.txt 18.7103448276
1993-Clinton.txt 22.9012345679
1997-Clinton.txt 21.9821428571
2001-Bush.txt 18.8144329897
2005-Bush.txt 25.0105263158
2009-Obama.txt 24.3392857143

所有文本的平均字长：

>>> sum([sum(len(sent) for sent in inaugural.sents(fileids=[fileid])) for fileid in inaugural.fileids()]) / len(inaugural.fileids())
2602.410714285714

语料库中每个文本的平均句子长度 (python3 & nltk)

Average sentence length for every text in corpus (python3 & nltk)

python

nlp

iterable

average

nltk