S13-Words-for-the-Future-notebooks/working-with-language/nltk-frequency-distribution.ipynb

NLTK - Frequency Distribution¶

https://www.nltk.org/book/ch01.html#frequency-distributions

In [1]:

import nltk
import random

In [2]:

lines = open('txt/language.txt').readlines()
sentence = random.choice(lines)
print(sentence)

The symbols of computer control languages inevitably do have semantic connotations simply because there exist no symbols with which humans would not associate some meaning.

Tokens¶

In [3]:

tokens = nltk.word_tokenize(sentence)
print(tokens)

['The', 'symbols', 'of', 'computer', 'control', 'languages', 'inevitably', 'do', 'have', 'semantic', 'connotations', 'simply', 'because', 'there', 'exist', 'no', 'symbols', 'with', 'which', 'humans', 'would', 'not', 'associate', 'some', 'meaning', '.']

In [ ]:

Frequency Distribution¶

In [4]:

# frequency of characters
fd = nltk.FreqDist(sentence)
print(fd)

<FreqDist with 26 samples and 173 outcomes>

In [5]:

print(fd.most_common(50))

[(' ', 24), ('o', 15), ('e', 14), ('s', 14), ('n', 12), ('t', 11), ('a', 11), ('i', 10), ('m', 8), ('h', 7), ('l', 7), ('c', 7), ('u', 5), ('y', 4), ('b', 4), ('r', 3), ('g', 3), ('w', 3), ('p', 2), ('v', 2), ('d', 2), ('T', 1), ('f', 1), ('x', 1), ('.', 1), ('\n', 1)]

In [ ]:

In [6]:

# frequency of words
fd = nltk.FreqDist(tokens)
print(fd)

<FreqDist with 25 samples and 26 outcomes>

In [7]:

print(fd.most_common(50))

[('symbols', 2), ('The', 1), ('of', 1), ('computer', 1), ('control', 1), ('languages', 1), ('inevitably', 1), ('do', 1), ('have', 1), ('semantic', 1), ('connotations', 1), ('simply', 1), ('because', 1), ('there', 1), ('exist', 1), ('no', 1), ('with', 1), ('which', 1), ('humans', 1), ('would', 1), ('not', 1), ('associate', 1), ('some', 1), ('meaning', 1), ('.', 1)]

In [ ]:

In [8]:

# frequency of a text
txt = open('txt/language.txt').read()
tokens = nltk.word_tokenize(txt)
fd = nltk.FreqDist(tokens)
print(fd)

<FreqDist with 944 samples and 2835 outcomes>

In [9]:

print(fd.most_common(50))

[(',', 172), ('.', 93), ('the', 88), ('of', 88), ('”', 66), ('“', 65), ('and', 61), ('a', 61), ('is', 58), ('languages', 54), ('in', 51), ('language', 47), ('to', 41), ('as', 37), ('computer', 32), ('that', 29), ('programming', 25), ('control', 23), ('are', 22), ('for', 21), ('’', 21), ('The', 18), ('can', 17), ('be', 16), ('it', 16), ('machine', 16), ('human', 15), ('not', 15), ('software', 14), ('formal', 14), ('or', 14), ('symbols', 14), ('s', 12), ('with', 12), (':', 11), ('its', 11), ('this', 11), ('common', 11), ('their', 10), ('example', 9), (';', 9), ('operations', 9), ('such', 9), ('from', 8), ('through', 8), ('code', 8), ('since', 7), ('different', 7), ('In', 7), ('like', 7)]

In [ ]:

In [10]:

# Requesting the frequency of a specific word
print(fd['language'])

47

In [ ]: