Count words in Python

len(text.split()) counts the words in a string: split() with no argument splits at every run of white space and drops empty strings, so leading, trailing and doubled spaces don’t add words.

text = "We don't ship on Fridays — the follow-up release goes out Monday and/or Tuesday 👍"
len(text.split())
# 15
text.split()
# ['We', "don't", 'ship', 'on', 'Fridays', '—', 'the', 'follow-up', 'release', 'goes', 'out', 'Monday', 'and/or', 'Tuesday', '👍']

That’s 15 because anything between spaces counts: the dash and the emoji are words, and and/or is one. The counter below says 14 for the same sentence. It leaves out the dash and the emoji, and counts and and or apart.

split(” “) is a different function

split(" ") splits at each single space and keeps the empty strings between two of them: "a b".split(" ") is ['a', '', 'b'], 3 words. split() also splits at tabs, line breaks and Unicode spaces such as the no-break space   and the ideographic space 　, but not at the zero-width space ​. An empty or blank string gives 0.

A regex that skips punctuation

\b\w+\b is the usual answer, and it splits at apostrophes and hyphens: 16 words here, starting ['We', 'don', 't', 'ship']. Let ', ’ and - join word characters instead:

import re

WORD = re.compile(r"\w+(?:[-'’]\w+)*")
len(WORD.findall(text))
# 14

That’s the tool’s count for this sentence, and \w matches letters in every script, so naïve café is 2. Where it still parts from the tool:

split()  WORD regex  tool  text
2        3           2     1,000 users
2        3           2     at 9:00
2        7           2     see https://example.com/notes?v=2
1        2           6     我吃苹果。今天天气很好。

The tool reads words by Unicode’s rules (UAX #29), as Intl.Segmenter does in a browser, keeps numbers, times, URLs and email addresses whole, and splits Chinese, Japanese and Thai with dictionaries. Python’s standard library has no word segmenter for text without spaces.

Decomposed accents

Unicode lets ï be spelled as i plus a combining diaeresis, ̈ (the NFD form), and \w doesn’t match the combining mark. WORD.findall("naïve") gives ['nai', 've']. Normalize first:

import unicodedata
len(WORD.findall(unicodedata.normalize("NFC", text)))

A whole file, and the most used words

with open("essay.txt", encoding="utf-8") as f:
    words = sum(len(line.split()) for line in f)

Line by line, a big file never sits in memory at once. For a frequency table like the tool’s, count lowercased words:

from collections import Counter
Counter(w.lower() for w in WORD.findall("The cat and the hat. The end.")).most_common(2)
# [('the', 3), ('cat', 1)]

The tool’s table also leaves out common words such as the, and and of.