How to Implement Text Tokenization using Regex in Python

Text tokenization is the process of breaking a stream of text up into words, phrases, symbols, or other meaningful elements called tokens. It is a fundamental step in building compilers, interpreters, and Natural Language Processing (NLP) systems.

While libraries like NLTK exist, building a tokenizer from scratch using Python's re (Regular Expression) module gives you complete control over defining your own grammar and rules.

This guide demonstrates how to build a lexer that categorizes parts of an equation (variables, numbers, operators) into structured tokens.

Defining the Token Structure

A token typically consists of a Type (what is it?) and a Value (what is the actual text?). Python's namedtuple is an excellent choice for this data structure because it is lightweight and immutable.

from collections import namedtuple

Token = namedtuple("Token", ["type", "value"])

t = Token("NUM", "42")
print(f"Type: {t.type}, Value: {t.value}")

Output:

Type: NUM, Value: 42

Designing Regex Patterns

To tokenize text, we need to define patterns for every valid element in our language. We map specific names (like NUM or ADD) to regex strings.

import re

token_specification = {
    "NAME": r"[a-zA-Z_][a-zA-Z_0-9]*",  # Variable names
    "NUM":  r"\d+",                     # Integers
    "ADD":  r"\+",                      # Plus sign
    "SUB":  r"-",                       # Minus sign
    "MUL":  r"\*",                      # Multiply
    "DIV":  r"/",                       # Divide
    "EQ":   r"=",                       # Assignment
    "WS":   r"\s+",                     # Whitespace
}

Implementing the Tokenizer Logic

We need a function that iterates through the text and matches these patterns.

The Naive vs. Efficient Approach

⛔️ The Incorrect/Inefficient Way: Iterating through the text character by character or using simple .split() often fails to separate operators from numbers without spaces.

text = "val=1+2"
tokens = text.split() 

✅ The Correct Way: We compile all patterns into a single Master Regex using Named Groups ((?P<NAME>pattern)). Python's re.finditer scans the text and returns match objects, allowing us to identify which group matched.

import re
from collections import namedtuple

Token = namedtuple("Token", ["type", "value"])

def generate_tokens(text):
    token_specification = {
        "NAME": r"[a-zA-Z_][a-zA-Z_0-9]*",
        "NUM":  r"\d+",
        "ADD":  r"\+",
        "MUL":  r"\*",
        "EQ":   r"=",
        "WS":   r"\s+",
    }

    regex = "|".join("(?P<%s>%s)" % pair for pair in token_specification.items())

    scanner = re.finditer(regex, text)
    
    for m in scanner:
        token_type = m.lastgroup # Gets the name of the matching group (e.g., 'NUM')
        token_value = m.group()  # Gets the actual text matched (e.g., '10')
        
        
        yield Token(token_type, token_value)

Testing and Output

Let's run the tokenizer on a sample string containing variables, numbers, and operators.

if __name__ == "__main__":
    sample_text = "total = 1 + 2 * 3"
    
    tokens = list(generate_tokens(sample_text))
    
    for t in tokens:
        print(t)

Output:

Token(type='NAME', value='total')
Token(type='WS', value=' ')
Token(type='EQ', value='=')
Token(type='WS', value=' ')
Token(type='NUM', value='1')
Token(type='WS', value=' ')
Token(type='ADD', value='+')
Token(type='WS', value=' ')
Token(type='NUM', value='2')
Token(type='WS', value=' ')
Token(type='MUL', value='*')
Token(type='WS', value=' ')
Token(type='NUM', value='3')

Conclusion

Building a tokenizer in Python relies on leveraging the re module effectively.

1. Define Rules: Map your token names to specific regex patterns.
2. Named Groups: Combine patterns using (?P<NAME>pattern) syntax.
3. Find Iter: Use re.finditer to scan the text linearly and extract tokens based on the matched group name.

This approach creates a robust foundation for building parsers or interpreters.