Regular expressions: a practical guide with Python examples

Regular expressions have a mixed reputation. On the one hand they are an extremely powerful tool: in one line you can search, validate or transform text in ways that would take dozens of lines of code. On the other, a complex regular expression can become unreadable even to whoever wrote it. The secret is to know the basic elements well and use them for what they excel at, without expecting them to solve everything.

What a regular expression is

A regular expression, or regex, is a pattern describing a set of strings. The pattern colou?r describes both "color" and "colour". A regex engine compares the pattern with a text and finds the parts that match.

In Python they are used through the standard library's re module:

import re

text = "Orders: A-1042, B-0077, A-2210"
codes = re.findall(r"[A-Z]-\d{4}", text)
# ['A-1042', 'B-0077', 'A-2210']

The r before the string marks a raw string: backslashes are passed as they are to the regex engine, without being interpreted by Python. Always use it for regular expressions.

The fundamental elements

Characters and classes

Quantifiers

Anchors

^ marks the start of the string and $ the end. \b marks a word boundary. Without anchors, a regex validating a five-digit postcode, \d{5}, would also accept "abc123456xyz", because it contains five consecutive digits. With anchors, ^\d{5}$, it accepts only strings made of exactly five digits.

Groups: extracting the parts

Parentheses create groups, which let you extract portions of the matched text:

line = "2024-03-15 ERROR Database connection failed"
m = re.match(r"(\d{4})-(\d{2})-(\d{2}) (\w+) (.*)", line)
if m:
    year, month, day, level, message = m.groups()

For expressions with many groups, named groups greatly improve readability:

pattern = re.compile(
    r"(?P<date>\d{4}-\d{2}-\d{2}) (?P<level>\w+) (?P<message>.*)"
)
m = pattern.match(line)
print(m["level"])   # ERROR

re.compile prepares the pattern once only: useful when the same pattern is applied to many lines.

Replacing text

re.sub replaces every match. A practical example: normalising multiple spaces and masking phone numbers in a text to be shared.

clean = re.sub(r"\s+", " ", text).strip()
masked = re.sub(r"\b(\d{3})\d{4}(\d{3})\b", r"\1****\2", clean)

In the replacement string, \1 and \2 refer back to the contents of the groups.

Greediness: the most common mistake

The * and + quantifiers are greedy: they take as much text as possible. Looking for tags in <b>one</b> and <b>two</b> with <b>.*</b>, you get a single match running from the first <b> to the last </b>. The non-greedy version, <b>.*?</b>, stops at the first possible closing tag and correctly finds the two separate elements.

When not to use them

Regular expressions are excellent for text with a regular, flat structure: codes, dates, log lines, simple formats. They become the wrong choice in other cases:

• HTML and XML: nested structures that a regex cannot handle reliably; use a dedicated parser;
• JSON, CSV and structured formats: there are standard modules (json, csv) that correctly handle quotes, special characters and edge cases;
• full email validation: the formal rules are so complex that a realistic regex will always be approximate; a simple check plus a confirmation email is better;
• simple string operations: text.startswith("http") or "error" in line are clearer and faster than a regex.

There is also a performance risk: some expressions with nested quantifiers, such as (a+)+$, can take an enormous time on certain inputs, because the engine tries a huge number of combinations. If a regex is applied to user-supplied text, avoid these constructions and limit the input length.

Writing them readably

The re.VERBOSE flag lets you split an expression over several lines and add comments, ignoring whitespace. For non-trivial regexes it is an investment that pays off for whoever has to read them. And, as with any other code, write tests with examples that must match and others that must not.

In short

• A regular expression describes a set of strings; in Python it is used with the re module and raw strings.
• Character classes, quantifiers, anchors and groups cover most cases.
• Use the ^ and $ anchors to validate, and non-greedy quantifiers to avoid overly long matches.
• Do not use them for HTML, JSON or CSV: dedicated parsers are more reliable.
• Make them readable with named groups, re.VERBOSE and tests with concrete examples.