How to Extract Text from HTML in Python

Learn how to use BeautifulSoup in Python to strip HTML tags and extract clean, readable text from web pages with proper handling of scripts, styles, and whitespace.

Use BeautifulSoup to strip tags and get clean, human-readable text.

1. Basic Extraction

from bs4 import BeautifulSoup

html = "<p>Hello <b>World</b>!</p>"
soup = BeautifulSoup(html, "lxml")

text = soup.get_text()
print(text) # "Hello World!"

2. Clean Extraction (Recommended)

Use separator and strip to prevent words from merging.

from bs4 import BeautifulSoup

html = "<div>Line 1</div><div>Line 2</div>"
soup = BeautifulSoup(html, "lxml")

text = soup.get_text(separator=" ", strip=True)
print(text) # "Line 1 Line 2"

3. Remove Script/Style Tags First

These tags contain non-human code. Delete them before extracting text.

for junk in soup(["script", "style"]):
    junk.decompose() # Removes from tree

clean_text = soup.get_text()

4. Target Specific Elements

Use CSS selectors to get text from a specific section.

for p in soup.select("article.body p"):
    print(p.get_text())

5. Summary

Method	Purpose
`get_text()`	All visible text
`separator=" "`	Prevent words from joining
`strip=True`	Remove whitespace
`decompose()`	Delete unwanted tags

Conclusion: Use lxml as the parser (fast) and always remove <script>/<style> before extraction.

BeautifulSoup provides reliable methods to strip HTML tags and extract clean, readable text from web pages and HTML documents.

Basic Text Extraction

The get_text() method removes all HTML tags and returns the text content:

from bs4 import BeautifulSoup

html = "<p>Hello <b>World</b>!</p>"
soup = BeautifulSoup(html, "lxml")

text = soup.get_text()
print(text)  # Hello World!

Install the required packages:

pip install beautifulsoup4 lxml

Preventing Text Concatenation

Without a separator, adjacent elements merge their text:

from bs4 import BeautifulSoup

html = "<div>First</div><div>Second</div>"
soup = BeautifulSoup(html, "lxml")

print(soup.get_text())                          # FirstSecond

print(soup.get_text(separator=" "))             # First Second

print(soup.get_text(separator=" ", strip=True)) # First Second

Removing Script and Style Content

Script and style tags contain code, not readable text. Remove them before extraction:

from bs4 import BeautifulSoup

html = """
<html>
<head>
    <style>body { color: red; }</style>
    <script>alert('hello');</script>
</head>
<body>
    <p>Actual content here.</p>
</body>
</html>
"""

soup = BeautifulSoup(html, "lxml")

for element in soup(["script", "style", "meta", "link"]):
    element.decompose()

text = soup.get_text(separator=" ", strip=True)
print(text)  # Actual content here.

Extracting from Specific Elements

Use CSS selectors to target particular sections:

from bs4 import BeautifulSoup

html = """
<article>
    <header>Article Title</header>
    <div class="content">
        <p>First paragraph.</p>
        <p>Second paragraph.</p>
    </div>
    <footer>Author info</footer>
</article>
"""

soup = BeautifulSoup(html, "lxml")

content = soup.select_one(".content")
print(content.get_text(separator=" ", strip=True))

for p in soup.select(".content p"):
    print(p.get_text())

Output:

First paragraph. Second paragraph.
First paragraph.
Second paragraph.

Complete Extraction Function

A reusable function for clean text extraction:

from bs4 import BeautifulSoup
import re

def extract_text(html, selector=None):
    """Extract clean text from HTML."""
    soup = BeautifulSoup(html, "lxml")
    
    for tag in soup(["script", "style", "meta", "link", "noscript"]):
        tag.decompose()
    
    if selector:
        element = soup.select_one(selector)
        if not element:
            return ""
        soup = element
    
    text = soup.get_text(separator=" ", strip=True)
    
    text = re.sub(r'\s+', ' ', text)
    
    return text.strip()

html = "<div><p>Hello   World</p><script>code();</script></div>"
print(extract_text(html))  # Hello World

Extracting Text with Structure

Preserve some structure by handling specific tags:

from bs4 import BeautifulSoup

def extract_with_newlines(html):
    """Extract text preserving paragraph breaks."""
    soup = BeautifulSoup(html, "lxml")
    
    for tag in soup(["script", "style"]):
        tag.decompose()
    
    for br in soup.find_all("br"):
        br.replace_with("\n")
    
    for tag in soup.find_all(["p", "div", "h1", "h2", "h3", "li"]):
        tag.insert_after("\n")
    
    text = soup.get_text()
    
    lines = [line.strip() for line in text.splitlines()]
    return "\n".join(line for line in lines if line)

html = """
<h1>Title</h1>
<p>First paragraph.</p>
<p>Second paragraph.</p>
<ul>
    <li>Item one</li>
    <li>Item two</li>
</ul>
"""

print(extract_with_newlines(html))

Output:

Title
First paragraph.
Second paragraph.
Item one
Item two

Handling Encoding Issues

Specify encoding for proper character handling:

from bs4 import BeautifulSoup

html_bytes = b"<p>Caf\xe9</p>"
soup = BeautifulSoup(html_bytes, "lxml", from_encoding="latin-1")
print(soup.get_text())  # Café

with open("page.html", "r", encoding="utf-8") as f:
    soup = BeautifulSoup(f, "lxml")
    text = soup.get_text(separator=" ", strip=True)

Parser Comparison

Parser	Speed	Lenient	Install
`lxml`	⚡ Fast	✅ Yes	`pip install lxml`
`html.parser`	Medium	✅ Yes	Built-in
`html5lib`	🐢 Slow	✅✅ Very	`pip install html5lib`

Summary

Use BeautifulSoup's get_text() with separator=" " and strip=True for clean extraction.
Always remove <script> and <style> tags first to avoid including code in your output.
For targeted extraction, use CSS selectors to focus on specific content areas.