How to Extract URLs from a String in Python

Learn Python methods to extract URLs from text using regex, the urlextract library, or BeautifulSoup for HTML, with validation and normalization techniques.

Finding links embedded in text is a common task in web scraping, log analysis, and content processing. Python offers several approaches with different accuracy and complexity trade-offs.

Using Regular Expressions

A regex pattern handles most common URL formats quickly:

import re

text = "Visit https://example.com or check www.blog.org for more info"

pattern = r'https?://[^\s<>"]+|www\.[^\s<>"]+'
urls = re.findall(pattern, text)

print(urls)

For more comprehensive matching including various protocols:

import re

def extract_urls(text):
    """Extract URLs with common protocols."""
    pattern = r'''
        (?:https?|ftp)://          # Protocol
        (?:[\w-]+\.)+[\w-]+        # Domain
        (?:/[^\s<>"]*)?            # Path (optional)
        |
        www\.[\w-]+\.[\w.-]+       # www URLs
        (?:/[^\s<>"]*)?
    '''
    return re.findall(pattern, text, re.VERBOSE)

text = """
    Check https://example.com/path?query=1
    Also ftp://files.server.com/data
    And www.github.com/user/repo
"""

print(extract_urls(text))

Using urlextract Library

For production accuracy, the urlextract library handles edge cases including TLD validation:

pip install urlextract

from urlextract import URLExtract

extractor = URLExtract()

text = "Visit example.com, check github.io/path, or email test@mail.com"

urls = extractor.find_urls(text)
print(urls)

Filter out email addresses if needed:

from urlextract import URLExtract

def extract_urls_only(text):
    """Extract URLs, excluding email addresses."""
    extractor = URLExtract()
    urls = extractor.find_urls(text)
    return [url for url in urls if '@' not in url]

text = "Contact support@example.com or visit example.com"
print(extract_urls_only(text))

Extracting URLs from HTML

When working with HTML, use BeautifulSoup to extract links properly:

from bs4 import BeautifulSoup

html = """
<html>
    <a href="https://example.com">Link 1</a>
    <a href="/relative/path">Link 2</a>
    <img src="https://cdn.example.com/image.png">
</html>
"""

soup = BeautifulSoup(html, "lxml")

links = [a.get('href') for a in soup.find_all('a', href=True)]
print(links)

sources = [img.get('src') for img in soup.find_all('img', src=True)]
print(sources)

Validating Extracted URLs

Verify that extracted strings are properly formatted URLs:

from urllib.parse import urlparse

def is_valid_url(url):
    """Check if string is a valid URL with scheme and domain."""
    try:
        result = urlparse(url)
        return all([result.scheme, result.netloc])
    except Exception:
        return False

urls = [
    "https://example.com/path",
    "not-a-url",
    "ftp://files.server.com",
    "www.example.com"  # Missing scheme
]

for url in urls:
    print(f"{url}: {is_valid_url(url)}")

Output:

https://example.com/path: True
not-a-url: False
ftp://files.server.com: True
www.example.com: False

Normalizing Extracted URLs

Clean and standardize URLs after extraction:

from urllib.parse import urlparse, urlunparse

def normalize_url(url):
    """Add scheme if missing and normalize."""
    if not url.startswith(('http://', 'https://', 'ftp://')):
        url = 'https://' + url
    
    parsed = urlparse(url)
    
    normalized = parsed._replace(netloc=parsed.netloc.lower())
    
    path = normalized.path.rstrip('/') if normalized.path != '/' else '/'
    normalized = normalized._replace(path=path)
    
    return urlunparse(normalized)

urls = ["WWW.EXAMPLE.COM", "https://site.com/path/", "example.org"]
for url in urls:
    print(normalize_url(url))

Output:

https://www.example.com
https://site.com/path
https://example.org

Complete Extraction Pipeline

Combine extraction, validation, and normalization:

from urlextract import URLExtract
from urllib.parse import urlparse

def extract_and_validate(text):
    """Extract, validate, and normalize URLs from text."""
    extractor = URLExtract()
    raw_urls = extractor.find_urls(text)
    
    valid_urls = []
    for url in raw_urls:
        if '@' in url:
            continue
        
        if not url.startswith(('http://', 'https://')):
            url = 'https://' + url
        
        parsed = urlparse(url)
        if parsed.scheme and parsed.netloc:
            valid_urls.append(url)
    
    return list(set(valid_urls))  # Remove duplicates

text = """
    Visit example.com for info.
    Contact: support@company.com
    Source: https://github.com/user/repo
    Mirror: github.com/user/repo
"""

print(extract_and_validate(text))

Method Comparison

Method	Accuracy	Speed	Dependencies	Best For
Regex	⭐⭐⭐	⚡ Fast	None	Quick scripts
`urlextract`	⭐⭐⭐⭐⭐	Medium	Package	Production use
BeautifulSoup	⭐⭐⭐⭐⭐	Medium	Package	HTML content
`urlparse`	N/A	⚡ Fast	Built-in	Validation only

Summary

Use regex for quick extraction in controlled environments.
For production applications, urlextract provides accurate results with proper TLD handling.
When processing HTML, use BeautifulSoup to extract href and src attributes.
Always validate and normalize extracted URLs before using them.