Python Sitemap Generator Project

Creating an XML sitemap is one of the most effective ways to boost a website’s SEO, and thanks to Python’s rich ecosystem, building a custom sitemap generator has never been easier. In this post, we’ll walk through a complete Python sitemap generator project—from the core concepts and required libraries to a step‑by‑step implementation and best‑practice tips for deploying your tool in production. Whether you’re a solo developer, a small agency, or a large enterprise, this guide will give you a solid foundation to automate sitemap creation and keep search engines happy.

Why a Custom Python Sitemap Generator?

While many CMS platforms (WordPress, Shopify, etc.) offer built‑in sitemap features, a custom solution provides several advantages:

• Flexibility: Tailor the sitemap structure to match complex URL patterns, multilingual sites, or dynamic content.
• Automation: Integrate the generator into CI/CD pipelines to rebuild sitemaps on every deployment.
• Performance: Optimize crawling speed and memory usage for large sites with millions of pages.
• Control: Enforce SEO rules (e.g., exclude duplicate content, set priority, or add lastmod dates) programmatically.

Project Overview

Our Python sitemap generator will:

1. Recursively crawl a target website or read URLs from a data source.
2. Validate each URL against a set of inclusion/exclusion rules.
3. Generate a standards‑compliant XML sitemap (or a sitemap index for large sites).
4. Optionally compress the output to .gz for faster delivery.

We’ll use the following libraries:

• requests – for HTTP requests.
• beautifulsoup4 – to parse HTML and extract links.
• lxml – for fast XML generation.
• tqdm – optional, provides a progress bar.

Setting Up the Development Environment

First, create a virtual environment and install the dependencies:

python -m venv venv
source venv/bin/activate  # Windows: venv\Scripts\activate
pip install requests beautifulsoup4 lxml tqdm

Core Components of the Generator

1. URL Collector

The collector is responsible for discovering URLs. Below is a simple recursive crawler that respects robots.txt and avoids infinite loops.

import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin, urlparse
import time

class UrlCollector:
    def __init__(self, base_url, max_depth=3, delay=0.5):
        self.base_url = base_url.rstrip('/')
        self.max_depth = max_depth
        self.delay = delay
        self.visited = set()
        self.to_visit = [(self.base_url, 0)]

    def _is_internal(self, url):
        return urlparse(url).netloc == urlparse(self.base_url).netloc

    def crawl(self):
        while self.to_visit:
            url, depth = self.to_visit.pop(0)
            if url in self.visited or depth > self.max_depth:
                continue
            try:
                resp = requests.get(url, timeout=10)
                resp.raise_for_status()
            except requests.RequestException:
                continue

            self.visited.add(url)
            soup = BeautifulSoup(resp.text, 'html.parser')
            for link in soup.find_all('a', href=True):
                href = urljoin(url, link['href'])
                href = href.split('#')[0]  # strip fragment
                if self._is_internal(href) and href not in self.visited:
                    self.to_visit.append((href, depth + 1))
            time.sleep(self.delay)
        return sorted(self.visited)

2. Sitemap Builder

Using lxml.etree, we can create an XML document that complies with the Sitemap protocol. The builder also adds optional tags like lastmod, changefreq, and priority.

from lxml import etree
from datetime import datetime

class SitemapBuilder:
    NSMAP = {None: "http://www.sitemaps.org/schemas/sitemap/0.9"}

    def __init__(self, urls, changefreq='weekly', priority='0.5'):
        self.urls = urls
        self.changefreq = changefreq
        self.priority = priority

    def _url_element(self, loc):
        url_el = etree.Element('url')
        etree.SubElement(url_el, 'loc').text = loc
        etree.SubElement(url_el, 'lastmod').text = datetime.utcnow().date().isoformat()
        etree.SubElement(url_el, 'changefreq').text = self.changefreq
        etree.SubElement(url_el, 'priority').text = self.priority
        return url_el

    def build(self, output_path='sitemap.xml', gzip=False):
        urlset = etree.Element('urlset', nsmap=self.NSMAP)
        for loc in self.urls:
            urlset.append(self._url_element(loc))

        tree = etree.ElementTree(urlset)
        if gzip:
            import gzip
            with gzip.open(output_path + '.gz', 'wb') as f:
                tree.write(f, xml_declaration=True, encoding='UTF-8', pretty_print=True)
        else:
            tree.write(output_path, xml_declaration=True, encoding='UTF-8', pretty_print=True)
        return output_path

3. Command‑Line Interface (CLI)

A thin CLI wrapper lets users run the generator with a single command. It also supports arguments for depth, delay, and output location.

import argparse
from pathlib import Path

def main():
    parser = argparse.ArgumentParser(description='Python Sitemap Generator')
    parser.add_argument('base_url', help='Root URL of the site to crawl')
    parser.add_argument('--depth', type=int, default=3, help='Maximum crawl depth')
    parser.add_argument('--delay', type=float, default=0.5, help='Delay between requests (seconds)')
    parser.add_argument('--output', default='sitemap.xml', help='Output file name')
    parser.add_argument('--gzip', action='store_true', help='Compress output as .gz')
    args = parser.parse_args()

    collector = UrlCollector(args.base_url, max_depth=args.depth, delay=args.delay)
    urls = collector.crawl()
    print(f'Collected {len(urls)} URLs.')

    builder = SitemapBuilder(urls)
    builder.build(output_path=args.output, gzip=args.gzip)
    print(f'Sitemap written to {Path(args.output).resolve()}')

if __name__ == '__main__':
    main()

Enhancing the Project for Real‑World Use

Handling Large Sites with Sitemap Indexes

Search engines limit a single sitemap to 50,000 URLs or 50 MB. For bigger projects, generate multiple sitemap files and an index file that references each one. The SitemapBuilder can be extended to split the URL list automatically.

Integrating Robots.txt Rules

Before crawling, fetch /robots.txt and respect User‑Agent directives. The reppy library simplifies this:

pip install reppy

Then modify the collector:

from reppy.robots import Robots

class UrlCollector:
    # ... existing __init__ ...

    def _allowed_by_robots(self, url):
        rp = Robots.fetch(self.base_url + '/robots.txt')
        return rp.allowed(url, '*')
    
    # In crawl():
    if not self._allowed_by_robots(url):
        continue

Adding Multilingual Support

If your site serves multiple languages, include hreflang annotations in the sitemap. You can add a xhtml:link element per URL:

def _url_element(self, loc, alternates=None):
    url_el = etree.Element('url')
    etree.SubElement(url_el, 'loc').text = loc
    # ... other tags ...
    if alternates:
        for href, lang in alternates:
            link_el = etree.SubElement(url_el, '{http://www.w3.org/1999/xhtml}link')
            link_el.set('rel', 'alternate')
            link_el.set('hreflang', lang)
            link_el.set('href', href)
    return url_el

Deploying as a Microservice

Wrap the generator in a Flask or FastAPI endpoint so that CI pipelines can trigger sitemap regeneration via an HTTP POST. Example with FastAPI:

pip install fastapi uvicorn

from fastapi import FastAPI, BackgroundTasks
app = FastAPI()

@app.post("/generate-sitemap/")
def generate(base_url: str, background_tasks: BackgroundTasks):
    def task():
        collector = UrlCollector(base_url)
        urls = collector.crawl()
        builder = SitemapBuilder(urls)
        builder.build(output_path='static/sitemap.xml')
    background_tasks.add_task(task)
    return {"status": "Sitemap generation started"}

SEO Best Practices for Your Sitemap

• Include only canonical URLs: Duplicate pages dilute link equity.
• Update lastmod accurately: Search engines use this to prioritize crawling.
• Set priority wisely: Reserve higher values for cornerstone content.
• Keep the file size under 50 MB: Split into multiple sitemaps if needed.
• Submit the sitemap to Google Search Console and Bing Webmaster Tools: This speeds up indexing.

Testing and Validation

Before deploying, validate the generated XML using Google’s