Blog

  • Python Beautifulsoup Web Scraping Beginner Guide

    Welcome to the ultimate Python BeautifulSoup web scraping beginner guide. Whether you’re a data enthusiast, a marketer hunting for competitor insights, or just curious about pulling information from the web, this tutorial will walk you through everything you need to start scraping HTML pages with Python’s most popular parsing library. By the end of this article you’ll understand how to install BeautifulSoup, fetch pages safely, extract data efficiently, and avoid common pitfalls that trip up newcomers.

    Why Choose BeautifulSoup for Web Scraping?

    BeautifulSoup is a Python library designed to parse HTML and XML documents. It shines in the following areas:

    • Ease of use: Intuitive API that feels natural for Python developers.
    • Robustness: Handles malformed HTML gracefully, a common issue on the live web.
    • Flexibility: Works well with other tools like requests, urllib, and headless browsers such as Selenium.
    • Community support: Extensive documentation and a vibrant community that continuously contributes plugins and examples.

    Combined with Python’s simplicity, BeautifulSoup makes it possible to turn a messy webpage into clean, structured data in just a few lines of code.

    Setting Up Your Environment

    1. Install Python (if you haven’t already)

    Download the latest stable version of Python from python.org and follow the installer instructions. Ensure the python command is available in your system’s PATH.

    2. Install Required Packages

    Open a terminal or command prompt and run the following command to install requests (for HTTP requests) and beautifulsoup4 (the library itself):

    pip install requests beautifulsoup4

    If you plan to scrape JavaScript‑heavy sites, consider adding lxml for faster parsing or selenium for browser automation:

    pip install lxml selenium

    Fetching a Web Page with requests

    The first step in any scraping project is to retrieve the raw HTML. Below is a minimal example that demonstrates how to request a page and check for a successful response:

    import requests
    
    url = "https://example.com"
    response = requests.get(url, headers={"User-Agent": "Mozilla/5.0"})
    if response.status_code == 200:
        html = response.text
    else:
        raise Exception(f"Failed to fetch page: {response.status_code}")

    Notice the custom User-Agent header—many websites block generic Python agents, so mimicking a real browser improves your chances of getting the content.

    Parsing HTML with BeautifulSoup

    Creating the Soup Object

    Once you have the HTML string, pass it to BeautifulSoup along with a parser. The built‑in html.parser works everywhere, while lxml is faster and more lenient:

    from bs4 import BeautifulSoup
    
    soup = BeautifulSoup(html, "html.parser")   # or "lxml"

    Finding Elements

    BeautifulSoup offers several methods to locate tags:

    • find(tag, attrs) – returns the first matching element.
    • find_all(tag, attrs) – returns a list of all matches.
    • select(css_selector) – uses CSS selectors for powerful queries.

    Example: extracting all headlines from a news site:

    headlines = soup.find_all("h2", class_="headline")
    for h in headlines:
        print(h.get_text(strip=True))

    Practical Example: Scraping a Product List

    Let’s walk through a realistic scenario—scraping product names, prices, and URLs from a fictional e‑commerce category page.

    import requests
    from bs4 import BeautifulSoup
    
    def scrape_category(url):
        response = requests.get(url, headers={"User-Agent": "Mozilla/5.0"})
        soup = BeautifulSoup(response.text, "html.parser")
        
        products = []
        for item in soup.select("div.product-card"):
            name = item.select_one("h3.product-name").get_text(strip=True)
            price = item.select_one("span.price").get_text(strip=True)
            link = item.select_one("a")["href"]
            products.append({"name": name, "price": price, "url": link})
        return products
    
    category_url = "https://example-store.com/category/widgets"
    data = scrape_category(category_url)
    for p in data:
        print(p)

    This snippet demonstrates:

    • Using select with CSS selectors for concise queries.
    • Extracting text via get_text(strip=True) to remove extra whitespace.
    • Accessing attribute values (e.g., href) directly from the tag.

    Handling Pagination

    Most real‑world sites spread data across multiple pages. A typical pagination loop looks like this:

    def scrape_all_pages(start_url):
        page_url = start_url
        all_items = []
        
        while page_url:
            print(f"Scraping {page_url}")
            response = requests.get(page_url, headers={"User-Agent": "Mozilla/5.0"})
            soup = BeautifulSoup(response.text, "html.parser")
            
            # Extract items on the current page
            all_items.extend(parse_items(soup))
            
            # Find the "next" button/link
            next_link = soup.select_one("a.next")
            page_url = next_link["href"] if next_link else None
        
        return all_items
    

    Make sure the href you retrieve is an absolute URL; if it’s relative, prepend the base domain using urllib.parse.urljoin.

    Best Practices for Ethical and Effective Scraping

    • Respect robots.txt: Before scraping, check the site’s /robots.txt file to see which sections are disallowed for bots.
    • Throttle your requests: Insert a short time.sleep() (e.g., 1–2 seconds) between calls to avoid overloading the server.
    • Use rotating User‑Agents and proxies: For larger projects, rotate headers and IP addresses to reduce the chance of being blocked.
    • Handle errors gracefully: Implement retries with exponential back‑off for transient network issues.
    • Store data responsibly: Save results in CSV, JSON, or a database, and always comply with the site’s terms of service.

    Common Pitfalls and How to Fix Them

    1. Empty Results Due to Dynamic Content

    If a page loads data via JavaScript after the initial HTML, BeautifulSoup won’t see it. Solutions:

    • Inspect the network tab in your browser to find the underlying API endpoint and call it directly with requests.
    • Use Selenium or Playwright to render the page before passing the final HTML to BeautifulSoup.

    2. Unexpected HTML Structure Changes

    Websites frequently update their markup. To make your scraper resilient:

    • Prefer CSS classes or data attributes that are less likely to change.
    • Add fallback selectors and log warnings when a tag isn’t found.

    3. Getting Banned

    Repeated requests from a single IP can trigger anti‑scraping mechanisms. Mitigate this by:

    • Randomizing request intervals.
    • Using a pool of residential proxies.
    • Limiting the total number of pages scraped per session.

    Putting It All Together – A Mini Project

    Below is a compact script that incorporates the concepts discussed: installing libraries, fetching pages, handling pagination, extracting data, and saving the output to a CSV file.

    import csv
    import time
    import requests
    from bs4 import BeautifulSoup
    from urllib.parse import urljoin

    BASE_URL = "https://books.toscrape.com/"

    def get_soup(url):
    resp = requests.get(url, headers={"User-Agent": "Mozilla/5.0"})
    resp.raise_for_status()
    return BeautifulSoup(resp.text, "html.parser")

    def parse_books(soup):
    books = []
    for article in soup.select("article.product_pod"):
    title = article.select_one("h3 a")["title"]
    price = article.select_one("p.price_color").text
    link = urljoin(BASE_URL, article.select_one("h3 a")["href"])
    books.append({"title": title, "price": price, "url": link})
    return books

    def scrape_all():
    page = "catalogue/page-1.html"
    all_books = []
    while page:
    full_url = urljoin(BASE_URL, page)
    soup = get_soup(full_url)
    all_books.extend(parse_books(soup))
    next_btn = soup.select_one("li.next a")
    page = next_btn["href"] if next_btn else None
    time.sleep(1) # polite delay
    return all_books

    def save_to_csv(data, filename="books.csv"):
    keys = data[0].keys()
    with open(filename, "w", newline="", encoding="utf-8") as f:
    writer = csv.DictWriter(f, fieldnames

  • Python Selenium Web Driver Complete Tutorial

    Welcome to the ultimate Python Selenium WebDriver complete tutorial! Whether you’re a beginner eager to automate your first browser test or an experienced QA engineer looking to sharpen your skills, this guide covers everything you need—from installation to advanced best practices. By the end of this article, you’ll be able to write robust, maintainable Selenium scripts in Python that can handle dynamic web pages, multiple browsers, and real‑world test scenarios.

    Why Choose Selenium with Python?

    Selenium is the de‑facto standard for web automation, and Python’s clean syntax makes it an ideal partner. Together they provide:

    • Cross‑browser support: Chrome, Firefox, Edge, Safari, and headless browsers.
    • Rich ecosystem: Integration with pytest, unittest, and CI/CD pipelines.
    • Ease of learning: Python’s readability speeds up script development.
    • Community power: Thousands of tutorials, plugins, and Stack Overflow answers.

    Prerequisites

    Before diving in, make sure you have the following installed on your machine:

    • Python 3.8+ (download from python.org)
    • pip (Python package manager, included with Python)
    • A supported web browser (Chrome, Firefox, Edge, etc.)
    • Corresponding browser driver (ChromeDriver, GeckoDriver, etc.)

    Step 1: Installing Selenium

    Open a terminal or command prompt and run the pip command:

    pip install selenium

    This installs the latest stable Selenium 4.x release, which includes the new webdriver manager and enhanced relative locators.

    Step 2: Setting Up the WebDriver

    Selenium requires a driver executable that acts as a bridge between Python code and the browser. You can manage drivers manually or let Selenium handle them automatically.

    2.1 Manual Driver Management

    1. Download the driver that matches your browser version:
    2. Extract the executable to a folder on your system PATH or note the full path.
    3. Instantiate the driver in your script, for example:
    from selenium import webdriver
    
    driver = webdriver.Chrome(executable_path="/path/to/chromedriver")
    driver.get("https://www.example.com")

    2.2 Automatic Driver Management (Recommended)

    Selenium 4 introduces webdriver-manager which downloads the correct driver on the fly.

    pip install webdriver-manager

    Then use it like this:

    from selenium import webdriver
    from webdriver_manager.chrome import ChromeDriverManager
    
    driver = webdriver.Chrome(ChromeDriverManager().install())
    driver.get("https://www.example.com")

    This approach eliminates version mismatches and works across CI environments.

    Basic Selenium Commands

    Once the driver is ready, you can perform a variety of actions. Below are the most frequently used commands.

    • Navigate: driver.get(url), driver.back(), driver.forward(), driver.refresh()
    • Window Management: driver.maximize_window(), driver.set_window_size(width, height)
    • Cookies: driver.get_cookies(), driver.add_cookie(dict), driver.delete_all_cookies()
    • Quit: driver.quit() (closes all windows and ends the session)

    Locating Elements

    Finding the right element is the cornerstone of any Selenium script. Selenium 4 supports several locator strategies:

    • By.ID
    • By.NAME
    • By.XPATH
    • By.CSS_SELECTOR
    • By.CLASS_NAME
    • By.TAG_NAME
    • By.LINK_TEXT / By.PARTIAL_LINK_TEXT

    Example using By.CSS_SELECTOR:

    from selenium.webdriver.common.by import By
    
    search_box = driver.find_element(By.CSS_SELECTOR, "input[name='q']")
    search_box.send_keys("Python Selenium tutorial")
    search_box.submit()

    Relative Locators (New in Selenium 4)

    Relative locators let you find elements based on their position to other known elements.

    from selenium.webdriver.common.by import By
    from selenium.webdriver.support.relative_locator import locate_with
    
    # Find an element below the logo
    below_logo = driver.find_element(locate_with(By.TAG_NAME, "input").below({"id": "logo"}))
    

    Explicit and Implicit Waits

    Web pages often load content asynchronously. Relying on time.sleep() makes tests flaky. Selenium provides two robust waiting mechanisms.

    Implicit Wait

    Sets a global timeout for all find_element calls.

    driver.implicitly_wait(10)  # seconds

    Explicit Wait

    Waits for a specific condition before proceeding.

    from selenium.webdriver.support.ui import WebDriverWait
    from selenium.webdriver.support import expected_conditions as EC
    
    wait = WebDriverWait(driver, 15)
    login_button = wait.until(EC.element_to_be_clickable((By.ID, "login")))
    login_button.click()

    Handling Common Web Elements

    2.1 Alerts and Pop‑ups

    alert = driver.switch_to.alert
    alert.accept()          # Click OK
    # or
    alert.dismiss()         # Click Cancel
    # or send text
    alert.send_keys("Sample input")
    

    2.2 Frames and iFrames

    # Switch to a frame by name or index
    driver.switch_to.frame("frameName")
    # Perform actions inside the frame
    driver.find_element(By.TAG_NAME, "button").click()
    # Return to the default content
    driver.switch_to.default_content()
    

    2.3 Multiple Windows/Tabs

    # Open a new tab
    driver.execute_script("window.open('https://www.google.com');")
    # Switch to the new window
    driver.switch_to.window(driver.window_handles[1])
    # Do something
    driver.title
    # Close and return
    driver.close()
    driver.switch_to.window(driver.window_handles[0])
    

    2.4 Taking Screenshots

    # Full page screenshot (available in recent Selenium versions)
    driver.save_screenshot("homepage.png")
    
    # Element screenshot
    logo = driver.find_element(By.ID, "logo")
    logo.screenshot("logo.png")
    

    Organizing Tests with the Page Object Model (POM)

    The Page Object Model separates page structure from test logic, making scripts easier to maintain.

    # file: pages/login_page.py
    from selenium.webdriver.common.by import By
    
    class LoginPage:
        def __init__(self, driver):
            self.driver = driver
            self.username_input = (By.ID, "username")
            self.password_input = (By.ID, "password")
            self.login_btn = (By.CSS_SELECTOR, "button.login")
    
        def load(self):
            self.driver.get("https://example.com/login")
    
        def login(self, user, pwd):
            self.driver.find_element(*self.username_input).send_keys(user)
            self.driver.find_element(*self.password_input).send_keys(pwd)
            self.driver.find_element(*self.login_btn).click()
    

    Test script using the page object:

    # file: tests/test_login.py
    import unittest
    from selenium import webdriver
    from webdriver_manager.chrome import ChromeDriverManager
    from pages.login_page import LoginPage
    
    class LoginTest(unittest.TestCase):
        def setUp(self):
            self.driver = webdriver.Chrome(ChromeDriverManager().install())
            self.login_page = LoginPage(self.driver)
    
        def test_valid_login(self):
            self.login_page.load()
            self.login_page.login("admin", "secret")
            self.assertIn("Dashboard", self.driver.title)
    
        def tearDown(self):
            self.driver.quit()
    
    if __name__ == "__main__":
        unittest.main()
    

    Best Practices for Stable Selenium Scripts

    • Use explicit waits instead of static sleeps.
    • Prefer CSS selectors over complex XPaths for speed and readability.
    • Encapsulate locators in page objects or separate constants.
    • Run tests in headless mode on CI servers to save resources:
      options = webdriver.ChromeOptions()
      options.add_argument("--headless")
      driver = webdriver.Chrome(ChromeDriverManager().install(), options=options)
    • Parameterize data using fixtures (pytest) or data‑driven approaches.
    • Log actions with Python’s logging module for easier debugging.

    Common Errors and How to Fix Them

    Error Typical Cause Solution
    SessionNotCreatedException Driver version mismatch with browser Update driver via webdriver-manager
  • Python Playwright Web Automation Guide

    Looking to super‑charge your web testing or data‑scraping projects with a modern, reliable tool? Python Playwright offers a fast, cross‑browser automation library that works with Chromium, Firefox, and WebKit—all from a clean, Pythonic API. In this guide we’ll walk through everything you need to know to get started, write robust scripts, and integrate Playwright into larger test suites. Whether you’re a beginner or an experienced QA engineer, you’ll find practical examples, best‑practice tips, and SEO‑friendly keywords to help your content rank higher in search results.

    What Is Playwright and Why Choose It for Python?

    Playwright was created by the same team behind Microsoft Edge and quickly became a favorite for developers who need:

    • Cross‑browser support: One script runs on Chromium, Firefox, and WebKit.
    • Auto‑waiting: Built‑in smart waits eliminate flaky tests.
    • Rich selector engine: CSS, text, XPath, and custom selectors.
    • Headless and headed modes: Perfect for CI pipelines or interactive debugging.

    Compared to Selenium, Playwright offers faster execution, better handling of modern web features (like Shadow DOM), and a more intuitive async API—making it a top choice for Python developers.

    Installing Playwright for Python

    Step‑by‑step installation

    pip install playwright
    # Install the browsers (Chromium, Firefox, WebKit)
    python -m playwright install
    

    For projects that use asyncio, you’ll also want the async version of the library:

    pip install playwright[async]
    

    If you prefer a virtual environment (highly recommended), set it up before installing:

    python -m venv venv
    source venv/bin/activate  # On Windows: venv\Scripts\activate
    pip install --upgrade pip
    

    Writing Your First Playwright Script

    Synchronous example

    from playwright.sync_api import sync_playwright
    
    def run():
        with sync_playwright() as p:
            # Launch Chromium in headless mode
            browser = p.chromium.launch(headless=True)
            page = browser.new_page()
            # Navigate to a website
            page.goto("https://example.com")
            # Capture a screenshot
            page.screenshot(path="example.png")
            # Extract the page title
            print("Page title:", page.title())
            browser.close()
    
    if __name__ == "__main__":
        run()
    

    Asynchronous version

    import asyncio
    from playwright.async_api import async_playwright
    
    async def main():
        async with async_playwright() as p:
            browser = await p.firefox.launch(headless=False)
            page = await browser.new_page()
            await page.goto("https://news.ycombinator.com")
            await page.screenshot(path="hn.png")
            print("Title:", await page.title())
            await browser.close()
    
    asyncio.run(main())
    

    Both snippets demonstrate the core workflow: launch → new_page → goto → actions → close. The auto‑waiting feature means you rarely need explicit time.sleep() calls.

    Understanding Selectors in Playwright

    Playwright’s selector engine is flexible. Here are the most common patterns:

    • page.click("button#submit") – CSS selector.
    • page.fill("input[name='q']", "Playwright") – Fill a text field.
    • page.click("text=Login") – Text selector (matches visible text).
    • page.locator("xpath=//div[@class='price']") – XPath.
    • page.locator("data-test-id=nav-item") – Custom attribute selector.

    When dealing with Shadow DOM or iframes, use the shadow and frame helpers:

    # Click a button inside a shadow root
    page.locator("css=custom-element").shadow_root().click("button")
    # Switch to an iframe
    frame = page.frame(name="payment-frame")
    await frame.fill("input#card-number", "4242424242424242")
    

    Smart Waiting and Assertions

    One of Playwright’s biggest advantages is its built‑in waiting mechanism. Instead of manually adding delays, you can rely on:

    • page.wait_for_selector() – Waits until an element appears.
    • page.wait_for_load_state() – Waits for load, domcontentloaded, or networkidle.
    • expect(page).to_have_url() – Assertion that automatically waits.
    from playwright.sync_api import expect
    
    page.goto("https://example.com/login")
    page.fill("input#email", "user@example.com")
    page.click("button#next")
    # Wait until the password field is visible
    page.wait_for_selector("input#password")
    page.fill("input#password", "Secret123")
    page.click("button#login")
    # Verify successful navigation
    expect(page).to_have_url("**/dashboard")
    

    These helpers dramatically reduce flaky test failures caused by timing issues.

    Capturing Screenshots, PDFs, and Video

    Visual artifacts are essential for debugging and reporting. Playwright supports:

    • Screenshots – Full page or element specific.
    • PDF generation – Ideal for invoices or reports.
    • Video recording – Capture the entire test run.
    # Full‑page screenshot
    page.screenshot(path="full.png", full_page=True)
    
    # Element screenshot
    logo = page.locator("img[alt='Logo']")
    logo.screenshot(path="logo.png")
    
    # Save page as PDF (Chromium only)
    page.pdf(path="report.pdf", format="A4")
    
    # Record video (needs context option)
    context = browser.new_context(record_video_dir="videos/")
    page = context.new_page()
    page.goto("https://example.com")
    page.wait_for_timeout(2000)  # Let the page settle
    context.close()  # Video saved automatically
    

    Handling Multiple Pages, Tabs, and Pop‑ups

    Modern web apps frequently open new tabs or windows. Playwright makes it easy to manage them:

    # Listen for a new page (tab) event
    def on_new_page(page):
        print("New page URL:", page.url)
    
    context = browser.new_context()
    context.on("page", on_new_page)
    
    page = context.new_page()
    page.goto("https://example.com")
    page.click("a[target='_blank']")  # Opens a new tab
    # The callback prints the URL of the newly opened page
    

    For pop‑ups like alerts, confirmations, or prompts, use the page.on("dialog") handler:

    def handle_dialog(dialog):
        print("Dialog message:", dialog.message)
        dialog.accept()  # Or dialog.dismiss()
    
    page.on("dialog", handle_dialog)
    page.click("button#delete")  # Triggers a confirm dialog
    

    Integrating Playwright with Test Frameworks

    Most teams combine Playwright with pytest for a clean test runner experience. Below is a minimal conftest.py that provides a browser fixture:

    # conftest.py
    import pytest
    from playwright.sync_api import sync_playwright
    
    @pytest.fixture(scope="session")
    def browser():
        with sync_playwright() as p:
            browser = p.chromium.launch(headless=False)
            yield browser
            browser.close()
    
    # test_login.py
    def test_successful_login(browser):
        page = browser.new_page()
        page.goto("https://example.com/login")
        page.fill("input#email", "user@example.com")
        page.fill("input#password", "Secret123")
        page.click("button#login")
        expect(page).to_have_url("**/dashboard")
        page.close()
    

    For async projects, replace sync_playwright with async_playwright and use pytest-asyncio to handle the event loop.

    Best Practices and Performance Tips

    • Reuse browser contexts instead of launching a new browser for each test. Contexts are lightweight and share the same executable.
    • Enable tracing for detailed logs: browser.start_tracing(...) and browser.stop_tracing(path="trace.zip").
    • Parallel execution: Use pytest-xdist or Playwright’s built‑in test runner to run multiple workers.
    • Keep selectors stable: Prefer data attributes (e.g., data-test-id) over volatile CSS classes.
    • Limit network traffic in CI by mocking or blocking unnecessary resources with page.route().
    # Block images and fonts to speed up tests
    page.route("**/*.{png,jpg,svg,woff,woff2}", lambda route: route.abort())
    

    Debugging Playwright Scripts

    If a test fails, these tools can help you pinpoint the issue:

    1. Playwright Inspector: Run pwdebug or set PWDEBUG=1 to launch an interactive UI.
    2. Trace Viewer: Open the .zip
  • Python Server-Sent Events Sse Web App

    Real‑time data streams have become a cornerstone of modern web applications, from live dashboards and chat rooms to collaborative editing tools. While WebSockets often steal the spotlight, Server‑Sent Events (SSE) provide a simpler, standards‑based alternative that works flawlessly with the HTTP/1.1 protocol. In this article we’ll explore how to build a robust Python SSE web app, dive into the underlying mechanics, compare SSE with other push technologies, and walk through a complete implementation using popular Python frameworks such as Flask and FastAPI. Whether you’re a seasoned Python developer or just starting out, you’ll walk away with a production‑ready pattern for delivering low‑latency, one‑way updates to browsers without the overhead of full‑duplex sockets.

    What Are Server‑Sent Events (SSE)?

    Server‑Sent Events, defined by the HTML Living Standard, enable a server to push a continuous stream of text‑based messages to a client over a single HTTP connection. The client opens the connection with a simple EventSource object, and the server responds with a text/event-stream MIME type, sending data in the following line‑based format:

    data: First line of the message
    data: Second line (optional)
    id: 42
    event: customEvent
    retry: 3000
    
    

    Key characteristics of SSE include:

    • Unidirectional flow: Server → client only; the client cannot send data over the same channel.
    • Automatic reconnection: Browsers automatically retry the connection after a configurable timeout.
    • Built‑in event IDs: Allows the client to resume from the last received message after a disconnect.
    • Text‑only payload: Perfect for JSON, CSV, or plain‑text updates.

    Why Choose SSE Over WebSockets for Python Apps?

    Both SSE and WebSockets deliver real‑time updates, but each shines in different scenarios. Below is a quick comparison to help you decide which fits your Python project best.

    Advantages of SSE

    • Simplicity: No need to manage a separate protocol handshake; a regular HTTP GET request suffices.
    • Compatibility with HTTP proxies and firewalls: Since SSE uses standard HTTP, it traverses corporate networks more reliably.
    • Built‑in reconnection logic: Reduces client‑side code complexity.
    • Lower server resource usage: One‑way streaming consumes less CPU than full‑duplex WebSocket handling.

    When WebSockets Might Be Better

    • Bidirectional communication (client needs to send frequent messages).
    • Binary data transfer (e.g., video streams, file uploads).
    • Very high message rates where low latency is critical.

    Setting Up a Python SSE Server with Flask

    Flask’s lightweight nature makes it an excellent choice for a quick SSE prototype. Below is a step‑by‑step guide to creating a basic SSE endpoint that streams random numbers every second.

    1. Install Required Packages

    pip install Flask
    

    2. Create the Flask Application

    from flask import Flask, Response, stream_with_context
    import time
    import random
    import json
    
    app = Flask(__name__)
    
    def generate_events():
        """Yield an SSE‑formatted message every second."""
        while True:
            data = {
                "timestamp": time.time(),
                "value": random.random()
            }
            # SSE format: 'data: \n\n'
            yield f"data: {json.dumps(data)}\n\n"
            time.sleep(1)
    
    @app.route('/stream')
    def stream():
        # Set the correct MIME type for SSE
        return Response(
            stream_with_context(generate_events()),
            mimetype='text/event-stream'
        )
    
    if __name__ == '__main__':
        app.run(debug=True, threaded=True)
    

    Key points to note:

    • stream_with_context ensures Flask’s request context stays alive for the duration of the stream.
    • Setting threaded=True (or using a production WSGI server like gunicorn) allows concurrent handling of multiple SSE connections.

    Client‑Side Implementation with JavaScript

    On the browser side, the native EventSource API handles everything from connection establishment to automatic reconnection. Below is a minimal HTML page that consumes the Flask SSE endpoint.

    <!DOCTYPE html>
    <html lang="en">
    <head>
        <meta charset="UTF-8">
        <title>Python SSE Demo</title>
    </head>
    <body>
        <h2>Live Random Numbers</h2>
        <pre id="log"></pre>
    
        <script>
            const log = document.getElementById('log');
            const source = new EventSource('/stream');
    
            source.onmessage = function(event) {
                const data = JSON.parse(event.data);
                const line = `⏱ ${new Date(data.timestamp * 1000).toLocaleTimeString()} – ${data.value.toFixed(4)}`;
                log.textContent += line + '\n';
            };
    
            source.onerror = function(err) {
                console.error('SSE error', err);
                // The browser will automatically try to reconnect.
            };
        </script>
    </body>
    </html>
    

    Handling Reconnection, Heartbeats, and Event IDs

    While browsers automatically retry a dropped connection, you can fine‑tune the experience:

    • Heartbeat comments: Send a comment line (:) every few seconds to keep intermediate proxies from timing out.
    • Custom retry field: Control the reconnection delay (in milliseconds) from the server side.
    • Event IDs: Include an id field so the client can resume after a disconnect without missing messages.

    Example generator with heartbeat and ID support:

    def generate_events():
        counter = 0
        while True:
            counter += 1
            payload = {"seq": counter, "value": random.random()}
            yield f"id: {counter}\n"
            yield f"data: {json.dumps(payload)}\n\n"
            # Send a comment as a heartbeat every 15 seconds
            if counter % 15 == 0:
                yield ": heartbeat\n\n"
            time.sleep(1)
    

    Best Practices and Performance Tips

    To ensure your SSE web app scales and remains reliable, follow these proven guidelines:

    1. Use a production‑grade ASGI server: For high concurrency, run Flask with gunicorn -k gevent or switch to an ASGI framework like FastAPI paired with uvicorn.
    2. Enable HTTP/2 or HTTP/3 if available: Modern browsers can multiplex multiple streams over a single connection, reducing latency.
    3. Compress payloads: Enable gzip or brotli compression for JSON messages to cut bandwidth usage.
    4. Limit message size: Keep each SSE chunk under a few kilobytes; large payloads can cause buffering delays.
    5. Graceful shutdown: Detect request.is_json or werkzeug.server.shutdown signals to close streams cleanly.
    6. Monitor connection count: Use server metrics (e.g., Prometheus) to track open SSE connections and set alerts before hitting OS limits.

    Common Pitfalls and Debugging Strategies

    Even a simple SSE setup can run into tricky issues. Here are the most frequent problems and how to resolve them:

    • Missing Content-Type header: Browsers ignore streams that aren’t declared as text/event-stream. Verify with browser dev tools → Network.
    • Proxy buffering: Some reverse proxies (e.g., Nginx) buffer responses by default. Add proxy_buffering off; to the location block.
    • Connection timeouts: If you see “ERR_CONNECTION_RESET”, send periodic comment heartbeats or adjust proxy_read_timeout.
    • CORS restrictions: When serving SSE from a different domain, include Access-Control-Allow-Origin: * (or a specific origin) in the response headers.
    • Client‑side memory leak: Continuously appending to the DOM can degrade performance. Use a rotating buffer or virtualized list for long‑running streams.

    Extending SSE to Django and FastAPI

    If Flask isn’t your primary framework, you can still leverage SSE with minimal changes.

    Django Example (using StreamingHttpResponse)

    from django.http import StreamingHttpResponse
    import json, time, random

    def event_stream():
    while True:
    data = {"time": time.time(), "value": random.random()}
    yield f"data: {json.dumps(data)}\n\n"
    time.sleep(

  • Python Https Local Development Setup

    Developing web applications with Python often means testing features that rely on secure connections—think OAuth callbacks, payment gateways, or strict Content‑Security‑Policy rules. While production environments run behind trusted SSL/TLS certificates, local development can feel clunky without a proper HTTPS setup. This guide walks you through a complete, SEO‑friendly workflow for creating a Python HTTPS local development environment, from generating a self‑signed certificate to configuring popular frameworks like Flask and Django. By the end, you’ll be able to serve https://localhost with confidence, avoid mixed‑content warnings, and keep your development pipeline smooth and secure.

    Why Use HTTPS in Local Development?

    Modern browsers and many third‑party APIs enforce HTTPS for security reasons. Running your app over plain HTTP during development can lead to:

    • Mixed‑content errors when loading resources over https from a http page.
    • Blocked authentication flows (e.g., Google, Facebook) that require a secure redirect URI.
    • Inaccurate performance testing, as HTTP/2 and TLS optimizations are missing.

    Setting up HTTPS locally mirrors production conditions, catching bugs early and ensuring a smoother deployment.

    Prerequisites

    Before diving into the setup, make sure you have the following installed on your workstation:

    1. Python 3.8+ (recommended latest stable release).
    2. pip and virtualenv or venv.
    3. OpenSSL (bundled with macOS and most Linux distros; Windows users can install Win32 OpenSSL).
    4. At least one Python web framework (Flask, Django, FastAPI, etc.).

    Step 1: Create a Self‑Signed Certificate

    A self‑signed certificate is sufficient for local development because you control the trust store on your machine. Follow these commands in a terminal:

    # Create a directory to store the cert and key
    mkdir -p ./certs
    cd ./certs
    
    # Generate a private key (2048‑bit RSA)
    openssl genrsa -out localhost.key 2048
    
    # Create a certificate signing request (CSR)
    openssl req -new -key localhost.key -out localhost.csr \
        -subj "/C=US/ST=California/L=SanFrancisco/O=MyDevOrg/OU=Dev/CN=localhost"
    
    # Sign the CSR with the private key to produce a self‑signed cert (valid for 365 days)
    openssl x509 -req -days 365 -in localhost.csr -signkey localhost.key -out localhost.crt
    

    After execution, you’ll have localhost.key and localhost.crt in the certs folder.

    Trust the Certificate on Your Machine

    Browsers will flag a self‑signed cert as untrusted until you add it to the system’s trust store.

    • macOS: Double‑click localhost.crt → “Add to Keychain” → set “Always Trust”.
    • Windows: Run certmgr.msc, import the cert into “Trusted Root Certification Authorities”.
    • Linux (Ubuntu/Debian): sudo cp localhost.crt /usr/local/share/ca-certificates/ && sudo update-ca-certificates.

    Step 2: Configure Your Python Framework

    Flask

    Flask’s built‑in development server can serve HTTPS directly. Install Flask if you haven’t already:

    python -m venv venv
    source venv/bin/activate   # Windows: venv\Scripts\activate
    pip install Flask
    

    Then create app.py:

    from flask import Flask
    
    app = Flask(__name__)
    
    @app.route('/')
    def index():
        return "Hello, secure Flask!"
    
    if __name__ == '__main__':
        # Provide the paths to the cert and key files
        app.run(host='127.0.0.1', port=5000, ssl_context=('certs/localhost.crt', 'certs/localhost.key'))
    

    Run the app with python app.py and visit https://localhost:5000. The browser should show a secure lock after you’ve trusted the certificate.

    Django

    Django’s development server also supports SSL via the runsslserver package. Install it first:

    pip install django-runsslserver
    

    Update settings.py to include the app:

    INSTALLED_APPS = [
        # ... other apps ...
        'django_runsslserver',
    ]
    

    Start the server with the certificate paths:

    python manage.py runsslserver 127.0.0.1:8000 \
        --certificate certs/localhost.crt \
        --key certs/localhost.key
    

    Navigate to https://localhost:8000 and you’ll see your Django site under HTTPS.

    FastAPI / Starlette (Uvicorn)

    FastAPI apps are typically served with uvicorn. Pass the certificate arguments when launching:

    pip install fastapi uvicorn
    uvicorn main:app --host 127.0.0.1 --port 8443 \
        --ssl-keyfile certs/localhost.key \
        --ssl-certfile certs/localhost.crt
    

    Replace main:app with the appropriate module reference for your project.

    Step 3: Automate the Setup with a Helper Script

    Repeating the same commands can be tedious. Create a small Python or Bash script to spin up the server with HTTPS in one step.

    Example Bash Wrapper

    #!/usr/bin/env bash
    # dev_https.sh – start a Python web server with HTTPS
    
    # Detect the framework from a simple argument
    FRAMEWORK=$1   # flask|django|fastapi
    
    case "$FRAMEWORK" in
      flask)
        python app.py
        ;;
      django)
        python manage.py runsslserver 127.0.0.1:8000 \
          --certificate certs/localhost.crt \
          --key certs/localhost.key
        ;;
      fastapi)
        uvicorn main:app --host 127.0.0.1 --port 8443 \
          --ssl-keyfile certs/localhost.key \
          --ssl-certfile certs/localhost.crt
        ;;
      *)
        echo "Usage: $0 {flask|django|fastapi}"
        exit 1
        ;;
    esac
    

    Make it executable (chmod +x dev_https.sh) and run ./dev_https.sh flask to launch a Flask app over HTTPS instantly.

    Testing Your HTTPS Setup

    After starting the server, verify that the connection is truly secure:

    • Open Chrome’s DevTools → Security tab. It should display “Connection is secure”.
    • Run curl -k https://localhost:5000 (replace the port). The -k flag ignores certificate validation, useful for quick checks.
    • Use Qualys SSL Labs’s “Test your server” feature with localhost only if you expose it via a tunnel (e.g., ngrok).

    Common Pitfalls & How to Fix Them

    • Browser still shows “Not Secure”. Double‑check that you added the certificate to the correct trust store and restarted the browser.
    • Port conflicts. Ensure the chosen port (5000, 8000, 8443, etc.) isn’t already in use. Use lsof -i :PORT to diagnose.
    • Mixed‑content warnings. All assets (CSS, JS, images) must be loaded via https://. Update any hard‑coded http:// URLs in templates or static files.
    • Certificate expiration. Self‑signed certs default to 365 days. Regenerate them before they expire to avoid sudden warnings.

    Advanced Tips for a Production‑Like Local Environment

    If you want to mimic production more closely, consider these enhancements:

  • Python Web Security Best Practices Guide

    When you’re building a Python‑powered web application, security isn’t an afterthought—it’s a core pillar of reliable, user‑friendly software. This Python web security best practices guide walks you through the essential steps, from secure coding habits to framework‑specific defenses, helping you protect your app against the most common threats while keeping performance and developer productivity high.

    Understanding the Threat Landscape

    Before you can defend your application, you need to know what you’re defending against. The most prevalent web security risks, as highlighted by the OWASP Top 10, still affect Python projects today.

    Key Threats to Watch

    • Injection attacks (SQL, NoSQL, command injection) that let attackers execute arbitrary code.
    • Cross‑Site Scripting (XSS) where malicious scripts run in a user’s browser.
    • Cross‑Site Request Forgery (CSRF) that tricks authenticated users into performing unwanted actions.
    • Broken authentication and session management exposing user credentials.
    • Insecure deserialization leading to remote code execution.
    • Security misconfiguration such as default credentials or open debug modes.

    Understanding these vectors sets the stage for a proactive security strategy.

    Secure Coding Practices for Python

    Good security starts with clean, defensive code. Below are habits every Python developer should adopt.

    Validate and Sanitize All Input

    # Example: Using Django's built‑in validators
    from django.core.validators import EmailValidator
    email = EmailValidator()(user_input.get('email'))
    
    • Never trust client‑side data; always perform server‑side validation.
    • Prefer whitelist validation (accept only known good patterns) over blacklist.
    • Use libraries like cerberus or pydantic for schema‑based validation.

    Employ Parameterized Queries

    # Using psycopg2 with PostgreSQL
    cursor.execute(
        "SELECT * FROM users WHERE email = %s",
        (email,)
    )
    
    • Avoid string concatenation for SQL statements.
    • Leverage ORM query methods (e.g., Django ORM, SQLAlchemy) which automatically parameterize.

    Escape Output Properly

    When rendering user‑generated content, escape HTML entities to stop XSS.

    # Jinja2 auto‑escaping (enabled by default)
    {{ user_comment }}
    

    If you must allow limited HTML, use a sanitizer like bleach.

    Limit Use of eval and exec

    Dynamic execution functions are powerful but dangerous. If you must evaluate expressions, consider ast.literal_eval instead.

    Framework‑Specific Safeguards

    Both Django and Flask provide built‑in security mechanisms. Knowing how to configure them correctly is vital.

    Django Security Settings

    • SECURE_SSL_REDIRECT = True – forces HTTPS.
    • SESSION_COOKIE_SECURE = True and CSRF_COOKIE_SECURE = True – protect cookies over TLS.
    • SECURE_HSTS_SECONDS = 31536000 – enable HTTP Strict Transport Security.
    • X_FRAME_OPTIONS = ‘DENY’ – prevent clickjacking.

    Don’t forget to run python manage.py check --deploy before going live; it flags many common misconfigurations.

    Flask Security Extensions

    • Use Flask-Talisman to enforce HTTPS, HSTS, CSP, and other headers.
    • Integrate Flask-WTF for CSRF protection on all forms.
    • Set SESSION_COOKIE_HTTPONLY = True and SESSION_COOKIE_SECURE = True in app.config.
    from flask_talisman import Talisman
    Talisman(app, content_security_policy={
        'default-src': ["'self'"],
        'script-src': ["'self'", 'cdnjs.cloudflare.com']
    })
    

    Authentication and Authorization Best Practices

    Strong identity management is the backbone of a secure web app.

    Password Handling

    • Never store plain‑text passwords; use argon2 or bcrypt via passlib.
    • Enforce minimum length, complexity, and rate‑limit login attempts.
    • Implement password‑reset tokens that expire after a short window.

    Token‑Based Authentication

    JSON Web Tokens (JWT) are popular, but they must be used carefully.

    # Example: Creating a signed JWT with PyJWT
    import jwt, datetime
    payload = {
        'sub': user.id,
        'iat': datetime.datetime.utcnow(),
        'exp': datetime.datetime.utcnow() + datetime.timedelta(hours=2)
    }
    token = jwt.encode(payload, app.config['SECRET_KEY'], algorithm='HS256')
    
    • Prefer short expiration times and refresh‑token flows.
    • Store JWTs in HttpOnly, Secure cookies rather than localStorage.
    • Validate the token’s signature and claims on each request.

    Role‑Based Access Control (RBAC)

    Define clear permission sets and check them at the view or route level.

    # Django example using @permission_required
    @permission_required('blog.change_post', raise_exception=True)
    def edit_post(request, pk):
        ...
    

    Protecting Data In Transit and At Rest

    Data security doesn’t stop at the application layer.

    Enforce HTTPS Everywhere

    • Obtain a TLS certificate from Let’s Encrypt or a commercial CA.
    • Redirect all HTTP traffic to HTTPS (use SECURE_SSL_REDIRECT or Flask-Talisman).
    • Enable HSTS to force browsers to use HTTPS for future requests.

    Encrypt Sensitive Data at Rest

    • Use field‑level encryption for personally identifiable information (PII) with libraries like cryptography.
    • Store encryption keys in a dedicated secrets manager (AWS KMS, HashiCorp Vault).
    • Never hard‑code secrets in source code or configuration files.

    Managing Dependencies and Secrets

    Third‑party packages are a common attack surface.

    Dependency Hygiene

    • Pin exact versions in requirements.txt or use Pipfile.lock.
    • Run pip-audit or Safety regularly to detect known vulnerabilities.
    • Prefer well‑maintained packages; check GitHub stars, recent commits, and issue activity.

    Secure Secret Storage

    • Leverage environment variables for API keys, database passwords, and secret tokens.
    • Combine with python-dotenv for local development, but never commit .env to version control.
    • For production, use cloud‑native secret managers (AWS Secrets Manager, GCP Secret Manager, Azure Key Vault).

    Monitoring, Logging, and Incident Response

    Detecting an attack early can dramatically reduce impact.

    Structured Logging

    import structlog
    log = structlog.get_logger()
    log.info("user_login", user_id=user.id, ip=request.remote_addr)
    
    • Include request IDs, user IDs, and timestamps.
    • Mask or omit sensitive fields (passwords, credit‑card numbers).

    Real‑Time Monitoring

    • Integrate with tools like Sentry, Datadog, or ELK stack for error tracking.
    • Set alerts for spikes in 4xx/5xx responses, authentication failures, or unusual traffic patterns.

    Incident Response Playbook

    1. Detect – Use alerts and logs to confirm a breach.
    2. Contain – Disable compromised accounts, rotate keys.
    3. Eradicate – Patch vulnerable code, update dependencies.
    4. Recover – Restore from clean backups, verify integrity.
    5. Post‑mortem – Document findings and improve controls.

    Testing and Auditing Your Python Web App

    Regular testing turns security from a checklist into a habit.

    Static Application Security Testing (SAST)

    • Run bandit
  • Python Multi-Factor Authentication Mfa Setup

    In today’s security‑first landscape, relying on a single password is no longer enough. Multi‑factor authentication (MFA) adds an extra layer of protection by requiring users to prove their identity with something they know, something they have, or something they are. If you’re a Python developer, setting up MFA can be straightforward and highly customizable, whether you’re building a Django web app, a Flask API, or a simple command‑line tool. This guide walks you through the core concepts, popular libraries, and step‑by‑step implementation strategies you need to secure your Python projects with MFA.

    Why MFA Matters for Python Applications

    Implementing MFA reduces the risk of credential stuffing, phishing, and brute‑force attacks. By demanding a second factor—typically a time‑based one‑time password (TOTP) or a push notification—you dramatically lower the chance that a compromised password alone can grant unauthorized access. For Python developers, MFA also offers:

    • Compliance readiness: Meet GDPR, HIPAA, and PCI DSS requirements that mandate strong authentication.
    • Improved user trust: Users feel safer knowing their accounts are protected beyond just a password.
    • Flexibility: Python’s rich ecosystem lets you integrate MFA with web frameworks, REST APIs, and even desktop applications.

    Core MFA Methods You Can Implement in Python

    1. Time‑Based One‑Time Passwords (TOTP)

    TOTP generates a short numeric code that changes every 30 seconds. It’s the method behind Google Authenticator, Authy, and Microsoft Authenticator. The algorithm is defined in RFC 6238 and can be implemented in Python using the pyotp library.

    2. HMAC‑Based One‑Time Passwords (HOTP)

    HOTP relies on a counter rather than time, making it useful for hardware tokens or environments where clocks may drift. It’s defined in RFC 4226 and also supported by pyotp.

    3. Push‑Based Authentication

    Push notifications send a “Approve or Deny” prompt to a mobile app (e.g., Duo, Authy). While this requires a third‑party service, Python SDKs make integration painless.

    4. Email or SMS OTP

    Sending a short code via email or SMS is a common fallback. Services like Twilio, SendGrid, or AWS SNS provide Python clients to dispatch these codes securely.

    Choosing the Right Python MFA Library

    Below is a quick comparison of the most popular libraries, helping you decide which fits your project best.

    • pyotp – Simple, pure‑Python implementation for TOTP/HOTP. No external dependencies.
    • django-otp – Extends Django’s authentication system with OTP support, integrates with django-two-factor-auth for a full‑featured UI.
    • Flask‑User – Offers optional 2FA for Flask apps, built on top of pyotp.
    • python-digital‑ocean‑2fa – Specialized for DigitalOcean’s API, useful for DevOps scripts.
    • duo_client – Official Duo Security SDK for push‑based MFA.

    Step‑by‑Step: Adding TOTP MFA to a Flask Application

    Prerequisites

    • Python 3.8+ installed
    • Flask 2.x
    • Virtual environment (recommended)
    • Basic user model with a password hash

    1. Install Required Packages

    pip install Flask pyotp qrcode[pil] pillow

    2. Extend Your User Model

    Add two new fields: mfa_secret to store the base32‑encoded secret, and mfa_enabled to toggle MFA per user.

    class User(db.Model):
        id = db.Column(db.Integer, primary_key=True)
        username = db.Column(db.String(150), unique=True, nullable=False)
        password_hash = db.Column(db.String(255), nullable=False)
        mfa_secret = db.Column(db.String(32), nullable=True)
        mfa_enabled = db.Column(db.Boolean, default=False)

    3. Generate a Secret and QR Code

    When a user opts into MFA, generate a secret and present a QR code that can be scanned by an authenticator app.

    import pyotp, qrcode, io
    from flask import send_file
    
    def enable_mfa(user):
        # Create a new secret if one doesn’t exist
        if not user.mfa_secret:
            user.mfa_secret = pyotp.random_base32()
            db.session.commit()
    
        # Build the otpauth URI
        uri = pyotp.totp.TOTP(user.mfa_secret).provisioning_uri(
            name=user.username,
            issuer_name="MyFlaskApp"
        )
    
        # Generate QR code image
        img = qrcode.make(uri)
        buf = io.BytesIO()
        img.save(buf, format='PNG')
        buf.seek(0)
        return send_file(buf, mimetype='image/png')

    4. Verify the Token During Login

    After the user supplies a valid password, check whether MFA is enabled. If so, prompt for the 6‑digit TOTP.

    def verify_mfa(user, token):
        if not user.mfa_enabled or not user.mfa_secret:
            return False
        totp = pyotp.TOTP(user.mfa_secret)
        return totp.verify(token, valid_window=1)  # Allows a 30‑second clock skew

    5. Update the Login Flow

    @app.route('/login', methods=['GET', 'POST'])
    def login():
        if request.method == 'POST':
            username = request.form['username']
            password = request.form['password']
            user = User.query.filter_by(username=username).first()
    
            if user and check_password_hash(user.password_hash, password):
                if user.mfa_enabled:
                    session['pre_mfa_user_id'] = user.id
                    return redirect(url_for('mfa_challenge'))
                login_user(user)
                return redirect(url_for('dashboard'))
    
        return render_template('login.html')
    
    @app.route('/mfa', methods=['GET', 'POST'])
    def mfa_challenge():
        user_id = session.get('pre_mfa_user_id')
        if not user_id:
            return redirect(url_for('login'))
    
        user = User.query.get(user_id)
    
        if request.method == 'POST':
            token = request.form['token']
            if verify_mfa(user, token):
                login_user(user)
                session.pop('pre_mfa_user_id')
                return redirect(url_for('dashboard'))
            flash('Invalid authentication code.')
    
        return render_template('mfa.html')

    6. Allow Users to Disable or Reset MFA

    Provide a secure flow that requires re‑authentication (password + existing MFA) before letting users turn MFA off or generate a new secret.

    Implementing MFA in Django with django‑two‑factor‑auth

    If you’re using Django, the django-two-factor-auth package offers a plug‑and‑play solution that integrates with the built‑in admin, REST frameworks, and custom login views.

    Installation

    pip install django-two-factor-auth[phonenumbers]  # optional phonenumbers for SMS

    Configuration Steps

    1. Add the required apps to INSTALLED_APPS:
      INSTALLED_APPS = [
          # …
          'django_otp',
          'django_otp.plugins.otp_static',
          'django_otp.plugins.otp_totp',
          'two_factor',
          # …
      ]
    2. Include the URL patterns:
      urlpatterns = [
          # …
          path('account/', include('two_factor.urls', 'two_factor')),
          # …
      ]
    3. Run migrations:
      python manage.py migrate
    4. Optionally enable the LoginView that forces MFA for all users or only those with user.is_staff etc.

    After these steps, Django automatically presents a QR code during the first login, stores the secret in the database, and validates TOTP codes on subsequent logins.

    Best Practices for Secure MFA Implementation

    • Never store plain‑text secrets. Use Django’s EncryptedField or encrypt the secret before persisting it.
    • Enforce rate limiting. Limit MFA verification attempts to mitigate brute‑force attacks (e.g., 5 attempts per 10 minutes).
    • Provide backup codes. Generate a set of one‑time use backup codes for users who lose access to their authenticator device.
    • Support multiple factors. Allow users to choose between TOTP, push, or SMS based on their preferences and threat model.
    • Log MFA events. Record successful and failed MFA attempts with timestamps and IP addresses for audit trails.
    • Use HTTPS everywhere. MFA tokens are sensitive; always transmit them over TLS.

    Testing Your MFA Flow

    Automated tests ensure that MFA works reliably across environments. Below is a pytest example for a Flask TOTP flow.

    import pytest, pyotp
    from app import create_app, db, User

    @pytest.fixture
    def client():
    app = create_app({'TESTING': True})
    with app.test_client() as client:
    with app.app_context():
    db.create_all()
    yield client
    db.session.remove()
    db.drop_all()

    def test_mfa_login(client

  • Python Password Reset Via Email Tutorial

    In today’s web‑driven world, a reliable password reset flow is not just a nice‑to‑have feature—it’s a security essential. Users inevitably forget passwords, and a well‑crafted email‑based reset process protects both the user experience and your application’s integrity. In this tutorial, we’ll walk through building a complete Python password‑reset system using Flask, itsdangerous for secure token generation, and SMTP for email delivery. By the end, you’ll have a production‑ready endpoint that sends a one‑time link, validates it, and lets users set a new password—all while following best practices for security and SEO‑friendly content.

    Why Choose Email for Password Resets?

    Emails serve as a trusted, out‑of‑band channel that confirms the user’s identity without exposing sensitive data. When implemented correctly, an email‑based reset offers:

    • Security: Tokens are time‑limited and single‑use, preventing replay attacks.
    • Usability: Users receive a familiar link they can click from any device.
    • Scalability: SMTP services handle high volumes, making the solution suitable for small apps and large SaaS platforms alike.

    Project Setup: What You’ll Need

    Python Packages

    First, install the core dependencies. Open a terminal and run:

    pip install Flask Flask-Mail itsdangerous Flask-WTF email-validator

    These libraries provide web routing, email handling, secure token creation, and form validation.

    Configuration Overview

    Create a config.py file to store sensitive settings. Never hard‑code passwords; use environment variables instead.

    # config.py
    import os
    
    class Config:
        SECRET_KEY = os.getenv('SECRET_KEY', 'dev-secret-key')
        SECURITY_PASSWORD_SALT = os.getenv('SECURITY_PASSWORD_SALT', 'dev-salt')
        MAIL_SERVER = os.getenv('MAIL_SERVER', 'smtp.gmail.com')
        MAIL_PORT = int(os.getenv('MAIL_PORT', 587))
        MAIL_USE_TLS = True
        MAIL_USERNAME = os.getenv('MAIL_USERNAME')
        MAIL_PASSWORD = os.getenv('MAIL_PASSWORD')
        MAIL_DEFAULT_SENDER = os.getenv('MAIL_DEFAULT_SENDER')

    Step‑by‑Step Implementation

    1. Initialize Flask and Flask‑Mail

    # app.py
    from flask import Flask
    from flask_mail import Mail
    from config import Config
    
    app = Flask(__name__)
    app.config.from_object(Config)
    
    mail = Mail(app)

    2. Create the Password Reset Request Form

    Using Flask‑WTF ensures CSRF protection and clean validation.

    # forms.py
    from flask_wtf import FlaskForm
    from wtforms import StringField, SubmitField
    from wtforms.validators import DataRequired, Email
    
    class PasswordResetRequestForm(FlaskForm):
        email = StringField('Email', validators=[DataRequired(), Email()])
        submit = SubmitField('Send Reset Link')

    3. Generate a Secure Token

    The itsdangerous library creates a signed, time‑limited token that encodes the user’s email address.

    # token.py
    from itsdangerous import URLSafeTimedSerializer
    from flask import current_app
    
    def generate_reset_token(email):
        serializer = URLSafeTimedSerializer(current_app.config['SECRET_KEY'])
        return serializer.dumps(email, salt=current_app.config['SECURITY_PASSWORD_SALT'])
    
    def confirm_reset_token(token, expiration=3600):
        serializer = URLSafeTimedSerializer(current_app.config['SECRET_KEY'])
        try:
            email = serializer.loads(
                token,
                salt=current_app.config['SECURITY_PASSWORD_SALT'],
                max_age=expiration
            )
        except Exception:
            return None
        return email

    4. Send the Reset Email

    Compose a clear, actionable email that includes the reset link.

    # email.py
    from flask_mail import Message
    from flask import url_for, render_template
    from app import mail
    
    def send_reset_email(user_email):
        token = generate_reset_token(user_email)
        reset_url = url_for('reset_password', token=token, _external=True)
        html_body = render_template('reset_email.html', reset_url=reset_url)
        msg = Message(subject='Your Password Reset Link',
                      recipients=[user_email],
                      html=html_body)
        mail.send(msg)

    5. Build the Request Route

    This endpoint renders the form, validates input, and triggers the email.

    # routes.py
    from flask import render_template, flash, redirect, url_for, request
    from app import app
    from forms import PasswordResetRequestForm
    from email import send_reset_email
    from models import User  # Assume a SQLAlchemy model
    
    @app.route('/reset_password', methods=['GET', 'POST'])
    def reset_request():
        form = PasswordResetRequestForm()
        if form.validate_on_submit():
            user = User.query.filter_by(email=form.email.data).first()
            if user:
                send_reset_email(user.email)
                flash('A password reset link has been sent to your email.', 'info')
                return redirect(url_for('login'))
            else:
                flash('Email not found in our system.', 'warning')
        return render_template('reset_request.html', form=form)

    6. Create the Reset Form and Route

    After clicking the link, the user lands on a page to set a new password.

    # forms.py (add)
    from wtforms import PasswordField, EqualTo
    
    class PasswordResetForm(FlaskForm):
        password = PasswordField('New Password', validators=[DataRequired()])
        confirm = PasswordField('Confirm Password',
                                validators=[DataRequired(),
                                            EqualTo('password', message='Passwords must match')])
        submit = SubmitField('Reset Password')
    
    # routes.py (add)
    @app.route('/reset_password/', methods=['GET', 'POST'])
    def reset_password(token):
        email = confirm_reset_token(token)
        if not email:
            flash('The reset link is invalid or has expired.', 'danger')
            return redirect(url_for('reset_request'))
    
        form = PasswordResetForm()
        if form.validate_on_submit():
            user = User.query.filter_by(email=email).first_or_404()
            user.set_password(form.password.data)  # Assume a method that hashes the password
            db.session.commit()
            flash('Your password has been updated! You can now log in.', 'success')
            return redirect(url_for('login'))
        return render_template('reset_password.html', form=form)

    Best Practices for a Secure Reset Flow

    • Limit Token Lifetime: Keep the expiration window short (e.g., 1 hour) to reduce exposure.
    • One‑Time Use: Invalidate the token after a successful password change.
    • Rate Limiting: Prevent brute‑force attempts by limiting reset requests per IP or user.
    • HTTPS Everywhere: Serve both the request and reset pages over TLS to protect token integrity.
    • Clear Messaging: Avoid revealing whether an email exists in your database; use generic success messages.

    Testing the Workflow

    Before deploying, run through these checks:

    1. Submit a valid email and verify the reset link arrives in the inbox.
    2. Click the link after the token expires to ensure the error handling works.
    3. Attempt to reuse the same token after a successful reset—should be rejected.
    4. Test edge cases such as malformed tokens or missing query parameters.

    Automated tests can be written with pytest and Flask’s test client to simulate each step.

    Deploying to Production

    When moving to a live environment, keep these deployment tips in mind:

    • Store SECRET_KEY, SECURITY_PASSWORD_SALT, and mail credentials in a secure vault (e.g., AWS Secrets Manager or Docker secrets).
    • Use a reputable transactional email service (SendGrid, Mailgun, Amazon SES) for higher deliverability.
    • Enable DKIM and SPF records for your domain to avoid spam filters.
    • Monitor bounce rates and set up alerts for abnormal reset request spikes.

    Conclusion

    Implementing a robust password reset via email in Python doesn’t have to be daunting. By leveraging Flask, Flask‑Mail, and itsdangerous, you can create a secure, user‑friendly flow that protects both your users and your brand. Remember to follow the security best practices outlined above, test thoroughly, and configure your production environment with care. With these steps in place, you’ll deliver a seamless experience that keeps users confident that their accounts are safe—even when they forget their passwords.

  • Python Github Oauth Authentication Flow

    Are you looking to let users log into your Python web app with their GitHub accounts? Leveraging GitHub’s OAuth 2.0 service not only streamlines the sign‑in experience but also opens the door to powerful GitHub APIs, enabling you to fetch repositories, issues, and more on behalf of authenticated users. In this guide we’ll walk through the entire Python GitHub OAuth authentication flow, from registering your OAuth app on GitHub to handling callbacks, exchanging authorization codes for access tokens, and finally securing the user session. Whether you’re building a Flask micro‑service or a full‑featured Django portal, the steps below will give you a solid, production‑ready foundation.

    Why Choose GitHub OAuth for Your Python Application?

    • Trusted Identity Provider: Millions of developers already have GitHub accounts, making sign‑in frictionless.
    • Rich API Access: Once authenticated, you can call GitHub’s REST or GraphQL APIs to read user data, repositories, pull requests, and more.
    • Secure Token Management: OAuth 2.0 uses short‑lived access tokens and optional refresh tokens, reducing the risk of credential leakage.
    • Easy Integration: Popular Python frameworks (Flask, Django, FastAPI) have mature libraries that abstract most of the boilerplate.

    Step‑by‑Step Overview of the GitHub OAuth Flow

    1. Register your application on GitHub. Obtain a Client ID and Client Secret.
    2. Redirect the user to GitHub’s authorization endpoint. Include scopes and a state parameter.
    3. GitHub authenticates the user and asks for consent.
    4. GitHub redirects back to your callback URL with an authorization code and the original state.
    5. Exchange the code for an access token. Make a POST request to GitHub’s token endpoint.
    6. Use the access token to call GitHub APIs or store it securely for later use.

    Registering Your OAuth App on GitHub

    Before any code is written, you must create an OAuth app in your GitHub account:

    • Navigate to GitHub Settings → Developer settings → OAuth Apps.
    • Click “New OAuth App”.
    • Fill in the required fields:
      • Application name – e.g., “MyPythonApp”.
      • Homepage URL – your public site or http://localhost:5000 for local testing.
      • Authorization callback URL – where GitHub will redirect after login (e.g., http://localhost:5000/auth/callback).
    • Save to receive a Client ID and Client Secret. Keep the secret private!

    Implementing the Flow in Flask

    Flask is lightweight and perfect for demonstrating the OAuth dance. Below is a minimal, yet production‑ready, implementation.

    Project Structure

    myapp/
    │
    ├─ app.py
    ├─ config.py
    └─ requirements.txt
    

    Installing Dependencies

    pip install Flask requests python-dotenv
    

    Configuration (config.py)

    import os
    from dotenv import load_dotenv
    
    load_dotenv()  # Loads variables from .env file
    
    GITHUB_CLIENT_ID = os.getenv('GITHUB_CLIENT_ID')
    GITHUB_CLIENT_SECRET = os.getenv('GITHUB_CLIENT_SECRET')
    BASE_URL = os.getenv('BASE_URL', 'http://localhost:5000')
    

    Core Application (app.py)

    from flask import Flask, redirect, request, session, url_for, jsonify
    import requests
    from config import GITHUB_CLIENT_ID, GITHUB_CLIENT_SECRET, BASE_URL
    
    app = Flask(__name__)
    app.secret_key = 'replace‑with‑a‑strong‑random‑value'  # Needed for session handling
    
    # ----------------------------------------------------------------------
    # Helper: Build the GitHub authorization URL
    # ----------------------------------------------------------------------
    def github_authorize_url(state):
        params = {
            'client_id': GITHUB_CLIENT_ID,
            'redirect_uri': f'{BASE_URL}{url_for("callback")}',
            'scope': 'read:user repo',   # Adjust scopes as needed
            'state': state
        }
        url = 'https://github.com/login/oauth/authorize'
        request_url = requests.Request('GET', url, params=params).prepare().url
        return request_url
    
    # ----------------------------------------------------------------------
    # Route: Home – offers a “Login with GitHub” button
    # ----------------------------------------------------------------------
    @app.route('/')
    def index():
        return (
            f'

    Welcome to MyPythonApp

    ' f'Login with GitHub' ) # ---------------------------------------------------------------------- # Route: Initiates the OAuth flow # ---------------------------------------------------------------------- @app.route('/login') def login(): # Generate a random state token to protect against CSRF state = os.urandom(16).hex() session['oauth_state'] = state return redirect(github_authorize_url(state)) # ---------------------------------------------------------------------- # Route: Callback – GitHub redirects here after user consent # ---------------------------------------------------------------------- @app.route('/auth/callback') def callback(): # Verify state parameter received_state = request.args.get('state') if received_state != session.get('oauth_state'): return 'Invalid state parameter', 400 code = request.args.get('code') token_url = 'https://github.com/login/oauth/access_token' headers = {'Accept': 'application/json'} payload = { 'client_id': GITHUB_CLIENT_ID, 'client_secret': GITHUB_CLIENT_SECRET, 'code': code, 'redirect_uri': f'{BASE_URL}{url_for(\"callback\")}', 'state': received_state } # Exchange code for access token token_response = requests.post(token_url, data=payload, headers=headers) token_json = token_response.json() access_token = token_json.get('access_token') if not access_token: return f'Error retrieving access token: {token_json}', 400 # Store token in session (or a database for longer‑term storage) session['github_token'] = access_token return redirect(url_for('profile')) # ---------------------------------------------------------------------- # Route: Profile – demonstrates an authenticated API call # ---------------------------------------------------------------------- @app.route('/profile') def profile(): token = session.get('github_token') if not token: return redirect(url_for('index')) api_url = 'https://api.github.com/user' headers = {'Authorization': f'token {token}'} user_resp = requests.get(api_url, headers=headers).json() # Simple JSON output; in a real app you’d render a template return jsonify(user_resp) if __name__ == '__main__': app.run(debug=True)

    This script performs the full OAuth cycle:

    • Generates a state token to mitigate CSRF attacks.
    • Redirects the user to GitHub’s /login/oauth/authorize endpoint.
    • Validates the returned state and exchanges the code for an access token.
    • Uses the token to request the authenticated user’s profile via https://api.github.com/user.

    Adapting the Flow for Django

    If you prefer Django, the same concepts apply, but you’ll typically rely on a library such as social-auth-app-django. Below is a concise configuration outline.

    Install Required Packages

    pip install Django social-auth-app-django python-dotenv
    

    Update settings.py

    import os
    from dotenv import load_dotenv
    
    load_dotenv()
    
    AUTHENTICATION_BACKENDS = (
        'social_core.backends.github.GithubOAuth2',
        'django.contrib.auth.backends.ModelBackend',
    )
    
    SOCIAL_AUTH_GITHUB_KEY = os.getenv('GITHUB_CLIENT_ID')
    SOCIAL_AUTH_GITHUB_SECRET = os.getenv('GITHUB_CLIENT_SECRET')
    SOCIAL_AUTH_GITHUB_SCOPE = ['read:user', 'repo']
    
    LOGIN_REDIRECT_URL = '/'
    LOGOUT_REDIRECT_URL = '/'
    

    URL Configuration (urls.py)

    from django.urls import path, include
    from django.contrib import admin
    
    urlpatterns = [
        path('admin/', admin.site.urls),
        path('auth/', include('social_django.urls', namespace='social')),
        path('', include('myapp.urls')),  # Your app’s own routes
    ]
    

    Template Snippet for Login

    <a href="{% url 'social:begin' 'github' %}">Login with GitHub</a>
    

    With social-auth-app-django handling the heavy lifting, you get automatic state verification, token storage, and user model integration out of the box.

    Best Practices for Secure Token Management

    • Never expose Client Secret on the client side. Keep it on the server and use environment variables.
    • Use HTTPS in production. OAuth redirects over plain HTTP are vulnerable to man‑in‑the‑middle attacks.
    • Validate the state parameter. It protects against CSRF and should be stored per‑session.
    • <

  • Python Google Oauth2 Login Integration

    Integrating Google OAuth2 login into a Python web application is a powerful way to boost user experience, improve security, and leverage Google’s trusted authentication system. Whether you’re building a small Flask prototype or a full‑featured Django portal, the process follows a clear set of steps that can be mastered quickly. In this guide we’ll walk through everything you need—from creating a Google API project and configuring credentials, to writing clean Python code that handles the OAuth2 flow, validates tokens, and securely logs users in. By the end, you’ll have a production‑ready implementation that not only simplifies sign‑ups but also enhances SEO with relevant keywords like “Python Google OAuth2 login integration.”

    What Is OAuth2 and Why Use Google for Authentication?

    OAuth2 is an open standard for access delegation, allowing users to grant limited access to their resources without sharing passwords. Google’s OAuth2 service is especially popular because:

    • Trust: Users already trust Google’s login UI, reducing friction.
    • Security: Tokens are short‑lived and can be revoked at any time.
    • Rich Profile Data: With the appropriate scopes you can fetch the user’s email, name, and profile picture.
    • SEO Benefits: Implementing a modern authentication flow signals a well‑maintained site to search engines.

    Step 1: Create a Google Cloud Project and OAuth2 Credentials

    1.1 Open the Google Cloud Console

    Navigate to Google Cloud Console and sign in with a Google account that has project‑creation rights.

    1.2 Set Up a New Project

    1. Click New Project and give it a descriptive name (e.g., MyPythonApp‑OAuth).
    2. Choose an organization or keep it under “No organization.”
    3. Click Create and wait for the project to provision.

    1.3 Enable the “Google+ API” (or “People API” for newer implementations)

    From the left navigation, select APIs & Services > Library, search for “People API,” and click Enable. This API provides the profile information you’ll request.

    1.4 Create OAuth2 Client ID

    1. Go to APIs & Services > Credentials and click Create Credentials > OAuth client ID.
    2. Choose Web application as the application type.
    3. Enter a name like PythonOAuthClient.
    4. Under Authorized redirect URIs, add the URL where Google should send the auth response (e.g., http://localhost:5000/auth/callback for Flask or http://localhost:8000/accounts/google/login/callback/ for Django).
    5. Click Create. Copy the Client ID and Client Secret – you’ll need them in your Python code.

    Step 2: Install Required Python Packages

    Both Flask and Django have dedicated extensions that simplify the OAuth2 flow. Install the core libraries using pip:

    pip install requests
    pip install google-auth
    pip install google-auth-oauthlib
    # Flask specific
    pip install Flask Flask-Login Flask-OAuthlib
    # Django specific
    pip install Django social-auth-app-django
    

    These packages handle token exchange, session management, and user model integration, letting you focus on business logic.

    Step 3: Implement Google OAuth2 in Flask

    3.1 Basic Flask App Skeleton

    from flask import Flask, redirect, url_for, session, request
    from flask_oauthlib.client import OAuth
    
    app = Flask(__name__)
    app.secret_key = 'YOUR_RANDOM_SECRET_KEY'
    
    oauth = OAuth(app)
    google = oauth.remote_app(
        'google',
        consumer_key='YOUR_CLIENT_ID',
        consumer_secret='YOUR_CLIENT_SECRET',
        request_token_params={
            'scope': 'email profile'
        },
        base_url='https://www.googleapis.com/oauth2/v1/',
        request_token_url=None,
        access_token_method='POST',
        access_token_url='https://oauth2.googleapis.com/token',
        authorize_url='https://accounts.google.com/o/oauth2/auth',
    )
    

    3.2 Login Route

    @app.route('/login')
    def login():
        return google.authorize(
            callback=url_for('authorized', _external=True)
        )
    

    3.3 Authorization Callback

    @app.route('/auth/callback')
    def authorized():
        resp = google.authorized_response()
        if resp is None or resp.get('access_token') is None:
            return 'Access denied: reason=%s error=%s' % (
                request.args['error_reason'],
                request.args['error_description']
            )
        session['google_token'] = (resp['access_token'], '')
        user_info = google.get('userinfo')
        # Example: store user info in session
        session['user'] = {
            'email': user_info.data['email'],
            'name':  user_info.data['name'],
            'picture': user_info.data['picture']
        }
        return redirect(url_for('profile'))
    

    3.4 Token Getter

    @google.tokengetter
    def get_google_oauth_token():
        return session.get('google_token')
    

    3.5 Protected Profile Page

    @app.route('/profile')
    def profile():
        if 'user' not in session:
            return redirect(url_for('login'))
        user = session['user']
        return f\"\"\"<h2>Welcome, {user['name']}!</h2>
                   <img src='{user['picture']}' alt='Profile picture'>
                   <p>Email: {user['email']}</p>\"\"\"
    

    Run the app with flask run, visit /login, and you’ll be redirected to Google’s consent screen. After granting access, the user lands on the profile page with their Google data.

    Step 4: Implement Google OAuth2 in Django Using Social‑Auth

    4.1 Configure Settings

    # settings.py
    INSTALLED_APPS = [
        # … default apps …
        'social_django',
    ]
    
    AUTHENTICATION_BACKENDS = (
        'social_core.backends.google.GoogleOAuth2',
        'django.contrib.auth.backends.ModelBackend',
    )
    
    SOCIAL_AUTH_GOOGLE_OAUTH2_KEY = 'YOUR_CLIENT_ID'
    SOCIAL_AUTH_GOOGLE_OAUTH2_SECRET = 'YOUR_CLIENT_SECRET'
    
    LOGIN_URL = 'login'
    LOGIN_REDIRECT_URL = 'profile'
    LOGOUT_REDIRECT_URL = 'home'
    
    # Optional: request extra profile fields
    SOCIAL_AUTH_GOOGLE_OAUTH2_SCOPE = ['email', 'profile']
    SOCIAL_AUTH_GOOGLE_OAUTH2_EXTRA_DATA = ['first_name', 'last_name']
    

    4.2 URL Configuration

    # urls.py
    from django.urls import path, include
    from . import views
    
    urlpatterns = [
        path('', views.home, name='home'),
        path('profile/', views.profile, name='profile'),
        path('auth/', include('social_django.urls', namespace='social')),
    ]
    

    4.3 Simple Views

    # views.py
    from django.shortcuts import render, redirect
    from django.contrib.auth.decorators import login_required
    
    def home(request):
        return render(request, 'home.html')
    
    @login_required
    def profile(request):
        user = request.user
        # Social auth stores extra data in user.social_auth
        extra = user.social_auth.get(provider='google-oauth2')
        context = {
            'email': user.email,
            'name': extra.extra_data.get('name'),
            'picture': extra.extra_data.get('picture')
        }
        return render(request, 'profile.html', context)
    

    4.4 Template Snippet for Login

    <a href="{% url 'social:begin' 'google-oauth2' %}?next={{ request.path }}">
        Sign in with Google
    </a>
    

    After adding the templates, run python manage.py migrate to create the required tables, then start the server. Clicking the “Sign in with Google” link launches the OAuth2 flow, and Django automatically creates or updates the user record.

    Step 5: Validate Tokens and Secure Your Integration

    Even though Google handles most of the heavy lifting, you should still verify the ID token on the server side to protect against token replay attacks.

    • Use google.oauth2.id_token.verify_oauth2_token() to decode and validate the token.
    • Check the aud (audience) claim matches your client ID.
    • Confirm the token has not expired (exp claim).
    from google.oauth2 import id_token
    from google.auth.transport import requests

    def verify_token(id_token_str):
    try:
    idinfo = id_token.verify_oauth2_token(
    id_token_str,
    requests.Request(),
    'YOUR_CLIENT_ID