RM of Macdonald Web Scraper Suite

Overview

This project contains a suite of asynchronous Python web scrapers designed to navigate, extract, and archive public municipal records from the RM of Macdonald website and its underlying All-Net Meetings portal.

Because the municipality relies heavily on legacy ASP.NET WebForms, embedded iframes, and custom Single Page Application (SPA) widgets, standard HTTP scraping libraries (like requests or BeautifulSoup) are ineffective. This suite utilizes Playwright to execute JavaScript, bypass complex DOM hydration, manage browser sessions, and intercept network requests, combined with Markdownify to cleanly convert raw HTML layouts into standardized, offline-readable Markdown (.md) files. All associated attachments (PDFs, by-laws, and maps) are preserved alongside the text in an organized, chronological local folder structure.

---

Tech Stack

• Python 3.12+
• Playwright (Async API / async_playwright): Handles headless Chromium automation, dynamic DOM manipulation, server-side JavaScript postbacks, and request interception.
• Markdownify: Provides aggressive HTML-to-Markdown sanitization and structural conversion.
• Asyncio: Enables concurrent execution, non-blocking network operations, and polite delays.
• Standard Python Libraries: re, os, urllib.parse, mimetypes, datetime.

---

Project Scope

The suite is divided into three standalone scripts targeting different municipal record databases:

• scrape_minutes.py (Council Minutes)
  • Paginates through historical council minutes.
  • Converts complex HTML grids into clean .md files labeled chronologically (e.g., 2026-05-05_minutes_1234.md).
• scrape_meetings.py (Council Meetings & Agendas)
  • Deep-crawls meeting agendas and correlates sessions with their attached documents (By-Laws, Resolutions, Maps).
  • Generates a nested directory structure for each meeting date (e.g., YYYY-MM-DD_meeting/).
  • Hunts down obfuscated file requests, downloads them locally, and dynamically rewrites the generated Markdown links to map to the local PDFs.
• scrape_documents.py (Documents & Downloads)
  • Recursively crawls a custom AJAX-driven SPA virtual file explorer.
  • Mimics human clicks to traverse nested folders, download files, and automatically replicate the municipal folder hierarchy locally (e.g., ./documents_downloads/Public_Works/).
  • Generates a master documents_index.md file at the root to serve as a complete offline directory index.

---

Technical Challenges & Solutions

Throughout development, the municipal portal presented several significant architectural hurdles natively implemented by the All-Net software. Here is how they were resolved:

1. The Iframe Same-Origin Trap

• Problem: The RM website embeds the All-Net meeting portal inside an <iframe>. Because the iframe is hosted on a separate domain (macdonald.allnetmeetings.com), standard Playwright DOM queries (page.query_selector_all) failed to see any links due to browser Same-Origin Policy restrictions, returning 0 links. Conversely, navigating to the iframe URL directly triggered a login redirect wall.
• Solution: We established the initial session on the RM's main website, then implemented frame-traversal logic (for frame in page.frames:) to dynamically identify the All-Net iframe context. This allowed the scripts to inject and execute JavaScript directly within the isolated iframe boundary while keeping the valid session headers alive.

2. ASP.NET Server-Side Pagination

• Problem: Meeting records were split across 23 pages, but the navigation links did not alter the URL parameter (e.g., no ?page=2). Instead, ASP.NET uses javascript:__doPostBack() to submit stateful forms back to the server and redraw the data table dynamically using AJAX UpdatePanels, causing traditional scrapers to loop on page one indefinitely.
• Solution: Engineered a robust pagination loop that evaluates the DOM within the target frame context, explicitly fires physical .click() events on the ASP.NET numeric pagination and ellipsis forward controls, and waits for the wait_for_load_state("networkidle") event alongside a localized timeout buffer to ensure the new table data hydated fully before scanning.

3. Authentication / Login Walls

• Problem: Accessing individual All-Net meeting data URLs directly often triggered a hard redirect to a protected backend login screen.
• Solution: Discovery revealed that the portal validates traffic based on the HTTP Referer header. The scraper contexts were updated to spoof the RM of Macdonald's root domain (referer="https://www.rmofmacdonald.com/") during direct document navigation requests, seamlessly bypassing the security check.

4. Obfuscated JavaScript PDF Links (The popsmall Function)

• Problem: Document attachments were not standard <a href="file.pdf"> elements. The portal masked files behind custom client-side popup triggers: onclick="return popsmall('download.aspx?ty=ag&atid=GUID').
• Solution: Designed a surgical regular expression parser to isolate the hidden download.aspx query strings from inside the onclick attributes. The script then resolves these targets into absolute URLs using urljoin and leverages Playwright's background APIRequestContext to perform direct HTTP GET requests. This approach downloads the binary files silently without opening UI popups and verifies file types accurately via the server's incoming Content-Type headers.

5. Localized Markdown Linking & File Correlation

• Problem: Converting the page layout directly to Markdown left document links pointing back to the external All-Net server, rendering the final archive dependent on the live website and breaking offline viewing.
• Solution: We inverted the traditional scraping pipeline. The script assigns a unique temporary HTML id to every element on the live webpage and downlods the files. Once an attachment successfully downloads, the script tells Playwright to execute a JavaScript block that modifies the active browser DOM directly—stripping the onclick variable and swapping the href to point directly to the relative path of the local file (e.g., ./local_filename.pdf). When markdownify finally processes the modified HTML, the offline relative links are naturally baked in.

6. The SPA Recursive File Explorer

• Problem: The "Documents & Downloads" center used a custom JavaScript widget where folders only existed as abstract <li data-directory="GUID"> entries. Files inside a directory were completely absent from the page HTML until that specific folder button was clicked.
• Solution: Built a recursive tree-traversal algorithm. The script systematically reads the unique directory GUIDs on screen, fires click events to let the page load the next tier of data via AJAX, downloads any files exposed, and utilizes the dynamic <ol class="breadCrumb"> UI to programmatically click "Back," allowing it to systematically map the entire hidden tree into physical local folders.

---

Installation & Setup

1. Clone the repository and set up a virtual environment:

   python3 -m venv .venv
   source .venv/bin/activate
   
   

2. Install core dependencies:

   pip install playwright markdownify
   
   

3. Install Playwright Browsers & System Dependencies: (Crucial for headless Linux environments to prevent execution crashes due to missing X11 or rendering shared objects like libnspr4.so)

   playwright install chromium
   playwright install-deps chromium