Git Scraping

Inspired by Simon Willison.

Download a file regularly using Git Actions, check it into Git and see how it changes over time.

Scraping Project

Projects	Notes	Updated
Toronto Lobbyist Registry Data	Downloads from City of Toronto Lobbyist Registry The files here are quite large, so they need to be stored in Git LFS, so Github Git Blame does not work Collects the data for Lobbying in Toronto project	Daily at 21:30 UTC ie 5:30 pm EST (The data source is updated around 4:30 pm everyday)
City of Toronto Contracts Data	Downloads Competitive Contracts, Non Competitive Contracts and Solicitations for the Purchase of Goods and Services issued by the City of Toronto. Currently not working due to a change in how data is returned. I am following up with the team that manages SAP Ariba—the procurement software for the City of Toronto. It should be working again by May 2024	Daily at midnight UTC
Ontario Marriage Officiants	Downloads list of registered marriage officiants in Ontario	Daily at midnight UTC
City of Toronto Short Term Rentals Registration	Downloads Short Term Rentals Registrations	Daily at 16:30 UTC ie 12:30 pm EST (The data source is updated around 12 pm everyday)
WHOIS TLDs	Downloads a list of WHOIS servers for top-level domains Unlike other git-scaping projects, updates to the repo require me to manually approve a PR as opposed to being automatically merged in	Daily at midnight UTC
Ontario Legislature Bills	Downloads a list of bills for each session of the Ontario legislature since the 36th Parliament, 1st Session (September 26, 1995) This is not particularly useful. I wrote this code so I see a list of private bills, but since I have it I thought why not hook it up to a GitHub action	Monthly on the 1st

Future scraping pages:

TODO:

Create Github: template repository