07-DataSources

100 Plus Data Sources Data Science#

This is a section with links to data sources. During my data engineer coaching we need to find good data sets to work with. So, I started this section to make it easier to find good sources.

I've taken these links from articles and blog posts. Why not only link the articles? You know, these posts can go away at any time. I want to keep the links to the platforms either way.

I haven't had the chance to check each link myself. Please let me know if something isn't right.

You can find the articles on the bottom of this section to read more. They include even more data sources I haven't had time to add to this list.

Contents#

Student Favorites#

In my Coaching program my students learn by doing a project. And the foundation of every project is selecting a dataset. That can be an API or a file source, depending a lot on the student's goals and interests.

Working out goals for the dataset, figuring out the data modeling, creating the architecture and building it. It's a fun way to learn and get better at Data Engineering.

Here's a list of my student's favorite datasets and APIs

Building a project? Use my project README template to document it properly: template-readme.md

Datasets#

Dataset	Description
Fraud detection	Simulated credit card transactions labeled as fraudulent or legitimate, ideal for classification and imbalanced-data projects.
Industrial equipment monitoring	Sensor readings from industrial equipment (temperature, pressure, vibration) for predictive maintenance and anomaly detection.
Energy demand & generation	Four years of hourly electricity consumption, generation, prices and weather data for Spain, great for time-series forecasting.
Online Retail	Transactional data from a UK online retailer, useful for customer segmentation, market basket and RFM analysis.
Brazilian E-commerce	Olist's public dataset of 100k+ orders with customers, products, payments, reviews and geolocation for end-to-end analytics.
Beijing Air Quality	Hourly air pollutant readings across multiple Beijing stations with weather data, good for time-series and environmental analysis.
NYC Taxi	New York City taxi trip and revenue data for cab drivers, useful for revenue, tipping and time-series analysis.
E-commerce clickstream & transactions	Simulated online shopping clickstream and transaction data — user sessions, page views, clicks and purchases — good for funnel and behavior analysis.
IESO Ontario electricity data	Public report files from Ontario's grid operator: electricity demand, generation by fuel type, market prices and more, updated regularly as CSV/XML.
IoT device temperature readings	Temperature sensor readings from IoT devices with indoor/outdoor tags and timestamps, great for time-series and anomaly detection.
IoT agriculture 2024	Smart-farming sensor data (soil moisture, temperature, humidity, etc.) for agriculture monitoring and IoT analytics projects.

APIs#

API	Description
Bike sharing Bluebikes	Trip and station data for Boston's Bluebikes bike-share system, including GBFS real-time feeds and downloadable trip history.
Bike sharing Divvy Bikes	Trip and station data for Chicago's Divvy bike-share system, with GBFS real-time feeds and historical ride files.
Weather API	Weather API with current conditions, forecasts, historical data and astronomy; generous free tier.
Bluesky API	API for the Bluesky social network (AT Protocol): posts, profiles, feeds and social graph data.
Guardian news API	Access to The Guardian's articles, tags and sections; great for text and news analysis, free with an API key.
Football API	Football (soccer) data API: fixtures, live scores, standings, teams, players and statistics across many leagues.
HSL Helsinki public transport	Open transit data for the Helsinki region: routes, schedules, GTFS feeds, real-time vehicle positions and a routing API.
Digitransit map API	Background map and vector tiles API for building transit and mobility maps (Finland/Helsinki region).
Open-Meteo	Free open-source weather API with forecasts, historical data and air quality, no API key required.
Finnish Meteorological Institute open data	Weather observations, forecasts and climate data for Finland, served via open WFS/download APIs.
OpenWeatherMap	Weather API with current conditions, forecasts and historical data via the One Call API; free tier plus paid plans.
SMARD German electricity market API	API for Germany's electricity market data: generation, consumption and prices from the Bundesnetzagentur (documentation in German).

General And Academic#

Content Marketing#

Crime#

Drugs#

Education#

Entertainment#

Environmental And Weather Data#

Financial And Economic Data#

Government And World#

Health#

Human Rights#

Labor And Employment Data#

Politics#

California Field Poll
Crowdpac
Gallup
Open Secrets
Rand State Statistics
Real Clear Politics
Roper Center for Public Opinion Research
US Voter Files Note only some states are free, and most do not allow voter files to be used for commercial purposes - this map allows you to see the rules/cost for each state.