Data Quality Checks in Pure SQL: Auditing 3 Million Taxi Rides in 115 Milliseconds
· 8 min read
Three million taxi rides, nine quality checks, 115 milliseconds. The dataset failed three of them.
The NYC yellow-taxi data is probably the most-analyzed public dataset in existence. It has starred in a thousand benchmarks and ten thousand tutorials. It also contains 124 exact duplicate rides, a fare of −$899, and a trip that claims to have happened in 2002 — in the January 2024 file.
None of that is a criticism of the TLC. It's what real data looks like. The interesting question is how fast you can find it.