AppSec California 2020

As a company scales, keeping track of user data becomes an increasingly hard problem to solve, as data is constantly generated and propagated across different data stores. With the rise of new privacy laws such as GDPR and CCPA, tackling this problem is more important than ever before. To address this challenge, we built a platform for data discovery and classification across all of our data stores, such as S3, MySQL and Hive, providing powerful privacy and security engineering capabilities. In this talk, we are going to share the experience we had building and operating this platform for Airbnb. We will present the high level architecture and technical specifics of the platform that allow it to leverage traditional algorithms and machine learning to scan petabytes of user data against growing numbers of data types, every single day.