M-Lab

Measurement Lab is a partnership between New America's Open Technology Institute, Google Inc., Princeton University's PlanetLab, and other supporting partners.

Learn more about M-Lab

Home
About
Visualizations
Data
- Overview
- Docs
- Platform Status
- Tools
Tests
Publications
Blog
Learn
Contribute
- Ways to Contribute
- Developer

Home / Blog / February 2018

New ETL Pipeline and Transition to New BigQuery Tables

Posted by Chris Ritzo on 2018-02-09
pipeline, bigquery, versioning

Since May 2017, the M-Lab team has been working on an updated, open source pipeline, which pulls raw data from our servers, saves it to Google Cloud Storage, and then parses it into our BigQuery tables. The team is particularly excited about this update because it means that the pipeline no longer relies on closed source libraries.

Categories

Archive

November 2018 (2)
October 2018 (2)
September 2018 (1)
July 2018 (3)
April 2018 (1)
February 2018 (1)
January 2018 (2)
August 2017 (1)
May 2017 (1)
April 2017 (1)
March 2017 (1)
November 2016 (1)
October 2016 (1)
May 2016 (1)
March 2016 (1)
January 2016 (1)
June 2015 (1)
April 2015 (1)
February 2015 (1)
November 2014 (2)
October 2014 (2)
March 2014 (2)
November 2013 (1)
September 2013 (1)
August 2013 (1)

GitHub
Twitter
Google
Email
RSS

About
Contact
Discuss List

FAQ
Blog
RSS

Code
Privacy Policy
Acceptable Use Policy

All original material on Measurement Lab is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 4.0 International License. M-Lab is a collaborative effort led by researchers in partnership with companies and other institutions.