SIGCOMM 2018 Hackathon Outcomes
M-Lab had the pleasure of attending the first ever SIGCOMM hackathon on August 25, 2018, at the Nokia Skypark headquarters in Budapest, Hungary. The hackathon, sponsored by Nokia, DECIX, and Netflix, invited network research faculty, students, and industry professionals from around the world to form teams and develop tools, new features or analyses during the Saturday following the SIGCOMM conference.
M-Lab 10 Year Anniversary Event
We’ve reached a point in human history where, for many of us, the internet has become a standard presence in our daily lives. In the United States, the internet is simply part of how many of us engage with the world. In other countries (and parts of this one), the internet remains unaffordable, unreliable, and inaccessible. The internet unites us in many ways, and at the center of work on the future of the internet is a dedicated community of experts exploring the questions that will move the internet to the next level of its evolution: What is an open internet? What is a healthy internet? What factors contribute to the internet ecosystem’s health?
Switch discard investigation for early 2018
On February 1st, 2018, during a regular data quality review, we identified an increase in switch discards at sites with 10Gbps equipment connected to 1Gbps uplinks. We used our switch telemetry data to assess whether there were any negative consequences for tests contained in our SideStream or NDT data sets, and then we used the same data sets to determine whether our remediation strategy had any negative effects. In both cases, we found no observable effects, indicating that everything was below the noise floor for Internet performance data.
SIGCOMM 2018 Hackathon
- When: Saturday, August 25, 2018
- Where: SIGCOMM, Budapest, Hungary
M-Lab will be at this year’s SIGCOMM hackathon in Budapest, Hungary. Check out the projects we’re hoping to work on with folks on the hackathon wiki.
M-Lab is Turning 10!
- When: Aug. 7, 2018 - Aug. 8, 2018, 9AM - 5PM
- Where: New America, 740 15th St NW #900, Washington, D.C. 20005
Measurement Lab is turning 10! On August 7 and 8, we look forward to gathering the Measurement Lab community to showcase how the platform has evolved, learn from you about how you are using M-Lab, and discuss how we plan for the next 10 years of measuring the Internet and providing public data to the world. So much has changed over the last 10 years (and that’s not just our expanding volume of longitudinal data!), come celebrate, brainstorm, analyze, and share with us.
New DISCO switch telemetry dataset
Since June 2016, M-Lab has collected high resolution switch telemetry for each M-Lab server and site uplink.
Originally designed to detect switch discards from server traffic microbursts, we now support the DIScard COllection (a.k.a. DISCO) dataset as a standard M-Lab BigQuery table: measurement-lab.base_tables.switch
New ETL Pipeline and Transition to New BigQuery Tables
Since May 2017, the M-Lab team has been working on an updated, open source pipeline, which pulls raw data from our servers, saves it to Google Cloud Storage, and then parses it into our BigQuery tables. The team is particularly excited about this update because it means that the pipeline no longer relies on closed source libraries.
Transitioning to a New Backend Pipeline and Data Availability
M-Lab data is collected from distributed experiments hosted on servers all over the world, processed in a pipeline, and published for free in both raw and parsed (structured) formats. The back end processing component for this has served us well for many years, but it’s been showing its age recently. As M-Lab collects an increasing amount of data thanks to new partnerships, we have been concerned that it will not be as reliable.
Making it Easier to Use M-Lab Data
In January, M-Lab launched a beta test of new BigQuery tables for M-Lab data. Today, M-Lab is pleased to announce that the beta test was successful. The new, faster-performing tables will be M-Lab’s new standard BigQuery tables.
Before we move on to specifics, when we say faster performing, we mean a lot faster. As in, certain queries that used to take over 2 hours now complete in 8 seconds. That means that playing with the data just became a lot more fun.
To help users dig in to this data as quickly and seamlessly as possible, M-Lab has consolidated all of its data documentation and updated it to show how to take advantage of the new tables.
Announcing improved performance for M-Lab BigQuery data
Today, M-Lab is happy to announce the public beta of new M-Lab BigQuery tables. These tables provide substantially improved performance and reduce the difficulty of writing BigQuery SQL.