BGPKIT Broker Backend

BGPKIT Broker Backend is the backend API and data collection implementation for the BGPKIT Broker project.

Users can use this repository to deploy a BGPKIT Broker instance on their own equipment, and it is free under MIT license! See more in Deployment and LICENSE sections.

Table of Content

• BGPKIT Broker API
• Data Sources
• Deployment
• LICENSE

BGPKIT Broker API

At a high level, the BGPKIT Broker is a REST API service that provides search capability for BGP MRT archive files from various data collection projects. An user can search for matching archive files with simple REST API calls.

For example, we want to learn about exactly what BGP updates were involved that caused Facebook’s outage for multiple hours on Oct. 4th, 2021. We can use the following REST call to the BGPKIT Broker API to gather the full list of data files that may contain the relevant updates for the event:

GET https://api.broker.bgpkit.com/v2/search?ts_start=1633362000&ts_end=1633363200&data_type=update&page_size=5&page=1

{
  "count": 5,
  "page": 1,
  "page_size": 5,
  "error": null,
  "data": [
    {
      "ts_start": "2021-10-04T15:30:00+00:00",
      "ts_end": "2021-10-04T15:45:00+00:00",
      "collector_id": "route-views2",
      "data_type": "update",
      "url": "http://archive.routeviews.org/bgpdata/2021.10/UPDATES/updates.20211004.0830.bz2",
      "rough_size": 1153433,
      "exact_size": 0
    },
    {
      "ts_start": "2021-10-04T15:30:00+00:00",
      "ts_end": "2021-10-04T15:45:00+00:00",
      "collector_id": "route-views3",
      "data_type": "update",
      "url": "http://archive.routeviews.org/route-views3/bgpdata/2021.10/UPDATES/updates.20211004.0830.bz2",
      "rough_size": 8388608,
      "exact_size": 0
    },
    {
      "ts_start": "2021-10-04T15:30:00+00:00",
      "ts_end": "2021-10-04T15:45:00+00:00",
      "collector_id": "route-views5",
      "data_type": "update",
      "url": "http://archive.routeviews.org/route-views5/bgpdata/2021.10/UPDATES/updates.20211004.0830.bz2",
      "rough_size": 22528,
      "exact_size": 0
    },
    {
      "ts_start": "2021-10-04T15:30:00+00:00",
      "ts_end": "2021-10-04T15:45:00+00:00",
      "collector_id": "route-views.amsix",
      "data_type": "update",
      "url": "http://archive.routeviews.org/route-views.amsix/bgpdata/2021.10/UPDATES/updates.20211004.0830.bz2",
      "rough_size": 7864320,
      "exact_size": 0
    },
    {
      "ts_start": "2021-10-04T15:30:00+00:00",
      "ts_end": "2021-10-04T15:45:00+00:00",
      "collector_id": "route-views.chicago",
      "data_type": "update",
      "url": "http://archive.routeviews.org/route-views.chicago/bgpdata/2021.10/UPDATES/updates.20211004.0830.bz2",
      "rough_size": 1887436,
      "exact_size": 0
    }
  ]
}
// GET https://api.broker.bgpkit.com/v2/search?ts_start=1633362000&ts_end=1633363200&data_type=update&page_size=5&page=1
// HTTP/1.1 200 OK
// Content-Length: 1395
// Content-Type: application/json
// Date: Mon, 25 Apr 2022 21:54:29 GMT
// Server: Caddy
// Server: uvicorn
// Request duration: 1.434210s

Here is a break-down of the call parameters:

• ts_start=1633362000: all BGP files after 15:40 UTC on that day
• ts_end=1633362022: all BGP files before 16:00 UTC on that day
• data_type=update: only return updates files
• page_size=5: return 5 items per page
• page=1: return the 1st page (pages starting from 1)

For more examples and API spec, please checkout our documentation site.

Data Sources

Broker backend currently support two data collection projects: RouteViews and RIPE RIS. Specifically, we have designed tailored data collection procedures to efficiently crawl the data publication sites from the two projects and collect basic information of the published BGP MRT files.

The information we currently collect for each data file includes:

• data type: either BGP updates files, or BGP table dump files.
• timestamp: the time of the beginning of the data dump, usually aligns on the full five-minute marks.
• url: the URL for the downloadable data file.
• collector ID: the ID of the data collector where that data comes from.
• rought and exact file sizes

Note that the data dump frequencies for RouteViews and RIPE RIS are different. Here is a table summarizing their frequencies:

Project	Update frequency	Table dump frequency
RouteViews	one per 15 minutes	one per 2 hours
RIPE RIS	one per 5 minutes	one per 8 hours

In essence, you would see more frequent BGP updates dumps from RIPE RIS and more frequent full table dumps from RouteViews.

Deployment

Here is a step-by-step guide for deploying BGPKIT Broker with Docker

1. Checkout the repository
2. run docker-compose -f deployment/docker-compose.yml -p bgpkit-broker up

The initial database bootstrap phase would take about 30 minutes to one hour depending on your deployment environment. It requires at least 30GB of free storage for the container and build processes.

After the initial bootstrap phase is done, the API service should be up and running, currently hosted at port 18888. You can modify the port in docker-compose.yml file. In the mean time, a cronjob service also started, crawling collectors for recent data every 5 minutes. The frequency can be configured in the update.cron. It is not recommended to go more frequent than one crawl per 5 minutes.

You can check out if the API is running by running:

curl 0.0.0.0:18888/latest
curl "0.0.0.0:18888/search?ts_start=1643673600&ts_end=2022-02-02&collector_id=rrc00"

LICENSE

BGPKIT Broker backend code is under MIT LICENSE.

Built with ❤️ by BGPKIT Team

BGPKIT is a small-team focuses on building the best open-source tooling for BGP data processing and analysis in Rust. Learn more about what we are offering at https://github.com/bgpkit.