Load shedding is a mechanism that web services often use to avoid overloading the network and making it unavailable for all users.

In distributed data center environments that rely on global server load balancing, load shedding plays an important role in providing efficient application delivery by preventing outages from saturation at the load balancer level.

NS1 is a DNS services provider that supports load shedding. It can use data directly from your load balancers, such as load metrics or connection counts, and ensure that none of them get pushed to the limits.

You can use HAProxy's Send Metrics module to provide NS1 with key metrics from your load balancers in real-time, thus offloading traffic to a different load balancer if the geographically closest one is too busy.

This section shows you how to configure DNS records in NS1 and your HAProxy load balancers for load shedding.

NOTE: NS1 refers to a geographically separated data center as a Points of Presence (POP).

Prerequisites

  • HAProxy Enterprise version 1.8 or newer

  • An NS1 account from https://ns1.com/

  • A purchased domain name

  • Two or more geographically dispersed instances of HAProxy Enterprise

Configure the NS1 records

When there are several IP addresses (i.e answer) for a given zone, NS1 must choose the best one based on metadata associated with each answer. Set up metadata to inform this decision.

  1. In the NS1 Customer Portal, set up a DNS zone (e.g. www.foo.com) and create A records within it for each of your load balancers.

  2. For each answer, click Edit Answer Metadata to display a settings window.

    [Edit the answer metadata]
  3. Set the appropriate Geographic region (e.g. "US-EAST").

    [Set the geographic region]
  4. Check that you associate each answer with a region.

    [Check the region added]
  5. Add metadata for the Up/down status and set it to "Up". This takes into consideration the status of the load balancer in addition to its geographic location.

Configure data feeds for HAProxy

Use the NS1 API to create data feeds where HAProxy can push data.

  1. Go to the NS1 portal and generate an API key. This key needs the following permissions:

    1. push to datafeeds

    2. manage datasources

    3. manage datafeeds

  2. Create a new data source with curl to call the /v1/data/sources NS1 API endpoint. Make sure you set your API key for the X-NSONE-Key HTTP header:

    $ curl -sH 'X-NSONE-Key: <API_KEY>' \
        -X PUT 'https://api.nsone.net/v1/data/sources' \
        -d '{"sourcetype": "nsone_v1", "name": "HA_PROXY_CONNECT"}'
    
    {"status": "ok", "name": "HA_PROXY_CONNECT", "feeds": [], "config": {}, "id": "760e670096f4f59dec045bed383aac5c", "sourcetype": "nsone_v1"}
  3. Create a new data feed for each of your load balancers:

    1. Call the /v1/data/feeds/[id] NS1 API endpoint.

    2. Set id in the URL to the "id" returned from the previous step.

    In the following example, we set name and label to "us-east", but you can choose any value. NS1 uses this when it selects the data feed to monitor for the current number of connections:

    curl -sH 'X-NSONE-Key: <API_KEY>' \
         -X PUT 'https://api.nsone.net/v1/data/feeds/760e670096f4f59dec045bed383aac5c' \
         -d '{"name": "us-east", "config": {"label": "us-east"}, "destinations": []}'
  4. Repeat the previous step to create a data feed for each load balancer, but change the name and label for each (e.g. "us-east", "us-west").

  5. Verify that the data feeds are listed in the NS1 Customer Portal on the Integrations window.

Associate DNS answers with data feeds

Add metadata to associate your A record answers with the data feeds.

  1. In the NS1 Customer Portal, edit the A records for the zone.

  2. Click Edit Answer Metadata for each answer and select Active connections.

  3. Click the "Feed" button to toggle the source of the data to become a feed.

  4. Choose the appropriate data feed for the current answer.

    [Choose the data feed]
  5. Add record metadata that applies to all answers:

    1. Specify a Low watermark that sets the threshold for when NS1 can begin shedding traffic away from a load balancer.

    2. Specify a High watermark that sets the threshold for when NS1 must completely stop sending traffic to a load balancer.

    [Set watermarks]

Configure a filter chain

When NS1 decides which IP address to return for a DNS query, it bases its decision on a chain of filters. Each filter discards answers that fail to match some criteria. Set a filter chain to inform NS1 on how to make its decision.

  1. In the NS1 Customer Portal, edit your A records and click Edit Filter Chain.

  2. Add the following filters:

    • Up

    • Geotarget Regional

    • Shed Load

    • Select First N

  3. In the Active Filters window, select:

    • the Shed Load filter and choose connections from the drop-down list.

    • the Select First N filter and set its value to 1.

  4. Save the filter chain. Verify that the filters are listed in the correct order as shown in the image below.

    [Verify the filter chain]

Configure HAProxy to send data

Use the HAProxy Send Metrics module to send the count of active connections for each load balancer to NS1.

  1. Log into each load balancer server and install the Send Metrics module.

  2. Edit the HAProxy configuration file to send connection information to NS1 by adding the following to the global section:

    module-path /opt/hapee-1.8/modules
    module-load hapee-lb-send-metrics.so
    send-metrics-url POST https://api.nsone.net/v1/feed/760e670096f4f59dec045bed383aac5c xdelay 1m 5s 1s 1s timeout 100ms retries 3 log verify none
    send-metrics-header 'X-NSONE-Key: <API_KEY>'
    send-metrics-content-type application/json
    send-metrics-data '{ "us-east": { "connections": "%ac" }}'
  3. Be sure to update the URL with your data source ID, the X-NSONE-Key header with your API key, and the label sent with the send-metrics-data line.

  4. Reload HAProxy.

  5. Verify that the HAProxy access logs show that the Send Metrics module sent data successfully:

    Send Metrics: metrics data successfully updated (1/0).
  6. Repeat this procedure for each load balancer.