Load Shedding with NS1

Prerequisites

• An NS1 account from https://ns1.com/

• A domain name to add DNS records for

• Two or more geographically dispersed instances of HAProxy Enterprise

Configure the NS1 records

When there are several IP addresses (i.e answer) for a given zone, NS1 must choose the best one based on metadata associated with each answer. Set up metadata to inform this decision.

1. In the NS1 Customer Portal, set up a DNS zone (e.g. www.foo.com) and create A records within it for each of your load balancers.

2. For each answer, click Edit Answer Metadata to display a settings window.

   

3. Set the appropriate Geographic region (e.g. "US-EAST").

   

4. Check that you associate each answer with a region.

   

5. Add metadata for the Up/down status and set it to "Up". This takes into consideration the status of the load balancer in addition to its geographic location.

Configure data feeds for HAProxy

Use the NS1 API to create data feeds where HAProxy Enterprise can push data.

1. Go to the NS1 portal and generate an API key. This key needs the following permissions:

   1. push to datafeeds

   2. manage datasources

   3. manage datafeeds

2. Create a new data source with curl to call the /v1/data/sources NS1 API endpoint. Make sure you set your API key for the X-NSONE-Key HTTP header:

   $ curl -sH 'X-NSONE-Key: <API_KEY>' \
       -X PUT 'https://api.nsone.net/v1/data/sources' \
       -d '{"sourcetype": "nsone_v1", "name": "HA_PROXY_CONNECT"}'
   
   {"status": "ok", "name": "HA_PROXY_CONNECT", "feeds": [], "config": {}, "id": "760e670096f4f59dec045bed383aac5c", "sourcetype": "nsone_v1"}

3. Create a new data feed for each of your load balancers:

   1. Call the /v1/data/feeds/[id] NS1 API endpoint.

   2. Set id in the URL to the "id" returned from the previous step.

   In the following example, we set name and label to "us-east", but you can choose any value. NS1 uses this when it selects the data feed to monitor for the current number of connections:

   $ curl -sH 'X-NSONE-Key: <API_KEY>' \
        -X PUT 'https://api.nsone.net/v1/data/feeds/760e670096f4f59dec045bed383aac5c' \
        -d '{"name": "us-east", "config": {"label": "us-east"}, "destinations": []}'

4. Repeat the previous step to create a data feed for each load balancer, but change the name and label for each (e.g. "us-east", "us-west").

5. Verify that the data feeds are listed in the NS1 Customer Portal on the Integrations window.

Associate DNS answers with data feeds

Add metadata to associate your A record answers with the data feeds.

1. In the NS1 Customer Portal, edit the A records for the zone.

2. Click Edit Answer Metadata for each answer and select Active connections.

3. Click the "Feed" button to toggle the source of the data to become a feed.

4. Choose the appropriate data feed for the current answer.

   

5. Add record metadata that applies to all answers:

   1. Specify a Low watermark that sets the threshold for when NS1 can begin shedding traffic away from a load balancer.

   2. Specify a High watermark that sets the threshold for when NS1 must completely stop sending traffic to a load balancer.

   

Configure a filter chain

When NS1 decides which IP address to return for a DNS query, it bases its decision on a chain of filters. Each filter discards answers that fail to match some criteria. Set a filter chain to inform NS1 on how to make its decision.

1. In the NS1 Customer Portal, edit your A records and click Edit Filter Chain.

2. Add the following filters:

   • Up

   • Geotarget Regional

   • Shed Load

   • Select First N

3. In the Active Filters window, select:

   • the Shed Load filter and choose connections from the drop-down list.

   • the Select First N filter and set its value to 1.

4. Save the filter chain. Verify that the filters are listed in the correct order as shown in the image below.

   

Configure HAProxy Enterprise to send data

Use the HAProxy Enterprise Send Metrics module to send the count of active connections for each load balancer to NS1.

1. Log into each load balancer server and install the Send Metrics module.

2. Edit the HAProxy Enterprise configuration file to send connection information to NS1 by adding the following to the global section:

   global
     # ...
     module-path /opt/hapee-1.8/modules
     module-load hapee-lb-send-metrics.so
          send-metrics-url POST https://api.nsone.net/v1/feed/760e670096f4f59dec045bed383aac5c xdelay 1m 5s 1s 1s timeout 100ms retries 3 log verify none
          send-metrics-header 'X-NSONE-Key: <API_KEY>'
     send-metrics-content-type application/json
     send-metrics-data '{ "us-east": { "connections": "%ac" }}'

3. Be sure to update the URL with your data source ID, the X-NSONE-Key header with your API key, and the label sent with the send-metrics-data line.

4. Reload HAProxy Enterprise.

5. Verify that the HAProxy Enterprise access logs show that the Send Metrics module sent data successfully:

   Send Metrics: metrics data successfully updated (1/0).

6. Repeat this procedure for each load balancer.