How to Monitor the Price of Bitcoin/Chainlink (or any Crypto) with Datadog & AWS Lambda & Kraken

Intro

[If you don’t care to hear some background about Datadog skip to Step 0. Preparation section]

Trek10 has deep roots with Datadog. We use it extensively for our clients who trust us with supporting their AWS environment 24x7 and we also help clients implement Datadog with professional services. Depending on which generation you identify with you might call me one of the following terms:

• boomer/boomerII: Datadog advocate
• millennial: Datadog fanboy
• gen z: Datadog stan

I constantly get asked the question: “What is the most powerful feature/differentiator for Datadog?” I think people expect me to say a specific service Datadog offers such as “Their logging tool is great, lets you filter, etc.” but what I get jazzed up to say is “Datadog’s best feature and differentiator is how they let you visualize and monitor on any data inside of Datadog as much as you want for FREE”. That’s right you can create as many dashboards and monitors on the data inside of Datadog at no extra cost. If you are like most people my favorite feature is leaving you underwhelmed as a “best feature” … but I think the underwhelming sense comes from a lack of concrete examples. So in this post, I’m going to show off how simple it is to get data in Datadog and how to easily create powerful dashboards/monitors on said data.

Much to the chagrin of my financial advisor, family, and myself, I do dabble in purchasing Crypto coins/tokens. I have tried a couple of different ways to get alerts when the price of tokens/coins I follow have price swings and other noteworthy events. Each type of alert I have used has been helpful, but missing the total control I want- namely being able to name the alert, specify exactly when I get the alert, direct the alert in any format I want, and alert on topics other than just price. As one does when they can’t find the tool they want, they build it… or at least v1 of it! I went ahead and built that v1 and I will cover how to feed data into Datadog and create Datadog monitors over the price of both Chainlink (LINK) and Bitcoin (BTC) using Kraken’s API. There will be a bonus section that shows you a rudimentary way to get whale alerts (i.e. if there is a large purchase or sale). If crypto is not your cup of tea you can choose any data feed with an API polling endpoint and still use this blog as a guide.

Step 0. Preparation

What you will need:

• an AWS account
• a Datadog account
• an email address

What you will get:

• email alerts when price changes for LINK or BTC
• email alerts when there are rapid shifts in the amount of LINK or BTC purchased
• a dashboard to visualize LINK & BTC

What you will pay:

• <1$/mo

Step 1. The Kraken Work

We will be using Kraken to retrieve the pricing data for LINK and BTC. This section doesn’t include any necessary steps for the tutorial, but just explanations. If you wish to just copy/paste feel free to skip to Step 2.

If you want to use this as a tutorial for something other than LINK and BTC please use the documentation https://docs.kraken.com/rest/#operation/getTickerInformation and find the current parameter (i.e. ticker) for the CURL request. Otherwise, the steps below show where we get the price information that we will be pumping into Datadog.

• Open a Linux terminal
• Enter curl "https://api.kraken.com/0/public/Ticker?pair=LINKUSD" → this returns a JSON payload of the current market data for Chainlink in relation to US dollars
• Enter curl "https://api.kraken.com/0/public/Ticker?pair=BTCUSD" → this returns a JSON payload of the current market data for Bitcoin in relation to US dollars
• Your responses should look like:

• The format of this data is not intuitive so to understand so here an explanation for each key in the JSON (credit to algotrading101):
  • a = ask array(<price>, <whole lot volume>, <lot volume>)
  • b = bid array(<price>, <whole lot volume>, <lot volume>)
  • c = last trade closed array(<price>, <lot volume>)
  • v = volume array(<today>, <last 24 hours>)
  • p = volume weighted average price array(<today>, <last 24 hours>)
  • t = number of trades array(<today>, <last 24 hours>)
  • l = low array(<today>, <last 24 hours>)
  • h = high array(<today>, <last 24 hours>)
  • o = today’s opening price

• We will be using the values from c and v. Also make note of the key for the last nested JSON that represents the ticker/currency you picked:

• We will be using the price value from key c as the “current price” of LINK/BTC. It may not be the exact value the market is currently trading at, but it is the value of the last market trade which should be very close. (Note the irony of not using a price feed powered by LINK to determine the price of LINK is not lost on me LINKmarines).
• We will use the last 24 hours value from key v and the difference of that value from the value 1 hour ago to detect major shifts in the amount of LINK/BTC either bought or sold.

Step 2. The AWS Work

In this section, we will be setting up a lambda function to grab the price of LINK and BTC every 10 minutes and push that data into Datadog. This step is critical as we need the data inside of Datadog to start taking advantage of Datadog’s features.

I will be trying to explain the concepts and what is happening as I go. At the end of this step, I will give an overview with short and exact steps you need to take. If you get lost or confused try skipping to that section to get the end product and work backward.

• Go to AWS lambda console and click the Create Function button
• Fill in python 3.8, architecture is x86_64, and the name of your choosing (note I was in us-east-2)

• Go to the code source and we will start manually editing the code:
  • We will want to set up our way to grab the current price of LINK (we will do the time-honored tradition of writing this code for LINK and then copying/pasting for BTC once done)
    • add import json to the top (we will need to use this later)
    • add import urllib3 underneath
    • add http = urllib3.PoolManager() underneath
  • We will then want to grab the price from Kraken’s API for LINKUSD as we did in Step 1 inside the handler
    • add r = http.request("GET", "https://api.kraken.com/0/public/Ticker?pair=LINKUSD") right underneath the handler
  • The code should look like this:

• If you test the code with a dummy event it should look like this:
  

• As you can see the payload is a bytes string (denoted by the b)
• We need to get our data in a useable state, add the following lines beneath the request to Kraken’s API:
  • data=r.data.decode('UTF-8')
  • data=json.loads(data)
• Now we need to create variables to store the relevant information from the payload we have. For this demonstration, we will be using the last sale price as the current price and the current volume of LINK that has moved in the last 24 hours. To get the results from the nested json add the following lines beneath the data=json.loads(data) line. (Note that if you are wanting to monitor a different Crypto you will have a different ticker rather than LINKUSD to add here).
  • current_price=float(data["result"]["LINKUSD"]["c"][0])
  • last_24_volume=float(data["result"]["LINKUSD"]["v"][1])
• If you want you can try printing out the current price and testing the function.
• Your code should look like this:

import json
import urllib3
http = urllib3.PoolManager()

def lambda_handler(event, context):
    r = http.request("GET", "https://api.kraken.com/0/public/Ticker?pair=LINKUSD")
    data=r.data.decode('UTF-8')
    data=json.loads(data)
    current_price=float(data["result"]["LINKUSD"]["c"][0])
    last_24_volume=float(data["result"]["LINKUSD"]["v"][1])
    print(current_price)

• Now that we have the data from Kraken inside our lambda function and assigned to variables our next step is to send it to Datadog. To do that we will use AWS lambda layers.
• Specifically, Datadog has its own lambda layer. Scroll up to the Layers section of the AWS function you’ve been working on and hit the Layers icon inside the function overview. Click Add Layer
  • Choose the option to specify an ARN

• You will want to add the ARN by following the format listed here, for me it was arn:aws:lambda:us-east-2:464622532012:layer:Datadog-Python38:50 since I choose us-east-2 to work in (if you choose a different region, substitute the region, same for python version, etc.)
• Hit Add
• Now we can use this Datadog Blog as a starting point to send our metrics.
  
  • add from datadog_lambda.metric import lambda_metric to the top of your file
  • add from datadog_lambda.wrapper import datadog_lambda_wrapper to the top of your file
  • add this line before your handler function @datadog_lambda_wrapper
  • add the following lines at the end of your handler function

lambda_metric(
        "chainlink.current_price",              # Metric name
        current_price,                          # Metric value
        tags=['ticker:LINK']                    # Associated tags
    )
    lambda_metric(
        "chainlink.last_24_volume",              # Metric name
        last_24_volume,                          # Metric value
        tags=['ticker:LINK']                    # Associated tags
    )
    

• At this point your code should look like this:

import json
import urllib3
http = urllib3.PoolManager()
from datadog_lambda.metric import lambda_metric
from datadog_lambda.wrapper import datadog_lambda_wrapper


@datadog_lambda_wrapper
def lambda_handler(event, context):
    r = http.request("GET", "https://api.kraken.com/0/public/Ticker?pair=LINKUSD")
    #print(r.data.decode('UTF-8'))
    data=r.data.decode('UTF-8')
    data=json.loads(data)
    #print(data)
    current_price=float(data["result"]["LINKUSD"]["c"][0])
    last_24_volume=float(data["result"]["LINKUSD"]["v"][1])
    print(current_price)
    lambda_metric(
        "chainlink.current_price",              # Metric name
        current_price,                          # Metric value
        tags=['ticker:LINK']                    # Associated tags
    )
    lambda_metric(
        "chainlink.last_24_volume",              # Metric name
        last_24_volume,                          # Metric value
        tags=['ticker:LINK']                    # Associated tags
    )
    return {
        'statusCode': 200,
        'body': json.dumps('Hello from Lambda!')
    }

• Create an env variable and insert your API key

• Now we want to send BTC metrics along with the LINK metrics - copy everything from the top of the handler r =... to the two lambda_metric functions inclusive and paste it again.
  • Now change the URL from r = http.request("GET", "https://api.kraken.com/0/public/Ticker?pair=LINKUSD") to r = http.request("GET", "https://api.kraken.com/0/public/Ticker?pair=BTCUSD")
  • In the two lines where we are assigning values to current_price and last_24_volume change the keys from LINKUSD to XXBTZUSD
  • In the lines where we mentioned chainlink.current_price and chainlink.last_24_volume change those names to bitcoin.xxx and change the tag for ticker:LINK to ticker:BTC
  • Your lambda code should now mirror this:
    • https://github.com/trek10inc/datadog-crypto-monitoring-example/blob/master/lambda_code/lambda_function.py
    • If you want, you can deploy the code and test the code (the test event can be anything as the lambda function does not use it)
  • To recap, we have a function that reaches out to Kraken, grabs the pricing information, and sends this information to Datadog. All we need to do is set it up on a schedule to do this every 10 minutes.
  • Scroll back up to the Function overview section and click add trigger.
    • Select CloudWatch Events
    • Enter any name you want
    • Select Schedule expression and enter cron(0,10,20,30,40,50 * ? * * *)
    • It should look like:

• Click Add
• Hit the Deploy button
  
• Boom. Now sit back and relax. We have accomplished the following:
  • Creating a lambda function
  • Having it reach out to Kraken to grab pricing info for LINK and BTC
  • Send that information to Datadog
  • Set a schedule for the lambda to run every 10 minutes
• Confirm the metrics are coming into Datadog by manually invoking your function 3-5 times and going to https://app.datadoghq.com/metric/explorer?exp_metric=chainlink.current_price&exp_agg=avg&exp_row_type=metric and confirming you see data points
• Ideally, let the function run two or three hours before moving to the next step

Recap of Step 2:

1. Create a lambda function using python 3.8 in the console, named what you want, x86_64
2. Paste code from https://github.com/trek10inc/datadog-crypto-monitoring-example/blob/master/lambda_code/lambda_function.py
3. In function overview pane hit Add layer → arn:aws:lambda:<<your_region>>:464622532012:layer:Datadog-Python38:50
4. In the configuration tab select Environment variables → add env variable in console named DD_API_KEY with an API key found at: https://app.datadoghq.com/organization-settings/api-keys
5. In function overview pane hit Add trigger → choose EventBridge (Cloud Watch Events) → schedule expression → use `cron(0,10,20,30,40,50 * ? * * *) as the expression
6. Hit Deploy

Step 3. The Datadog Work

This is the section where the magic happens. We will cover how to create a customizable dashboard that lets you look at the historical pricing data in any way you want, how to create alerts for price changes in LINK and BTC, and how to create alerts when there are large drops or upswings in the amount of BTC/LINK bought (which could indicate a whale move). Gone will be the days of only being able to see prebuilt graphs of price data or only being able to have a set number of alerts set up. This section will give you control of the visualization of pricing data and alerting on it.

• First, we will go over a dashboard
  • Login to Datadog
  • Navigate to the Dashboards section
  • Select New Dashboard
  • In the upper right corner click the gear icon
  • Click Import dashboard JSON
  • It should look like this:

• Copy the json from https://github.com/trek10inc/datadog-crypto-monitoring-example/blob/master/datadog_dashboard_json/main_dash.json and paste it in
• Your dashboard should now look like this:

• We will go over one “widget” in the dashboard to explain it for those who are curious
  • Click on the pencil icon on the How much Chainlink has been bought in the last hour widget
  • You will see we are using two different metrics to make this graph. The first is the Chainlink metric we are submitting via our lambda function. The second is that metric from an hour ago. We are then graphing the difference between now vs one hour ago. This gives us a visual representation of if there has been an uptake in the amount of LINK bought in the last hour or a fall off.

• You can see from this dashboard is pretty bare-bones, but it will make it easy to see historical trends, price swings, swings in the amount of LINK/BTC bought, etc.
• But you now have the entire arsenal of Datadog functions at your fingertips. Timeshift? Regressions? Anomaly detection? These are all available to you if you select to graph the pricing metric and click the sigma button and select from the possible options.

• Next up we have monitoring. I will cover how to create one monitor, specifically if the price of BTC goes up. Then I will give you dashboard JSONs to copy/create monitors for: price going down for BTC, price going up for LINK, price going down for LINK, a large amount of BTC bought over the last hour, and a large amount of LINK bought in the last hour.
  • Head over to create new monitor and select metric monitor

• Under section 1 Choose the detection method select Threshold Alert
• Under section 2 Define the metrics start typing bitcoin.current_price (the option to autocomplete should pop up)
• Under section 3 Set alert conditions select when the metric is above / at least once and 15 minutes
  • These are variables are what you would want if you wanted to know when the price was going above a certain threshold. By changing them you change the fundamentals of the alert. IE if you change the variable for above to below you would change the alert from being when the price goes up to when the price falls.
  • For the alert threshold, I picked 46000 and 45000 for the warning threshold (at the time of writing this BTC was trading at ~43000)
  • Change the setting to Notify if data is missing and select 30 minutes. We will use this to let us know if our lambda stops running for some reason
• Leave section 4 Notify your team as it is… we will be manually adding alerts in section 5.
• Section 5 the Say what's happening section. This section is what allows you to customize the message you send out and which notification endpoints get which messages. I would strongly recommend reading Datadog’s documentation to see all that is available to you.
  • We will be setting the alarm up to email us at the warning threshold, texting us at the alert threshold, and emailing us if the lambda function stops working.
  • Set a title
  • Inside the writing block enter:

{{#is_alert}}
Bitcoin Price is really really growing!
enter_your_phone_#_here@txt.att.net (change email after @ depending on your provider T-Mobile: @tmomail.net Sprint: @messaging.sprintpcs.com Verizon @vtext.com )
{{/is_alert}} 

{{#is_warning}}
Bitcoin Price is growing!
enter_your_email_here@gmail.com
{{/is_warning}} 

{{#is_no_data}}
The price of bitcoin is no longer getting reported to Datadog - check out the lambda function to make sure all is working! enter_your_email_here@gmail.com
{{/is_no_data}} 

• As you can see Datadog’s custom mark-up language lets you filter alerts depending on certain conditions. The example above is fairly basic, but the mark-up language is very powerful and you can route alerts based on tags, match statements, etc.
• Confirm your monitor looks like this:

• Click Create!
• Congrats you now have a monitor that alerts you BTC’s price goes up!
• Here is a gitlab link to the JSON objects for the other monitors mentioned above (price increases/decreases and when large amounts of LINK/BTC are bought). To create monitors off of the JSON objects, go to a new monitor and instead of selecting metric monitor select Import Monitor From JSON

Step 4. The Wrap Up

So now we have accomplished the following: getting pricing information on LINK/BTC, getting that information from Kraken into Datadog via AWS, and visualizing and monitoring off of that data. Now all that is left is to figure out how much we are spending on this solution and extrapolate what this solution could do.

• As far as cost, this solution will run you all-in less than one dollar.
  • AWS Lambda ~$0.006/month based on ~4380 invocations at 256MB (see https://dashbird.io/lambda-cost-calculator/ to tweak your numbers if you are running more frequently)
  • Datadog pricing ~$0.00 based on the fact that Datadog does NOT charge per monitor or dashboard, but rather based on the resources being monitored. Assuming you have at least one Datadog host you will have 100 custom metrics you can use for free.
  • Kraken API $0.00 - it is free! Send them a thank you tweet!

With less than one hour’s worth of effort, you can use Datadog to text you when the price of BTC/LINK drops. You could go on to create monitors based on anomaly detection for the price of LINK. Or you could even have one of those Datadog monitors invoke a webhook that calls another lambda function to execute a purchase order to buy more LINK. The powerful monitoring engine and dashboarding visualization tools are completely free once data is inside Datadog. Trek10 uses these tools to get our managed service clients real-time alerts about the health of their AWS account and even call webhooks to execute client runbooks.

If your company is looking for help implementing Datadog, building automatic remediation based on AWS events, or looking for someone to manage your AWS account for you - please contact our team by filling out the form located here. Trek10 is proud to be a Gold Datadog Partner and Managed Service Provider! Choose the right AWS consulting partner (hint: we think it's us).