Splunk Dev

REST modular input JSON custom handler for AWS Pricing Data

pgreer_splunk
Splunk Employee
Splunk Employee

Having a bit of a struggle. AWS has a pricing API available at:

AWS JSON Pricing API URL

Because of how the JSON is formatted, it looks like a custom handler is needed. Snipped of the JSON is:

{
  "formatVersion" : "v1.0",
  "disclaimer" : "This pricing list is for informational purposes only. All prices are subject to the additional terms included in the pricing pages on http://aws.amazon.com. All Free Tier prices are also subject to the terms included at https://aws.amazon.com/free/",
  "offerCode" : "AmazonEC2",
  "version" : "20170921013650",
  "publicationDate" : "2017-09-21T01:36:50Z",
  "products" : {
    "76V3SF2FJC3ZR3GH" : {
      "sku" : "76V3SF2FJC3ZR3GH",
      "productFamily" : "Compute Instance",
      "attributes" : {
        "servicecode" : "AmazonEC2",
        "location" : "Asia Pacific (Mumbai)",
        "locationType" : "AWS Region",
        "instanceType" : "d2.4xlarge",
        "currentGeneration" : "Yes",
        "instanceFamily" : "Storage optimized",
        "vcpu" : "16",
        "physicalProcessor" : "Intel Xeon E5-2676v3 (Haswell)",
        "clockSpeed" : "2.4 GHz",
        "memory" : "122 GiB",
        "storage" : "12 x 2000 HDD",
        "networkPerformance" : "High",
        "processorArchitecture" : "64-bit",
        "tenancy" : "Host",
        "operatingSystem" : "Windows",
        "licenseModel" : "No License required",
        "usagetype" : "APS3-HostBoxUsage:d2.4xlarge",
        "operation" : "RunInstances:0002",
        "ecu" : "56",
        "enhancedNetworkingSupported" : "Yes",
        "normalizationSizeFactor" : "32",
        "preInstalledSw" : "NA",
        "processorFeatures" : "Intel AVX; Intel AVX2; Intel Turbo",
        "servicename" : "Amazon Elastic Compute Cloud"
      }
    },
    "G2N9F3PVUVK8ZTGP" : {
      "sku" : "G2N9F3PVUVK8ZTGP",
      "productFamily" : "Compute Instance",
      "attributes" : {
        "servicecode" : "AmazonEC2",
        "location" : "Asia Pacific (Seoul)",
        "locationType" : "AWS Region",
        "instanceType" : "i2.xlarge",
        "currentGeneration" : "No",
        "instanceFamily" : "Storage optimized",
        "vcpu" : "4",
        "physicalProcessor" : "Intel Xeon E5-2670 v2 (Ivy Bridge)",
        "clockSpeed" : "2.5 GHz",
        "memory" : "30.5 GiB",
        "storage" : "1 x 800 SSD",
        "networkPerformance" : "Moderate",
        "processorArchitecture" : "64-bit",
        "tenancy" : "Host",
        "operatingSystem" : "Windows",
        "licenseModel" : "No License required",
        "usagetype" : "APN2-HostBoxUsage:i2.xlarge",
        "operation" : "RunInstances:0102",
        "ecu" : "14",
        "enhancedNetworkingSupported" : "Yes",
        "normalizationSizeFactor" : "8",
        "preInstalledSw" : "SQL Ent",
        "processorFeatures" : "Intel AVX; Intel Turbo",
        "servicename" : "Amazon Elastic Compute Cloud"
      }
....

I added a handler to responsehandlers.py as:

#handler for AWS pricing API call, split and print out all product stanzas
class AWSPriceHandler:

    def __init__(self,**args):
        pass

    def __call__(self, response_object,raw_response_output,response_type,req_args,endpoint):
        if response_type == "json":
            output = json.loads(raw_response_output)
            for product in output["products"]:
                print_xml_stream(json.dumps(product))
        else:
            print_xml_stream(raw_response_output)

But that isn't working for me. It doesn't ingest any events at all when referring to that handler. Take the handler out, and the ingest is a single event that gets truncated at the line length limit.

Any suggestions?

0 Karma

Damien_Dallimor
Ultra Champion

What does your configuration look like , post a screenshot or copy/paste the inputs.conf entry.

0 Karma
Get Updates on the Splunk Community!

Infographic provides the TL;DR for the 2023 Splunk Career Impact Report

We’ve been shouting it from the rooftops! The findings from the 2023 Splunk Career Impact Report showing that ...

Splunk Lantern | Getting Started with Edge Processor, Machine Learning Toolkit ...

Splunk Lantern is Splunk’s customer success center that provides advice from Splunk experts on valuable data ...

Enterprise Security Content Update (ESCU) | New Releases

In the last month, the Splunk Threat Research Team (STRT) has had 2 releases of new security content via the ...