topic Re: Getting JSON-data in Splunk (preferably with Streamsets) in Getting Data In

Getting JSON-data in Splunk (preferably with Streamsets)

JosIJntema — Mon, 16 Jan 2017 10:15:46 GMT

Hi all,

I am new to Splunk and am struggling to get this to work.

I use Streamsets to add data to my streams. For now I save it in a dummy MongoDB, but I would like to save it directly into Splunk.

I know about the HTTP Event Collector, but how should I make this connections? What headers and URL should I use?

Thanks!

Re: Getting JSON-data in Splunk (preferably with Streamsets)

metadaddy — Tue, 17 Jan 2017 20:42:27 GMT

I was able to make this work using the StreamSets HTTP Client processor, and the Splunk receivers/simple REST API. Modeling my HTTP client after the Splunk example, I used the following settings:

Resource URL: http://localhost:8089/services/receivers/simple?source=${record:value('/source')}&sourcetype=${record:value('/sourcetype')} (You can hardcode source, sourcetype if you like)
HTTP Method: POST
Request Data: ${record:value('/timestamp')} User ${record:value('/username')} logged in successfully. (Change this to whatever you want to send to Splunk)
Default Request Content Type: application/x-www-form-urlencoded
Authentication Type: Basic

You'll need to configure an appropriate username/password in the Credentials tab. I set the data format to XML and passed the API response to a 'Local FS' destination for debugging.

Note - for simplicity, I turned off SSL on the Splunk REST API.

EDIT: The above mechanism works record-by-record, so it's not very efficient for large amounts of data. A better approach is to use a script to send a single request per batch to the Splunk HTTP Event Collector:

import sys
# Set to wherever the requests package lives on your machine
sys.path.append('/Library/Python/2.7/site-packages')
import requests
import json

# Endpoint for Splunk HTTP Event Collector
url = 'http://localhost:8088/services/collector'

# Read Splunk token from file and cache in state
if state.get('headers') is None:
  state['headers'] = {'Authorization': 'Splunk ${runtime:loadResource('splunkToken', false)}'}

buffer = ''

# Loop through batch, building request payload
for record in records:
  try:
    # Strip host & time fields from record and pass to Splunk as event attributes
    event = dict((key, record.value[key]) for key in record.value if key not in ['time', 'host'])
    buffer += json.dumps({
      'host': record.value['host'],
      'time': record.value['time'],
      'event': event,
    }) + '\n'

    # Write record to processor output
    output.write(record)

  except Exception as e:
    # Send record to error
    error.write(record, str(e))

# Now submit a single request for the entire batch
r = requests.post(url, 
                  headers=state['headers'],
                  data=buffer).json()

# Check for errors from Splunk
if r['code'] != 0:
  log.error('Splunk error: {}: {}', r['code'], r['text'])
  raise Exception('Splunk API error {0}: {1}'.format(r['code'], r['text']))

# All is good
log.info('Splunk API response: {}', r['text'])

Re: Getting JSON-data in Splunk (preferably with Streamsets)

metadaddy — Wed, 18 Jan 2017 20:01:01 GMT

I wrote this up more fully at https://streamsets.com/blog/ingest-data-splunk-streamsets-data-collector/

Re: Getting JSON-data in Splunk (preferably with Streamsets)

aaraneta_splunk — Mon, 23 Jan 2017 02:21:34 GMT

@JosIJntema - Were you able to test out metadaddy's solution? Did it work? If yes, please don't forget to resolve this post by clicking on "Accept". If you still need more help, please provide a comment with some feedback. Thanks!

Re: Getting JSON-data in Splunk (preferably with Streamsets)

nareshinsvu — Wed, 17 Apr 2019 02:48:37 GMT

Hi @metadaddy

I have a similar requirement. I dont have any knowledge on json / java.

Requirement is to pass everything from a JSON stream (eg: everything from https://10.24.113.206/container-ws/hystrix.stream) to Splunk. I have created token for HEC in splunk and googling all blogs to get me work done.

What should my syntax be if I follow your below example to send everything to Splunk

Request Data: ${record:value('/timestamp')} User ${record:value('/username')} logged in successfully. (Change this to whatever you want to send to Splunk)

Re: Getting JSON-data in Splunk (preferably with Streamsets)

metadaddy — Wed, 17 Apr 2019 21:16:56 GMT

Hi @nareshinsvu - you could try the Splunk destination - that was created since I answered this question. Also, we have a few options for our community to interact with the StreamSets team directly - see https://streamsets.com/community/