Getting Data In

Splunk TA TypeError for multiple data inputs

Path Finder

Hey Splunk Community,

I am in the process of creating a TA with Splunk Add-On Builder and I have run into a problem for which I cannot seem to find an answer for either on here or a related on through online searches.

I have created a modular input using a Python script, and it is feeding data into Splunk fine, parsing the data, extracting fields, everything with no issues. Some background, a user enters environment details for set-up, then a system ID per individual data input to collect stats from. It works fine for just 1 input, but when I try to add a second I get the following error in splunkd.log:

2016-11-21 11:14:16,522 ERROR pid=15294 tid=MainThread | Get error when collecting events.
Traceback (most recent call last):
File "/opt/stack/splunk/etc/apps/TA-test-01/bin/TA-test-01/modinput_wrapper/", line 173, in stream_events
self.collect_events(inputs, ew)
File "/opt/stack/splunk/etc/apps/TA-test-01/bin/", line 53, in collect_events
input_module.collect_events(self, inputs, ew)
File "/opt/stack/splunk/etc/apps/TA-test-01/bin/", line 56, in collect_events
File "/opt/stack/splunk/etc/apps/TA-test-01/bin/TA-test-01/splunklib/modularinput/", line 60, in write_event
File "/opt/stack/splunk/etc/apps/TA-test-01/bin/TA-test-01/splunklib/modularinput/", line 106, in write_to
File "/opt/stack/splunk/lib/python2.7/xml/etree/", line 1126, in tostring
ElementTree(element).write(file, encoding, method=method)
File "/opt/stack/splunk/lib/python2.7/xml/etree/", line 820, in write
serialize(write, self._root, encoding, qnames, namespaces)
File "/opt/stack/splunk/lib/python2.7/xml/etree/", line 939, in _serialize_xml
_serialize_xml(write, e, encoding, qnames, None)
File "/opt/stack/splunk/lib/python2.7/xml/etree/", line 937, in _serialize_xml
write(_escape_cdata(text, encoding))
File "/opt/stack/splunk/lib/python2.7/xml/etree/", line 1075, in _escape_cdata
File "/opt/stack/splunk/lib/python2.7/xml/etree/", line 1052, in _raise_serialization_error
"cannot serialize %r (type %s)" % (text, type(text).
TypeError: cannot serialize {'TAB_example': 'TA-test-01:sourcetype:type1', 'System 2': 'TA-test-01:sourcetype:type1'} (type dict)

The TA extracts information through a REST API, collates the JSON information, and writes a JSON event with all info combined into Splunk.

What I can't understand is why it works fine for one input, but once a second one is introduced it fails. I have also tested this outside of Splunk TA Builder and see the same behaviour, and have reduced the TA down to a simple example where only 1 event is processed per data input and still see the same TypeError coming back.

Many Thanks!

0 Karma

Path Finder

Confirming that the fix of switching worked perfectly :

scheme.use_single_instance = True ==> scheme.use_single_instance = False

Thanks Gordan!

0 Karma

Splunk Employee
Splunk Employee

Hi Michael,
Would you mind share the code snippet about the collect_events function?

0 Karma

Path Finder

Hi Gwang,

Yes no problem, I tried a similar approach using the example code you use in TA builder to reproduce this error.

# encoding = utf-8

import os
import sys
import time
import datetime

def validate_input(helper, definition):
    """Implement your own validation logic to validate the input stanza configurations"""
    # This example accesses the modular input variable
    # string_label = definition.parameters.get('string_label', None)

def collect_events(helper, inputs, ew):
    """Implement your data collection logic here"""
    # The following example writes a random number as an event
    import random
    import json

    data = str(random.randint(0,100))

    event_data = {
        "info": data,
        "info2": data
    json_data = json.dumps(event_data)

    event = helper.new_event(source=helper.get_input_name(), index=helper.get_output_index(), 
            sourcetype=helper.get_sourcetype(), data=json_data)
    except Exception as e:
        raise e

When I run the above and attempt to add more than one data input of the same type I get the same error in splunkd.log:

11-23-2016 14:36:40.354 +0000 ERROR ExecProcessor - message from "python /opt/stack/splunk/etc/apps/TA-comm_example/bin/" ERRORcannot serialize {'example2': 'example_st', 'TAB_example': 'example_st'} (type dict)

The thing is, I get this error in the TA UI (cannot serialize) if I dont dump the dict to make it JSON serialized before writing it using event writer. Is the issue here that Splunk is expecting a JSON object but is instead getting passed a dict from a Splunk TA function that is meant to pass in the details of the inputs?

0 Karma
Get Updates on the Splunk Community!

Using Machine Learning for Hunting Security Threats

WATCH NOW Seeing the exponential hike in global cyber threat spectrum, organizations are now striving more for ...

Observability Newsletter Highlights | March 2023

 March 2023 | Check out the latest and greatestSplunk APM's New Tag Filter ExperienceSplunk APM has updated ...

Security Newsletter Updates | March 2023

 March 2023 | Check out the latest and greatestUnify Your Security Operations with Splunk Mission Control The ...