Getting Data In

Splunk TA TypeError for multiple data inputs

Path Finder

Hey Splunk Community,

I am in the process of creating a TA with Splunk Add-On Builder and I have run into a problem for which I cannot seem to find an answer for either on here or a related on through online searches.

I have created a modular input using a Python script, and it is feeding data into Splunk fine, parsing the data, extracting fields, everything with no issues. Some background, a user enters environment details for set-up, then a system ID per individual data input to collect stats from. It works fine for just 1 input, but when I try to add a second I get the following error in splunkd.log:

2016-11-21 11:14:16,522 ERROR pid=15294 tid=MainThread | Get error when collecting events.
Traceback (most recent call last):
File "/opt/stack/splunk/etc/apps/TA-test-01/bin/TA-test-01/modinput_wrapper/", line 173, in stream_events
self.collect_events(inputs, ew)
File "/opt/stack/splunk/etc/apps/TA-test-01/bin/", line 53, in collect_events
input_module.collect_events(self, inputs, ew)
File "/opt/stack/splunk/etc/apps/TA-test-01/bin/", line 56, in collect_events
File "/opt/stack/splunk/etc/apps/TA-test-01/bin/TA-test-01/splunklib/modularinput/", line 60, in write_event
File "/opt/stack/splunk/etc/apps/TA-test-01/bin/TA-test-01/splunklib/modularinput/", line 106, in write_to
File "/opt/stack/splunk/lib/python2.7/xml/etree/", line 1126, in tostring
ElementTree(element).write(file, encoding, method=method)
File "/opt/stack/splunk/lib/python2.7/xml/etree/", line 820, in write
serialize(write, self._root, encoding, qnames, namespaces)
File "/opt/stack/splunk/lib/python2.7/xml/etree/", line 939, in _serialize_xml
_serialize_xml(write, e, encoding, qnames, None)
File "/opt/stack/splunk/lib/python2.7/xml/etree/", line 937, in _serialize_xml
write(_escape_cdata(text, encoding))
File "/opt/stack/splunk/lib/python2.7/xml/etree/", line 1075, in _escape_cdata
File "/opt/stack/splunk/lib/python2.7/xml/etree/", line 1052, in _raise_serialization_error
"cannot serialize %r (type %s)" % (text, type(text).
TypeError: cannot serialize {'TAB_example': 'TA-test-01:sourcetype:type1', 'System 2': 'TA-test-01:sourcetype:type1'} (type dict)

The TA extracts information through a REST API, collates the JSON information, and writes a JSON event with all info combined into Splunk.

What I can't understand is why it works fine for one input, but once a second one is introduced it fails. I have also tested this outside of Splunk TA Builder and see the same behaviour, and have reduced the TA down to a simple example where only 1 event is processed per data input and still see the same TypeError coming back.

Many Thanks!

0 Karma

Path Finder

Confirming that the fix of switching worked perfectly :

scheme.use_single_instance = True ==> scheme.use_single_instance = False

Thanks Gordan!

0 Karma

Splunk Employee
Splunk Employee

Hi Michael,
Would you mind share the code snippet about the collect_events function?

0 Karma

Path Finder

Hi Gwang,

Yes no problem, I tried a similar approach using the example code you use in TA builder to reproduce this error.

# encoding = utf-8

import os
import sys
import time
import datetime

def validate_input(helper, definition):
    """Implement your own validation logic to validate the input stanza configurations"""
    # This example accesses the modular input variable
    # string_label = definition.parameters.get('string_label', None)

def collect_events(helper, inputs, ew):
    """Implement your data collection logic here"""
    # The following example writes a random number as an event
    import random
    import json

    data = str(random.randint(0,100))

    event_data = {
        "info": data,
        "info2": data
    json_data = json.dumps(event_data)

    event = helper.new_event(source=helper.get_input_name(), index=helper.get_output_index(), 
            sourcetype=helper.get_sourcetype(), data=json_data)
    except Exception as e:
        raise e

When I run the above and attempt to add more than one data input of the same type I get the same error in splunkd.log:

11-23-2016 14:36:40.354 +0000 ERROR ExecProcessor - message from "python /opt/stack/splunk/etc/apps/TA-comm_example/bin/" ERRORcannot serialize {'example2': 'example_st', 'TAB_example': 'example_st'} (type dict)

The thing is, I get this error in the TA UI (cannot serialize) if I dont dump the dict to make it JSON serialized before writing it using event writer. Is the issue here that Splunk is expecting a JSON object but is instead getting passed a dict from a Splunk TA function that is meant to pass in the details of the inputs?

0 Karma