Splunk TA TypeError for multiple data inputs

[MichaelMcAleer]

Hey Splunk Community,

I am in the process of creating a TA with Splunk Add-On Builder and I have run into a problem for which I cannot seem to find an answer for either on here or a related on through online searches.

I have created a modular input using a Python script, and it is feeding data into Splunk fine, parsing the data, extracting fields, everything with no issues. Some background, a user enters environment details for set-up, then a system ID per individual data input to collect stats from. It works fine for just 1 input, but when I try to add a second I get the following error in splunkd.log:

2016-11-21 11:14:16,522 ERROR pid=15294 tid=MainThread file=base_modinput.py:log_error:69 | Get error when collecting events.
Traceback (most recent call last):
File "/opt/stack/splunk/etc/apps/TA-test-01/bin/TA-test-01/modinput_wrapper/base_modinput.py", line 173, in stream_events
self.collect_events(inputs, ew)
File "/opt/stack/splunk/etc/apps/TA-test-01/bin/TA-test-01.py", line 53, in collect_events
input_module.collect_events(self, inputs, ew)
File "/opt/stack/splunk/etc/apps/TA-test-01/bin/TA-test-01.py", line 56, in collect_events
ew.write_event(array_event)
File "/opt/stack/splunk/etc/apps/TA-test-01/bin/TA-test-01/splunklib/modularinput/event_writer.py", line 60, in write_event
event.write_to(self.out)
File "/opt/stack/splunk/etc/apps/TA-test-01/bin/TA-test-01/splunklib/modularinput/event.py", line 106, in write_to
stream.write(ET.tostring(event))
File "/opt/stack/splunk/lib/python2.7/xml/etree/ElementTree.py", line 1126, in tostring
ElementTree(element).write(file, encoding, method=method)
File "/opt/stack/splunk/lib/python2.7/xml/etree/ElementTree.py", line 820, in write
serialize(write, self._root, encoding, qnames, namespaces)
File "/opt/stack/splunk/lib/python2.7/xml/etree/ElementTree.py", line 939, in _serialize_xml
_serialize_xml(write, e, encoding, qnames, None)
File "/opt/stack/splunk/lib/python2.7/xml/etree/ElementTree.py", line 937, in _serialize_xml
write(_escape_cdata(text, encoding))
File "/opt/stack/splunk/lib/python2.7/xml/etree/ElementTree.py", line 1075, in _escape_cdata
_raise_serialization_error(text)
File "/opt/stack/splunk/lib/python2.7/xml/etree/ElementTree.py", line 1052, in _raise_serialization_error
"cannot serialize %r (type %s)" % (text, type(text).name_)
TypeError: cannot serialize {'TAB_example': 'TA-test-01:sourcetype:type1', 'System 2': 'TA-test-01:sourcetype:type1'} (type dict)

The TA extracts information through a REST API, collates the JSON information, and writes a JSON event with all info combined into Splunk.

What I can't understand is why it works fine for one input, but once a second one is introduced it fails. I have also tested this outside of Splunk TA Builder and see the same behaviour, and have reduced the TA down to a simple example where only 1 event is processed per data input and still see the same TypeError coming back.

Many Thanks!
Michael

[MichaelMcAleer]

Confirming that the fix of switching worked perfectly :

scheme.use_single_instance = True ==> scheme.use_single_instance = False

Thanks Gordan!

[gwang_splunk]

Hi Michael,
Would you mind share the code snippet about the collect_events function?

[MichaelMcAleer]

Hi Gwang,

Yes no problem, I tried a similar approach using the example code you use in TA builder to reproduce this error.

# encoding = utf-8

import os
import sys
import time
import datetime


def validate_input(helper, definition):
    """Implement your own validation logic to validate the input stanza configurations"""
    # This example accesses the modular input variable
    # string_label = definition.parameters.get('string_label', None)
    pass

def collect_events(helper, inputs, ew):
    """Implement your data collection logic here"""
    # The following example writes a random number as an event
    import random
    import json

    data = str(random.randint(0,100))

    event_data = {
        "info": data,
        "info2": data
        }
    json_data = json.dumps(event_data)


    event = helper.new_event(source=helper.get_input_name(), index=helper.get_output_index(), 
            sourcetype=helper.get_sourcetype(), data=json_data)
    try:
        ew.write_event(event)
    except Exception as e:
        raise e

When I run the above and attempt to add more than one data input of the same type I get the same error in splunkd.log:

11-23-2016 14:36:40.354 +0000 ERROR ExecProcessor - message from "python /opt/stack/splunk/etc/apps/TA-comm_example/bin/example_input.py" ERRORcannot serialize {'example2': 'example_st', 'TAB_example': 'example_st'} (type dict)

The thing is, I get this error in the TA UI (cannot serialize) if I dont dump the dict to make it JSON serialized before writing it using event writer. Is the issue here that Splunk is expecting a JSON object but is instead getting passed a dict from a Splunk TA function that is meant to pass in the details of the inputs?