Getting Data In

Splunk TA TypeError for multiple data inputs

Path Finder

Hey Splunk Community,

I am in the process of creating a TA with Splunk Add-On Builder and I have run into a problem for which I cannot seem to find an answer for either on here or a related on through online searches.

I have created a modular input using a Python script, and it is feeding data into Splunk fine, parsing the data, extracting fields, everything with no issues. Some background, a user enters environment details for set-up, then a system ID per individual data input to collect stats from. It works fine for just 1 input, but when I try to add a second I get the following error in splunkd.log:

2016-11-21 11:14:16,522 ERROR pid=15294 tid=MainThread file=base_modinput.py:log_error:69 | Get error when collecting events.
Traceback (most recent call last):
File "/opt/stack/splunk/etc/apps/TA-test-01/bin/TA-test-01/modinput_wrapper/base_modinput.py", line 173, in stream_events
self.collect_events(inputs, ew)
File "/opt/stack/splunk/etc/apps/TA-test-01/bin/TA-test-01.py", line 53, in collect_events
input_module.collect_events(self, inputs, ew)
File "/opt/stack/splunk/etc/apps/TA-test-01/bin/TA-test-01.py", line 56, in collect_events
ew.write_event(array_event)
File "/opt/stack/splunk/etc/apps/TA-test-01/bin/TA-test-01/splunklib/modularinput/event_writer.py", line 60, in write_event
event.write_to(self.out)
File "/opt/stack/splunk/etc/apps/TA-test-01/bin/TA-test-01/splunklib/modularinput/event.py", line 106, in write_to
stream.write(ET.tostring(event))
File "/opt/stack/splunk/lib/python2.7/xml/etree/ElementTree.py", line 1126, in tostring
ElementTree(element).write(file, encoding, method=method)
File "/opt/stack/splunk/lib/python2.7/xml/etree/ElementTree.py", line 820, in write
serialize(write, self._root, encoding, qnames, namespaces)
File "/opt/stack/splunk/lib/python2.7/xml/etree/ElementTree.py", line 939, in _serialize_xml
_serialize_xml(write, e, encoding, qnames, None)
File "/opt/stack/splunk/lib/python2.7/xml/etree/ElementTree.py", line 937, in _serialize_xml
write(_escape_cdata(text, encoding))
File "/opt/stack/splunk/lib/python2.7/xml/etree/ElementTree.py", line 1075, in _escape_cdata
_raise_serialization_error(text)
File "/opt/stack/splunk/lib/python2.7/xml/etree/ElementTree.py", line 1052, in _raise_serialization_error
"cannot serialize %r (type %s)" % (text, type(text).
name_)
TypeError: cannot serialize {'TAB_example': 'TA-test-01:sourcetype:type1', 'System 2': 'TA-test-01:sourcetype:type1'} (type dict)

The TA extracts information through a REST API, collates the JSON information, and writes a JSON event with all info combined into Splunk.

What I can't understand is why it works fine for one input, but once a second one is introduced it fails. I have also tested this outside of Splunk TA Builder and see the same behaviour, and have reduced the TA down to a simple example where only 1 event is processed per data input and still see the same TypeError coming back.

Many Thanks!
Michael

0 Karma

Path Finder

Confirming that the fix of switching worked perfectly :

scheme.use_single_instance = True ==> scheme.use_single_instance = False

Thanks Gordan!

0 Karma

Splunk Employee
Splunk Employee

Hi Michael,
Would you mind share the code snippet about the collect_events function?

0 Karma

Path Finder

Hi Gwang,

Yes no problem, I tried a similar approach using the example code you use in TA builder to reproduce this error.

# encoding = utf-8

import os
import sys
import time
import datetime


def validate_input(helper, definition):
    """Implement your own validation logic to validate the input stanza configurations"""
    # This example accesses the modular input variable
    # string_label = definition.parameters.get('string_label', None)
    pass

def collect_events(helper, inputs, ew):
    """Implement your data collection logic here"""
    # The following example writes a random number as an event
    import random
    import json

    data = str(random.randint(0,100))

    event_data = {
        "info": data,
        "info2": data
        }
    json_data = json.dumps(event_data)


    event = helper.new_event(source=helper.get_input_name(), index=helper.get_output_index(), 
            sourcetype=helper.get_sourcetype(), data=json_data)
    try:
        ew.write_event(event)
    except Exception as e:
        raise e

When I run the above and attempt to add more than one data input of the same type I get the same error in splunkd.log:

11-23-2016 14:36:40.354 +0000 ERROR ExecProcessor - message from "python /opt/stack/splunk/etc/apps/TA-comm_example/bin/example_input.py" ERRORcannot serialize {'example2': 'example_st', 'TAB_example': 'example_st'} (type dict)

The thing is, I get this error in the TA UI (cannot serialize) if I dont dump the dict to make it JSON serialized before writing it using event writer. Is the issue here that Splunk is expecting a JSON object but is instead getting passed a dict from a Splunk TA function that is meant to pass in the details of the inputs?

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!