Getting Data In

Multiline field in modular input getting newline removed while indexed

Path Finder

I am creating a modular input. My input is a CSV and I convert it to JSON to be imported as a new event in Splunk. Several of the fields have newlines in the data. However, once indexed the newlines are removed. Here is the code that does it:

    csvdata = [row for row in csv.reader(data.splitlines())]
    header = csvdata.pop(0)
    for row in csvdata:
        e = {}
        for col, val in zip( header, row ):
            col = col.replace( " ", "_" )
            e[col] = val
        event_time = calendar.timegm(time.strptime(e["timefield"], time_pattern))
        event = helper.new_event(data=json.dumps(e), time=event_time, index=index, unbroken=True)

One thing I've tried is adding the SHOULD_LINEMERGE=0 to props.conf which didn't work. Is there a way to tell Splunk not to remove the newlines from fields?


0 Karma

Path Finder

I'm going to mark this as resolved.

The problem wasn't during indexing. It was actually here:

     csvdata = [row for row in csv.reader(data.splitlines())]

It mishandled the newlines. Getting rid of that and spliting on "\r\n" solved the problem

0 Karma
Get Updates on the Splunk Community!

Splunk Security Content for Threat Detection & Response, Q1 Roundup

Join Principal Threat Researcher, Michael Haag, as he walks through:An introduction to the Splunk Threat ...

Splunk Life | Happy Pride Month!

Happy Pride Month, Splunk Community! 🌈 In the United States, as well as many countries around the ...

SplunkTrust | Where Are They Now - Michael Uschmann

The Background Five years ago, Splunk published several videos showcasing members of the SplunkTrust to share ...