Deployment Architecture

Splunk is not correctly splitting Syslog and JSON messages by new line

daniel_splunk
Splunk Employee
Splunk Employee

We are experiencing issues with various logging sources where messages are not being correctly split by new line and therefore are resulting in a linecount of greater than 1. This is being experienced for our indexes security-apigw (syslog) and security-carbonblack (json).

I updated the props.conf to incude the following. However, some of the event still not split correctly.
SHOULD_LINEMERGE = false
LINE_BREAKER = }}([\r\n]+)

Tags (1)
0 Karma

daniel_splunk
Splunk Employee
Splunk Employee

The “INDEXED_EXTRACTION” came out in 6.x changes a few things on how Splunk works. Its used for well known log sources (IIS, JSON, XML) to have their fields parsed out by the forwarder, moving the work from the indexer to the forwarder.

So you need to install those props on the forwarder, not the indexer. When INDEXED_EXTRACTIONs is enabled, it will pass the parsed fields into the indexers. The indexers will not apply any parsing rules for data that was processed as INDEXED_EXTRACTIONs.

You can add the following into your forwarder configuration.

props.conf
[carbonblack:json]
INDEXED_EXTRACTIONS=json

Here is the doc for more information.
http://docs.splunk.com/Documentation/Splunk/6.5.2/Data/Extractfieldsfromfileswithstructureddata

Get Updates on the Splunk Community!

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...

Splunkbase | Splunk Dashboard Examples App for SimpleXML End of Life

The Splunk Dashboard Examples App for SimpleXML will reach end of support on Dec 19, 2024, after which no new ...

Understanding Generative AI Techniques and Their Application in Cybersecurity

Watch On-Demand Artificial intelligence is the talk of the town nowadays, with industries of all kinds ...