Splunk Search

Why am I getting inconsistent search results?

cmisztur
Explorer

I have configured Kepware IDF for Splunk and am ingesting data over TCP:51112. The source_type I have set ('opc') is arbitrary and does not exist. I have noticed when executing below search, the results are incorrect. The two records for "RunState" get omitted from the below search, and I am assuming that is because that collection of events contains an entry with Quality="bad", which does not meet my criteria.

Am I missing something fundamental, like not creating or setting a source_type?

WorkCenter="CNF59" | search RunState Quality="good"

alt text

Tags (2)
0 Karma
1 Solution

Richfez
SplunkTrust
SplunkTrust

I think your line breaking isn't right.

That whole thing looks like one event, and as one event it has both Quality="good" AND Quality="bad" in it. So, searching for Quality="good" brings up that entire event.

There are a variety of ways to fix this, possibly easiest may be - if it works - to change your props.conf for that sourcetype to

[my_custom_sourcetype]
BREAK_ONLY_BEFORE = ^\d{4}-\d{2}-\d{2}

There are quite a few other options. Personally, I think this issue could be caused because you are directly ingesting events over the network, and the source it sending them oddly. I'm pertty sure all the default line-breaking and timestamping would take care of this if you were ingesting them with syslog to a file, then reading those files off disk and into Splunk. (This method has a LOT of other advantages, too!).

View solution in original post

kbecker
Communicator

It looks like you need to tune the line breaking of these events as in your example Splunk is merging 4 log events into a separate Splunk events (check out props.conf). Once you have line breaking setup properly your searches should work as currently Splunk is trying to extract 4 values for this event and only one of them is winning.

Richfez
SplunkTrust
SplunkTrust

I think your line breaking isn't right.

That whole thing looks like one event, and as one event it has both Quality="good" AND Quality="bad" in it. So, searching for Quality="good" brings up that entire event.

There are a variety of ways to fix this, possibly easiest may be - if it works - to change your props.conf for that sourcetype to

[my_custom_sourcetype]
BREAK_ONLY_BEFORE = ^\d{4}-\d{2}-\d{2}

There are quite a few other options. Personally, I think this issue could be caused because you are directly ingesting events over the network, and the source it sending them oddly. I'm pertty sure all the default line-breaking and timestamping would take care of this if you were ingesting them with syslog to a file, then reading those files off disk and into Splunk. (This method has a LOT of other advantages, too!).

cmisztur
Explorer

+1 for the Syslog note.
Let me try the line break and see if that is the cause.

I have created a custom source_type and set the event break regex there. I hope that's the same as BREAK_ONLY_BEFORE.

0 Karma
Get Updates on the Splunk Community!

Splunk Lantern | Spotlight on Security: Adoption Motions, War Stories, and More

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...

Splunk Cloud | Empowering Splunk Administrators with Admin Config Service (ACS)

Greetings, Splunk Cloud Admins and Splunk enthusiasts! The Admin Configuration Service (ACS) team is excited ...

Tech Talk | One Log to Rule Them All

One log to rule them all: how you can centralize your troubleshooting with Splunk logs We know how important ...