Why am I getting inconsistent search results?

[cmisztur]

I have configured Kepware IDF for Splunk and am ingesting data over TCP:51112. The source_type I have set ('opc') is arbitrary and does not exist. I have noticed when executing below search, the results are incorrect. The two records for "RunState" get omitted from the below search, and I am assuming that is because that collection of events contains an entry with Quality="bad", which does not meet my criteria.

Am I missing something fundamental, like not creating or setting a source_type?

WorkCenter="CNF59" | search RunState Quality="good"

[kbecker]

It looks like you need to tune the line breaking of these events as in your example Splunk is merging 4 log events into a separate Splunk events (check out props.conf). Once you have line breaking setup properly your searches should work as currently Splunk is trying to extract 4 values for this event and only one of them is winning.

[Richfez · Accepted solution]

I think your line breaking isn't right.

That whole thing looks like one event, and as one event it has both Quality="good" AND Quality="bad" in it. So, searching for Quality="good" brings up that entire event.

There are a variety of ways to fix this, possibly easiest may be - if it works - to change your props.conf for that sourcetype to

[my_custom_sourcetype]
BREAK_ONLY_BEFORE = ^\d{4}-\d{2}-\d{2}

There are quite a few other options. Personally, I think this issue could be caused because you are directly ingesting events over the network, and the source it sending them oddly. I'm pertty sure all the default line-breaking and timestamping would take care of this if you were ingesting them with syslog to a file, then reading those files off disk and into Splunk. (This method has a LOT of other advantages, too!).

[cmisztur]

+1 for the Syslog note.
Let me try the line break and see if that is the cause.

I have created a custom source_type and set the event break regex there. I hope that's the same as BREAK_ONLY_BEFORE.