Splunk Search

Why am I getting inconsistent search results?

cmisztur
Explorer

I have configured Kepware IDF for Splunk and am ingesting data over TCP:51112. The source_type I have set ('opc') is arbitrary and does not exist. I have noticed when executing below search, the results are incorrect. The two records for "RunState" get omitted from the below search, and I am assuming that is because that collection of events contains an entry with Quality="bad", which does not meet my criteria.

Am I missing something fundamental, like not creating or setting a source_type?

WorkCenter="CNF59" | search RunState Quality="good"

alt text

Tags (2)
0 Karma
1 Solution

Richfez
SplunkTrust
SplunkTrust

I think your line breaking isn't right.

That whole thing looks like one event, and as one event it has both Quality="good" AND Quality="bad" in it. So, searching for Quality="good" brings up that entire event.

There are a variety of ways to fix this, possibly easiest may be - if it works - to change your props.conf for that sourcetype to

[my_custom_sourcetype]
BREAK_ONLY_BEFORE = ^\d{4}-\d{2}-\d{2}

There are quite a few other options. Personally, I think this issue could be caused because you are directly ingesting events over the network, and the source it sending them oddly. I'm pertty sure all the default line-breaking and timestamping would take care of this if you were ingesting them with syslog to a file, then reading those files off disk and into Splunk. (This method has a LOT of other advantages, too!).

View solution in original post

kbecker
Communicator

It looks like you need to tune the line breaking of these events as in your example Splunk is merging 4 log events into a separate Splunk events (check out props.conf). Once you have line breaking setup properly your searches should work as currently Splunk is trying to extract 4 values for this event and only one of them is winning.

Richfez
SplunkTrust
SplunkTrust

I think your line breaking isn't right.

That whole thing looks like one event, and as one event it has both Quality="good" AND Quality="bad" in it. So, searching for Quality="good" brings up that entire event.

There are a variety of ways to fix this, possibly easiest may be - if it works - to change your props.conf for that sourcetype to

[my_custom_sourcetype]
BREAK_ONLY_BEFORE = ^\d{4}-\d{2}-\d{2}

There are quite a few other options. Personally, I think this issue could be caused because you are directly ingesting events over the network, and the source it sending them oddly. I'm pertty sure all the default line-breaking and timestamping would take care of this if you were ingesting them with syslog to a file, then reading those files off disk and into Splunk. (This method has a LOT of other advantages, too!).

cmisztur
Explorer

+1 for the Syslog note.
Let me try the line break and see if that is the cause.

I have created a custom source_type and set the event break regex there. I hope that's the same as BREAK_ONLY_BEFORE.

0 Karma
Get Updates on the Splunk Community!

New in Observability - Improvements to Custom Metrics SLOs, Log Observer Connect & ...

The latest enhancements to the Splunk observability portfolio deliver improved SLO management accuracy, better ...

Improve Data Pipelines Using Splunk Data Management

  Register Now   This Tech Talk will explore the pipeline management offerings Edge Processor and Ingest ...

3-2-1 Go! How Fast Can You Debug Microservices with Observability Cloud?

Register Join this Tech Talk to learn how unique features like Service Centric Views, Tag Spotlight, and ...