topic Re: If a filed position change in logs then how to trace? in Splunk Search

If a field position change in logs then how to trace?

mk00928640 — Wed, 26 Jul 2023 18:03:49 GMT

Hello everyone

Please assist me in solving the problem below.

I'm attempting to determine how to track it in Splunk if a field's place changes in logs.

Is SPL tracing in SPLUNK possible?

Ex:

Logs : when we onboard the logs in splunk on the below positions.

if it changed to then How to trace it by SPL?

Please guide me

Re: If a filed position change in logs then how to trace?

PickleRick — Wed, 26 Jul 2023 17:32:40 GMT

Your question is not entirely clear. Most probably because english is not your native language. Don't worry. Many of us here are non-natives as well. 🙂

I assume that you wondered if your extractions will work properly if the order of your fields change within the event. The answer is - it depends on what kind of extraction is it (regex-based or delimited) and if your fields are extracted using regexes, how are the regexes defined. So there is no single "good" answer here. Of course if you change your logging format completely, no extraction will be good enough to catch that.

Re: If a filed position change in logs then how to trace?

isoutamo — Wed, 26 Jul 2023 22:24:49 GMT

it’s just like @PickleRick said. One way to try to tackle this is add “:#” at the end of sourcetype name. When I’m defining ST in first time I just add :0 as o suffix for it. When it has changed (format of log file), I add a new version by a new definition with same name, but just change to :1 this suffix. That way I can have several correct extractions for “same” log file which contents has evolved by time.

r. Ismo

Re: If a field position change in logs then how to trace?

yuanliu — Thu, 27 Jul 2023 04:10:41 GMT

In addition to what @PickleRick and @isoutamo said,

Try not to rely on regex for extraction. For example, lots of fields can be extracted as key value pairs with kv aka extract: try | kv pairdelim=";" kvdelim=":".
If you have any influence on developers, beg them to place quotation marks around values.
When you use regex for complex, large texts, avoid using a single rex for the entire event unless you have absolute control over source code and the power to forbid any change in log format. Use smaller rex commands like | rex "; (?<message>[^;]+)$".

Re: If a filed position change in logs then how to trace?

mk00928640 — Thu, 27 Jul 2023 07:17:29 GMT

Thank you for responding; how tragic. My question is fairly straightforward: When logs are added to Splunk after six months, the logs' balance may alter owing to system upgrades. If the logs location changed in this instance, how would SPL detect the change?

Ex: Records a position in the logs.

Following Splunk Onboarding: 1, 2, 3, 4, and 5.
Just consider:- Changed field positions: 1, 2, 3, 5, 4.

it doesn't matter if Extraction is happening by Regex or Delimiter.

Re: If a field position change in logs then how to trace?

PickleRick — Thu, 27 Jul 2023 08:48:22 GMT

Ad.3 One could argue about the performance of single big regex vs. many small regexes. But I suppose it depends on how bloated that single regex is so most probably YMMV.

Re: If a filed position change in logs then how to trace?

PickleRick — Thu, 27 Jul 2023 08:52:28 GMT

In case of structured data, there is a special case for CSV files with variable order of columns - it reads the header line and adjusts the order of fields accordingly. Unfortunately, it works only as indexed extractions.