topic Re: per event sourcetype override in Getting Data In

per event sourcetype override

ninisimonishvil — Tue, 29 Sep 2020 18:28:17 GMT

Hello,

I have events that by default are assigned to syslog sourcetype.

each of such event contains following sequence in it: Local7.Info 10.5.0.11 Feb 12 17:09:34 10.5.0.11 AlteonOS (and other info). I decided that describing this sequence via regex will identify those events that I would like to change a sourcetype for.

So I created stanza in TRANSFORMS

[sourcetypechange]
REGEX = \d{2}-\d{2}-\d{2}-\d{4}\s\d{2}:\d{2}:\d{2}\s\Local7.Info\s\d*.\d*.\d*.\d*\s\w{3}\s\d{2}\s\d{2}:\d{2}:\d{2}\s\d*.\d*.\d*.\d*\s\w\lteonOS\s\
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::Redsyslog

and also added following in PROPS

[syslog]
TRANSFORMS-sourcetype = sourcetypechange

As I understand, whenever splunk sees a syslog sourcetype - it has to check regex and if it matches it has to change sourcetype to Redsyslog, however I still have no result.

I was wondering, while writing a REGEX, shall it describe the whole event (from the beginning to the end) or just part (which is my case) is sufficient?

Re: per event sourcetype override

richgalloway — Tue, 13 Mar 2018 14:10:16 GMT

The REGEX string only needs to match part of the event.

Re: per event sourcetype override

ninisimonishvil — Tue, 13 Mar 2018 14:14:03 GMT

thanks. This means there is something missing.

Re: per event sourcetype override

tiagofbmm — Tue, 13 Mar 2018 14:19:21 GMT

I believe you have a mistake here:

SOURCE_KEY = _MetaData:Sourcetype

Re: per event sourcetype override

ninisimonishvil — Tue, 29 Sep 2020 18:26:30 GMT

You mean instead of DEST_KEY should be SOURCE_KEY?

Re: per event sourcetype override

tiagofbmm — Wed, 14 Mar 2018 07:58:19 GMT

Sorry it was my mistake about the syntax, nothing wrong with yours. Splunk applies the regex to the whole event. You should use a regex that uniquely identifies the kind of events you want to override Sourcertyoe. That is the criteria, a regex enoughly long and that let's your events be overridden.
Are you sure your regex is capturing the events?

Re: per event sourcetype override

ninisimonishvil — Wed, 14 Mar 2018 08:03:26 GMT

Well I check it at https://regex101.com/ and it highlight my event. Any suggestions where else I can check my regex?

Re: per event sourcetype override

tiagofbmm — Wed, 14 Mar 2018 08:10:03 GMT

Splunk uses Regex PCRE flavor of regular expressions, so anything that is PCRE-compliant is good to go. Did you make sure that was the flavor you were testing?

Re: per event sourcetype override

ninisimonishvil — Wed, 14 Mar 2018 09:07:08 GMT

OK I think that was the issue, now sourcetype changes. thanks a lot. however, now it does not do time extraction.

Re: per event sourcetype override

tiagofbmm — Tue, 29 Sep 2020 18:29:19 GMT

Ok good it is working.

The time may be missing for two reasons. Is the timestamp written after the 128th character in the event (which is the default of the parameter MAX_TIMESTAMP_LOOKAHEAD). Increase the MAX_TIMESTAMP_LOOKAHEAD if it is the case.

More, to make sure timestamp is recognised by Splunk, use the TIME_FORMAT parameter in the sourcetype definition (below the TRANSFORMS-sourcetype = sourcetypechange).

TIME_FORMAT=%b %d %H:%M:%S

For info on how the time variables work, http://docs.splunk.com/Documentation/Splunk/7.0.2/SearchReference/Commontimeformatvariables

TIME_FORMAT = <strptime-style format>
* Specifies a strptime format string to extract the date.
* strptime is an industry standard for designating time formats.
* For more information on strptime, see "Configure timestamp recognition" in
  the online documentation.
* TIME_FORMAT starts reading after the TIME_PREFIX. If both are specified,
  the TIME_PREFIX regex must match up to and including the character before
  the TIME_FORMAT date.
* For good results, the <strptime-style format> should describe the day of
  the year and the time of day.
* Defaults to empty.

Re: per event sourcetype override

ninisimonishvil — Tue, 29 Sep 2020 18:26:36 GMT

well problem is the following.

after assigning the new sourcetype, it shall break 1 event into several ones and also extract time as defined by regex.

When I indicated that sourcetype upon uploading a file it works perfectly, breaks down event into several ones and extracts correct time stamp. however when it comes to overriding, all the above mentioned points fail to work.

[mysourcetype]
pulldown_type = true
maxDist = 3
TIME_FORMAT = \d\d\/\d\d\/\d\d\d\d-\d\d:\d\d:\d\d
MAX_TIMESTAMP_LOOKAHEAD = 1500
TRANSFORMS = syslog-host
SHOULD_LINEMERGE = True
category = Operating System
BREAK_ONLY_BEFORE = |
DATETIME_CONFIG =
NO_BINARY_CHECK = true
disabled = false
TRANSFORMS-time1 = TIME
TIME_PREFIX = p
LINE_BREAKER = ()|
KV_MODE = none

and TIME from TRANSFORMS
[TIME]
REGEX=(?<=\w\w\w\s)(\d\d\/\d\d\/\d\d\d\d-\d\d:\d\d:\d\d)
FORMAT=TIME::$1
WRITE_META = True

Re: per event sourcetype override

tiagofbmm — Wed, 14 Mar 2018 11:21:56 GMT

The reason is that sourcetype override occurs after timestamp recognition and breaking into events. It doesn't work the way you want, which is override Sourcertyoe and only then break into events

It is expected behaviour. Each block only goes through the parsing pipeline once