Getting Data In

per event sourcetype override

ninisimonishvil
Path Finder

Hello,

I have events that by default are assigned to syslog sourcetype.

each of such event contains following sequence in it: Local7.Info 10.5.0.11 Feb 12 17:09:34 10.5.0.11 AlteonOS (and other info). I decided that describing this sequence via regex will identify those events that I would like to change a sourcetype for.

So I created stanza in TRANSFORMS

[sourcetypechange]
REGEX = \d{2}-\d{2}-\d{2}-\d{4}\s\d{2}:\d{2}:\d{2}\s\Local7.Info\s\d*.\d*.\d*.\d*\s\w{3}\s\d{2}\s\d{2}:\d{2}:\d{2}\s\d*.\d*.\d*.\d*\s\w\lteonOS\s\
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::Redsyslog

and also added following in PROPS

[syslog]
TRANSFORMS-sourcetype = sourcetypechange

As I understand, whenever splunk sees a syslog sourcetype - it has to check regex and if it matches it has to change sourcetype to Redsyslog, however I still have no result.

I was wondering, while writing a REGEX, shall it describe the whole event (from the beginning to the end) or just part (which is my case) is sufficient?

0 Karma

richgalloway
SplunkTrust
SplunkTrust

The REGEX string only needs to match part of the event.

---
If this reply helps you, Karma would be appreciated.
0 Karma

ninisimonishvil
Path Finder

thanks. This means there is something missing.

0 Karma

tiagofbmm
Influencer

I believe you have a mistake here:

SOURCE_KEY = _MetaData:Sourcetype

0 Karma

ninisimonishvil
Path Finder

You mean instead of DEST_KEY should be SOURCE_KEY?

0 Karma

tiagofbmm
Influencer

Sorry it was my mistake about the syntax, nothing wrong with yours. Splunk applies the regex to the whole event. You should use a regex that uniquely identifies the kind of events you want to override Sourcertyoe. That is the criteria, a regex enoughly long and that let's your events be overridden.
Are you sure your regex is capturing the events?

0 Karma

ninisimonishvil
Path Finder

Well I check it at https://regex101.com/ and it highlight my event. Any suggestions where else I can check my regex?

0 Karma

tiagofbmm
Influencer

Splunk uses Regex PCRE flavor of regular expressions, so anything that is PCRE-compliant is good to go. Did you make sure that was the flavor you were testing?

0 Karma

ninisimonishvil
Path Finder

OK I think that was the issue, now sourcetype changes. thanks a lot. however, now it does not do time extraction.

0 Karma

tiagofbmm
Influencer

Ok good it is working.

The time may be missing for two reasons. Is the timestamp written after the 128th character in the event (which is the default of the parameter MAX_TIMESTAMP_LOOKAHEAD). Increase the MAX_TIMESTAMP_LOOKAHEAD if it is the case.

More, to make sure timestamp is recognised by Splunk, use the TIME_FORMAT parameter in the sourcetype definition (below the TRANSFORMS-sourcetype = sourcetypechange).

TIME_FORMAT=%b %d %H:%M:%S

For info on how the time variables work, http://docs.splunk.com/Documentation/Splunk/7.0.2/SearchReference/Commontimeformatvariables

TIME_FORMAT = <strptime-style format>
* Specifies a strptime format string to extract the date.
* strptime is an industry standard for designating time formats.
* For more information on strptime, see "Configure timestamp recognition" in
  the online documentation.
* TIME_FORMAT starts reading after the TIME_PREFIX. If both are specified,
  the TIME_PREFIX regex must match up to and including the character before
  the TIME_FORMAT date.
* For good results, the <strptime-style format> should describe the day of
  the year and the time of day.
* Defaults to empty.
0 Karma

ninisimonishvil
Path Finder

well problem is the following.

after assigning the new sourcetype, it shall break 1 event into several ones and also extract time as defined by regex.

When I indicated that sourcetype upon uploading a file it works perfectly, breaks down event into several ones and extracts correct time stamp. however when it comes to overriding, all the above mentioned points fail to work.

[mysourcetype]
pulldown_type = true
maxDist = 3
TIME_FORMAT = \d\d\/\d\d\/\d\d\d\d-\d\d:\d\d:\d\d
MAX_TIMESTAMP_LOOKAHEAD = 1500
TRANSFORMS = syslog-host
SHOULD_LINEMERGE = True
category = Operating System
BREAK_ONLY_BEFORE = |
DATETIME_CONFIG =
NO_BINARY_CHECK = true
disabled = false
TRANSFORMS-time1 = TIME
TIME_PREFIX = p
LINE_BREAKER = ()|
KV_MODE = none

and TIME from TRANSFORMS
[TIME]
REGEX=(?<=\w\w\w\s)(\d\d\/\d\d\/\d\d\d\d-\d\d:\d\d:\d\d)
FORMAT=TIME::$1
WRITE_META = True

0 Karma

tiagofbmm
Influencer

The reason is that sourcetype override occurs after timestamp recognition and breaking into events. It doesn't work the way you want, which is override Sourcertyoe and only then break into events

It is expected behaviour. Each block only goes through the parsing pipeline once

0 Karma
Get Updates on the Splunk Community!

Sending Metrics to Splunk Enterprise With the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...

What's New in Splunk Cloud Platform 9.0.2208?!

Howdy!  We are happy to share the newest updates in Splunk Cloud Platform 9.0.2208! Analysts can benefit ...

Want a chance to win $500 to the Splunk shop? Take our IT Incident Management Survey!

  Top Trends & Best Practices in Incident ManagementSplunk is partnering up with Constellation Research to ...