Hello,
I have events that by default are assigned to syslog sourcetype.
each of such event contains following sequence in it: Local7.Info 10.5.0.11 Feb 12 17:09:34 10.5.0.11 AlteonOS (and other info). I decided that describing this sequence via regex will identify those events that I would like to change a sourcetype for.
So I created stanza in TRANSFORMS
[sourcetypechange]
REGEX = \d{2}-\d{2}-\d{2}-\d{4}\s\d{2}:\d{2}:\d{2}\s\Local7.Info\s\d*.\d*.\d*.\d*\s\w{3}\s\d{2}\s\d{2}:\d{2}:\d{2}\s\d*.\d*.\d*.\d*\s\w\lteonOS\s\
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::Redsyslog
and also added following in PROPS
[syslog]
TRANSFORMS-sourcetype = sourcetypechange
As I understand, whenever splunk sees a syslog sourcetype - it has to check regex and if it matches it has to change sourcetype to Redsyslog, however I still have no result.
I was wondering, while writing a REGEX, shall it describe the whole event (from the beginning to the end) or just part (which is my case) is sufficient?
The REGEX string only needs to match part of the event.
thanks. This means there is something missing.
I believe you have a mistake here:
SOURCE_KEY = _MetaData:Sourcetype
You mean instead of DEST_KEY should be SOURCE_KEY?
Sorry it was my mistake about the syntax, nothing wrong with yours. Splunk applies the regex to the whole event. You should use a regex that uniquely identifies the kind of events you want to override Sourcertyoe. That is the criteria, a regex enoughly long and that let's your events be overridden.
Are you sure your regex is capturing the events?
Well I check it at https://regex101.com/ and it highlight my event. Any suggestions where else I can check my regex?
Splunk uses Regex PCRE flavor of regular expressions, so anything that is PCRE-compliant is good to go. Did you make sure that was the flavor you were testing?
OK I think that was the issue, now sourcetype changes. thanks a lot. however, now it does not do time extraction.
Ok good it is working.
The time may be missing for two reasons. Is the timestamp written after the 128th character in the event (which is the default of the parameter MAX_TIMESTAMP_LOOKAHEAD). Increase the MAX_TIMESTAMP_LOOKAHEAD if it is the case.
More, to make sure timestamp is recognised by Splunk, use the TIME_FORMAT parameter in the sourcetype definition (below the TRANSFORMS-sourcetype = sourcetypechange).
TIME_FORMAT=%b %d %H:%M:%S
For info on how the time variables work, http://docs.splunk.com/Documentation/Splunk/7.0.2/SearchReference/Commontimeformatvariables
TIME_FORMAT = <strptime-style format>
* Specifies a strptime format string to extract the date.
* strptime is an industry standard for designating time formats.
* For more information on strptime, see "Configure timestamp recognition" in
the online documentation.
* TIME_FORMAT starts reading after the TIME_PREFIX. If both are specified,
the TIME_PREFIX regex must match up to and including the character before
the TIME_FORMAT date.
* For good results, the <strptime-style format> should describe the day of
the year and the time of day.
* Defaults to empty.
well problem is the following.
after assigning the new sourcetype, it shall break 1 event into several ones and also extract time as defined by regex.
When I indicated that sourcetype upon uploading a file it works perfectly, breaks down event into several ones and extracts correct time stamp. however when it comes to overriding, all the above mentioned points fail to work.
[mysourcetype]
pulldown_type = true
maxDist = 3
TIME_FORMAT = \d\d\/\d\d\/\d\d\d\d-\d\d:\d\d:\d\d
MAX_TIMESTAMP_LOOKAHEAD = 1500
TRANSFORMS = syslog-host
SHOULD_LINEMERGE = True
category = Operating System
BREAK_ONLY_BEFORE = |
DATETIME_CONFIG =
NO_BINARY_CHECK = true
disabled = false
TRANSFORMS-time1 = TIME
TIME_PREFIX = p
LINE_BREAKER = ()|
KV_MODE = none
and TIME from TRANSFORMS
[TIME]
REGEX=(?<=\w\w\w\s)(\d\d\/\d\d\/\d\d\d\d-\d\d:\d\d:\d\d)
FORMAT=TIME::$1
WRITE_META = True
The reason is that sourcetype override occurs after timestamp recognition and breaking into events. It doesn't work the way you want, which is override Sourcertyoe and only then break into events
It is expected behaviour. Each block only goes through the parsing pipeline once