Hi,
I am very new with Regex and have been struggling with simple task.
I need to change three values (Health, AuditTrail, Security) in a field called type into individual sourcetypes.
Any assistance would be greatly appreciated.
Thanks, Paul
Just as a note here:
Best practice would be to use a syslog server, like rsyslog or syslog-ng. Then pass the data to the indexers either by using an HTTP Event Collector or a UF or HF. It is harder to loose UDP data that way. Any restart of the Splunk (or syslog service, too) processing will result in a loss of data until the service comes back up. The UF and HF will take many times longer to restore the reception of the data.
If the amount of data coming in is not significant, then perhaps that doesn't matter, but I have one syslog server getting about 800GB/day of syslog data and it is working great (rsyslog -> nginx for load balancing -> indexers with HEC). You can get almost that with a UF alone, but you can't do any kind of parsing of that data to help you out, like separating data to different indexes. If you use an HF, then you will get about a third of that volume. But again, when you restart your Splunk process, you will loose more data than with a syslog server. I use rsyslog, and it's down less than a second, but when we used a UF, it took more than a minute, all the while dropping those UDP packets into the bit bucket.
It is also possible to sourcetype the data at the syslog level, which puts less strain on your indexers.
Something to think about while you are implementing your solution.
I still don't quite understand the need for a different source type here. Do these logs have different formats, or do you just want to spilt the log sources by sourcetype? Because there are many other possibilities of splitting/grouping events (think of eventtypes etc)
Yes, we want to split the log sources by sourcetype.
With this I simply moved the "type" field into the "sourcetype" field, but I want the values from "type" into "sourcetype".
REGEX = type=(?P[^;]+);
FORMAT = sourcetype::$1
DEST_KEY = MetaData:Sourcetype
Good call, somesoni2! I wasn't even thinking index time until I saw what you said, then I looked at the question again to see that it said sourcetypes. Hopefully everyone that reads this one will give answers that are index time answers.