Splunk Search

Regex to move three values into sourcetype field with transforms.conf

pbugeja
New Member

Hi,

I am very new with Regex and have been struggling with simple task.

I need to change three values (Health, AuditTrail, Security) in a field called type into individual sourcetypes.

Any assistance would be greatly appreciated.

Thanks, Paul

Tags (1)
0 Karma
1 Solution

cpetterborg
SplunkTrust
SplunkTrust

Just as a note here:

Best practice would be to use a syslog server, like rsyslog or syslog-ng. Then pass the data to the indexers either by using an HTTP Event Collector or a UF or HF. It is harder to loose UDP data that way. Any restart of the Splunk (or syslog service, too) processing will result in a loss of data until the service comes back up. The UF and HF will take many times longer to restore the reception of the data.

If the amount of data coming in is not significant, then perhaps that doesn't matter, but I have one syslog server getting about 800GB/day of syslog data and it is working great (rsyslog -> nginx for load balancing -> indexers with HEC). You can get almost that with a UF alone, but you can't do any kind of parsing of that data to help you out, like separating data to different indexes. If you use an HF, then you will get about a third of that volume. But again, when you restart your Splunk process, you will loose more data than with a syslog server. I use rsyslog, and it's down less than a second, but when we used a UF, it took more than a minute, all the while dropping those UDP packets into the bit bucket.

It is also possible to sourcetype the data at the syslog level, which puts less strain on your indexers.

Something to think about while you are implementing your solution.

View solution in original post

0 Karma

DMohn
Motivator

I still don't quite understand the need for a different source type here. Do these logs have different formats, or do you just want to spilt the log sources by sourcetype? Because there are many other possibilities of splitting/grouping events (think of eventtypes etc)

0 Karma

pbugeja
New Member

Yes, we want to split the log sources by sourcetype.

0 Karma

pbugeja
New Member

With this I simply moved the "type" field into the "sourcetype" field, but I want the values from "type" into "sourcetype".

REGEX = type=(?P[^;]+);
FORMAT = sourcetype::$1
DEST_KEY = MetaData:Sourcetype

0 Karma

cpetterborg
SplunkTrust
SplunkTrust

Good call, somesoni2! I wasn't even thinking index time until I saw what you said, then I looked at the question again to see that it said sourcetypes. Hopefully everyone that reads this one will give answers that are index time answers.

0 Karma
Get Updates on the Splunk Community!

See just what you’ve been missing | Observability tracks at Splunk University

Looking to sharpen your observability skills so you can better understand how to collect and analyze data from ...

Weezer at .conf25? Say it ain’t so!

Hello Splunkers, The countdown to .conf25 is on-and we've just turned up the volume! We're thrilled to ...

How SC4S Makes Suricata Logs Ingestion Simple

Network security monitoring has become increasingly critical for organizations of all sizes. Splunk has ...