topic Re: Regex to move three values into sourcetype field with transforms.conf in Splunk Search

Regex to move three values into sourcetype field with transforms.conf

pbugeja — Thu, 22 Jun 2017 20:40:17 GMT

Hi,

I am very new with Regex and have been struggling with simple task.

I need to change three values (Health, AuditTrail, Security) in a field called type into individual sourcetypes.

Any assistance would be greatly appreciated.

Thanks, Paul

Re: Regex to move three values into sourcetype field with transforms.conf

somesoni2 — Thu, 22 Jun 2017 21:29:51 GMT

We'd need sample events and mock output to help you accurately. Also, assuming this needs to be done at index-time. Correct me if I'm wrong.

Re: Regex to move three values into sourcetype field with transforms.conf

cpetterborg — Thu, 22 Jun 2017 23:44:48 GMT

Good call, somesoni2! I wasn't even thinking index time until I saw what you said, then I looked at the question again to see that it said sourcetypes. Hopefully everyone that reads this one will give answers that are index time answers.

Re: Regex to move three values into sourcetype field with transforms.conf

pbugeja — Fri, 23 Jun 2017 12:17:00 GMT

Appreciate you responding to my question.

I would like to do query my data at search time.

field=type
value=Health,AuditTrail,Security

Need to create new sourcetypes with each value: sourcetype=Health;sourcetype=AuditTrail;sourcetype=Security

Thanks, Paul

Re: Regex to move three values into sourcetype field with transforms.conf

DMohn — Fri, 23 Jun 2017 12:43:26 GMT

I'm afraid this is not possible. The sourcetype of an event is a indexed field, and cannot be changed during search time.

Can you please elaborate why you want to have different sourcetypes here? Maybe there is a easy solution for your problem.

Re: Regex to move three values into sourcetype field with transforms.conf

pbugeja — Fri, 23 Jun 2017 12:57:02 GMT

The reason for the new sourcetypes , each of the values represent a different log source as I am parsing data from a firewall.

Re: Regex to move three values into sourcetype field with transforms.conf

DMohn — Fri, 23 Jun 2017 13:29:08 GMT

I still don't quite understand the need for a different source type here. Do these logs have different formats, or do you just want to spilt the log sources by sourcetype? Because there are many other possibilities of splitting/grouping events (think of eventtypes etc)

Re: Regex to move three values into sourcetype field with transforms.conf

pbugeja — Fri, 23 Jun 2017 13:32:23 GMT

Yes, we want to split the log sources by sourcetype.

Re: Regex to move three values into sourcetype field with transforms.conf

pbugeja — Fri, 23 Jun 2017 13:34:44 GMT

With this I simply moved the "type" field into the "sourcetype" field, but I want the values from "type" into "sourcetype".

REGEX = type=(?P[^;]+);
FORMAT = sourcetype::$1
DEST_KEY = MetaData:Sourcetype

Re: Regex to move three values into sourcetype field with transforms.conf

cpetterborg — Fri, 23 Jun 2017 13:54:29 GMT

Just as a note here:

Best practice would be to use a syslog server, like rsyslog or syslog-ng. Then pass the data to the indexers either by using an HTTP Event Collector or a UF or HF. It is harder to loose UDP data that way. Any restart of the Splunk (or syslog service, too) processing will result in a loss of data until the service comes back up. The UF and HF will take many times longer to restore the reception of the data.

If the amount of data coming in is not significant, then perhaps that doesn't matter, but I have one syslog server getting about 800GB/day of syslog data and it is working great (rsyslog -> nginx for load balancing -> indexers with HEC). You can get almost that with a UF alone, but you can't do any kind of parsing of that data to help you out, like separating data to different indexes. If you use an HF, then you will get about a third of that volume. But again, when you restart your Splunk process, you will loose more data than with a syslog server. I use rsyslog, and it's down less than a second, but when we used a UF, it took more than a minute, all the while dropping those UDP packets into the bit bucket.

It is also possible to sourcetype the data at the syslog level, which puts less strain on your indexers.

Something to think about while you are implementing your solution.

Re: Regex to move three values into sourcetype field with transforms.conf

pbugeja — Fri, 23 Jun 2017 14:04:36 GMT

I appreciate your recommendation, but I have been tasked with the segmenting of the different logs with individual sourcetypes as the data is combined from the cloud firewall.

Re: Regex to move three values into sourcetype field with transforms.conf

cpetterborg — Fri, 23 Jun 2017 14:14:44 GMT

Note in the above answer:

It is also possible to sourcetype the data at the syslog level, which puts less strain on your indexers.

Which is what you said that you were tasked with doing.

Re: Regex to move three values into sourcetype field with transforms.conf

pbugeja — Tue, 29 Sep 2020 14:39:01 GMT

currently the following rex command is creating new sourcetypes, but I still need assistance the props or transforms conf files.

index="cato_dev" source=CATOLOG.TXT | rex field=type "(?.*)"

adds the Health, AuditTrail and Security sourcetypes

props.conf
[cato:logs]
CHARSET = UTF-8
SHOULD_LINEMERGE = True
KV_MODE = json
TRANSFORMS-changesourcetype = cato:logs

[source::/opt/splunk/etc/apps/Cato_Input/CATOLOG.TXT]
TRANSFORMS-changesourcetype = cato:logs

transforms.conf
[cato:logs]
REGEX field=type "(?.*)"
FORMAT = sourcetype::cato:logs
DEST_KEY = MetaData:Sourcetype

Re: Regex to move three values into sourcetype field with transforms.conf

pbugeja — Tue, 29 Sep 2020 14:34:49 GMT

Currently I am able to create the new sourcetypes with the following rex command, but still have a problem with either the props or transforms conf file. I guess there is a syntax issue.

This works > index="cato_dev" | rex field=type "(?.*)"

sourcetypes:Health;AuditTrail;Security

transform.conf
[cato:logs]
REGEX field=type "(?.*)"
FORMAT = sourcetype::cato:logs
DEST_KEY = MetaData:Sourcetype

props.conf
[cato:logs]
CHARSET = UTF-8
SHOULD_LINEMERGE = True
KV_MODE = json
TRANSFORMS-changesourcetype = cato:logs

[source::/opt/splunk/etc/apps/Cato_Input/CATOLOG.TXT]
TRANSFORMS-changesourcetype = cato:logs

Re: Regex to move three values into sourcetype field with transforms.conf

cpetterborg — Sun, 25 Jun 2017 20:07:58 GMT

Try:

FORMAT = sourcetype::$1

Re: Regex to move three values into sourcetype field with transforms.conf

pbugeja — Sun, 25 Jun 2017 20:11:17 GMT

Hi cpetterborg,
I unfortunately started with "FORMAT = sourcetype::$1" with the same effect.

Re: Regex to move three values into sourcetype field with transforms.conf

cpetterborg — Sun, 25 Jun 2017 20:53:01 GMT

Okay, here we go. Let's try this:

props.conf:

CHARSET = UTF-8
SHOULD_LINEMERGE = True
# If your data is JSON, then the transforms.conf file will need a different REGEX statement the you were using
KV_MODE = json
TRANSFORMS-changesourcetype = cato:logs

transforms.conf:

[cato:logs]
# From what I could understand, you gave the explanation of the JSON data with the field that looked like:
#     "type"="value"
# so the regex might be your problem. The following will put the value in the capture group:
REGEX = \"type\"=\"(\w+)\"
# which would make it find the field value from the JSON string
#
# this FORMAT really should say $1, which is the capture group designation for what is matched in the parens:
FORMAT = sourcetype::$1
DEST_KEY = MetaData:Sourcetype

If I'm wrong about the actual format (you didn't give an actual example of the data), then please provide an example of the data. That will make it much easier to help you. I'm going just from the fact that you provided a rex command which looks like it might work. Hopefully I got it right, and didn't make any typos. I think I got it right, though.

Re: Regex to move three values into sourcetype field with transforms.conf

pbugeja — Sun, 25 Jun 2017 22:27:01 GMT

I tried your regex and unfortunately did not work.

My rex command did add my the new sourcetypes as a search.
index="cato_dev" | rex field=type (?.*)
with the above it added sourcetype: Health, AuditTrail, Security

Re: Regex to move three values into sourcetype field with transforms.conf

pbugeja — Sun, 25 Jun 2017 22:36:06 GMT

This is my command minus the two astericks before and after the search command

index="cato_dev" | rex field=type (?.*)

Re: Regex to move three values into sourcetype field with transforms.conf

pbugeja — Sun, 25 Jun 2017 22:45:24 GMT

sorry but my regex command is being edited.