topic Re: How can I override source type? in Splunk Search

How can I override source type?

ninisimonishvil — Tue, 29 Sep 2020 18:17:21 GMT

Hello

I have an event that starts like this:

02-12-2018 17:07:33 Local7.Info 10.5.0.11 Feb 12 17:07:32 10.5.0.11 AlteonOS :

10.5.0.1 58696 10.5.0.101 80 tcp 12/02/2018-17:07:10 12/02/2018-17:07:11 10.6.0.101 80 0.0.0.0 2060 |

10.5.0.1 58697 10.5.0.101 80 tcp 12/02/2018-17:07:10 12/02/2018-17:07:11 10.6.0.101 80 0.0.0.0 2075 |

I want to override the sourcetype while indexing. Here are my transforms and props file contents:

Transforms:
[sourcetypechange]
REGEX = \d{2}-\d{2}-\d{2}-\d{4}\s\d{2}:\d{2}:\d{2}\s\Local7.Info\s\d*.\d*.\d*.\d*\s\w{3}\s\d{2}\s\d{2}:\d{2}:\d{2}\s\d*.\d*.\d*.\d*\s\w\lteonOS\s\
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::Redsyslog

Props
[syslog3.txt]
TRANSFORMS-sourcetype = sourcetypechange

syslog3.txt is a source, I upload it to test if there are changes in sourcetype, no success. Can anyone tell me if I'm doing anything wrong?

Re: How can I override source type?

gcusello — Fri, 02 Mar 2018 13:29:53 GMT

Hi ninisimonishvili,
if you put [syslog3.txt] in a props.conf stanza, Splunk takes it as a sourcetype.
You should use

[source::<source>]

using the full path of your source

See https://docs.splunk.com/Documentation/Splunk/7.0.2/Admin/Propsconf

Bye.
Giuseppe

Re: How can I override source type?

ninisimonishvil — Mon, 05 Mar 2018 06:33:33 GMT

Thank you Giuseppe,

I tried indicating full path (also used sourcetype instead of source) however no success.
The regex that I'm using in Transforms - shall it describe the whole event or just the beginning?

because the start of the event will follow this sequence
: 02-12-2018 17:07:33 Local7.Info 10.5.0.11 Feb 12 17:07:32 10.5.0.11 AlteonOS :

however, the number of followed IP addresses and timestamps may vary from event to event.

Re: How can I override source type?

gcusello — Mon, 05 Mar 2018 07:34:29 GMT

Hi ninisimonishvili,
the regex to find could be in every part of your event: you can verify it in Splunk using the rex command:

index=my_index | rex "my_regex"

in this way you should have as results the event to discard.

The job is to find (if exist) the correct regex to find all the events to discard: you could also think to use more than one regex, anyway the method is the one above: use rex command in Splunk search, eventually more than one.

Bye.
Giuseppe

Re: How can I override source type?

serjandrosov — Mon, 05 Mar 2018 08:06:11 GMT

Hi ninisimonishvili,
How your full path stanza looks now?

Also could you please explain why you choose this solution? Do you have different sourcetypes in one dataflow? Or maybe you can't change it at monitor stanza level?

If you need only to rename existing sourcetype, you could do that by using splunk sourcetype rename feature.
https://docs.splunk.com/Documentation/Splunk/7.0.2/Data/Renamesourcetypes

Please keep in ming that REGEX can put some overhead to indexer depending on EPS.

Re: How can I override source type?

ninisimonishvil — Tue, 29 Sep 2020 18:18:12 GMT

Hello,

to clarify, I need to change sourcetype in order to make some field and correct timestamp extraction, however I need to make these extraction not for all the event from a particular sourcetpe, that is why I'm describing sequence of event via regex where I need to change sourcetype.

now I'm just testing the syntax via test.txt file. keep uploading it after conf file alterations and restart. however getting no results.

here are the conf file configurations:

PROPS
[source::C:\Users\Administrator\Desktop\test.txt]
TRANSFORMS-sourcetype = sourcetypechange

TRANSFORMS
[sourcetypechange]
REGEX = \d{2}-\d{2}-\d{2}-\d{4}\s\d{2}:\d{2}:\d{2}\s\Local7.Info\s\d*.\d*.\d*.\d*\s\w{3}\s\d{2}\s\d{2}:\d{2}:\d{2}\s\d*.\d*.\d*.\d*\s\w\lteonOS\s\
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::Redsyslog

I checked regex and it is fine.

Re: How can I override source type?

ninisimonishvil — Tue, 29 Sep 2020 18:18:20 GMT

PROPS
[source::C:\Users\Administrator\Desktop\test.txt]
TRANSFORMS-sourcetype = sourcetypechange

TRANSFORMS

[sourcetypechange]
REGEX = \d{2}-\d{2}-\d{2}-\d{4}\s\d{2}:\d{2}:\d{2}\s\Local7.Info\s\d*.\d*.\d*.\d*\s\w{3}\s\d{2}\s\d{2}:\d{2}:\d{2}\s\d*.\d*.\d*.\d*\s\w\lteonOS\s\
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::Redsyslog

I'm changing sourcetype because need to to some field and timestamp extractions however not for all events, but only for those that start with : 02-02-12-2018 17:07:33 Local7.Info 10.5.0.11 Feb 12 17:09:32 10.5.0.11 AlteonOS

I'm using test.txt file to test if the sourcetype override works (I keep uploading it after every change, and keep restarting splunk). No result so far.

Re: How can I override source type?

serjandrosov — Mon, 05 Mar 2018 13:22:57 GMT

Regarding
https://docs.splunk.com/Documentation/Splunk/7.0.2/Admin/Propsconf

Considerations for Windows file paths:

When you specify Windows-based file paths as part of a [source::] stanza, you must escape any backslashes contained within the specified file path.

Example: [source::c:\path_to\file.txt]