topic Reassigning / replacing a sourcetype based on source in Getting Data In

Reassigning / replacing a sourcetype based on source

beataficek — Thu, 24 Oct 2024 18:10:05 GMT

I am looking to replace a sourcetype using props.conf / transforms.conf so far with no luck.

props.conf [original_sourcetype] NO_BINARY_CHECK = 1 SHOULD_LINEMERGE = false TIME_PREFIX = oldtimeprefix TIME_FORMAT=oldtimeformat pulldown_type = 1 TRANSFORMS-set_new=set_new_sourcetype [new_sourcetype_with_new_timeformat] NO_BINARY_CHECK=1 SHOULD_LINEMERGE=false TIME_PREFIX=newtimeprefix TIME_FORMAT=newtimeformat pulldown_type = 1 #rename=original_sourcetype transforms.conf [set_new_sourcetype] SOURCE_KEY = MetaData:Source REGEX = ^source::var/log/path/tofile.log FORMAT = sourcetype::new_sourcetype_with_new_timeformat DEST_KEY = MetaData:Sourcetype

tried different REGEX's, including

REGEX = var/log/path/tofile.log

Also tried setting it like this in props.conf

[source::var/log/path/tofile.log] TRANSFORMS-set_new=set_new_sourcetype

I am also looking at inputs.conf, which has monitoring stanzas for all syslog traffic, perhaps some blacklisting/ whitelisting based on source can be done there.

But I am curious as to what is not working with my props/transforms.

Thanks

Re: Reassigning / replacing a sourcetype based on source

richgalloway — Thu, 24 Oct 2024 12:26:48 GMT

The value of the REGEX attribute must be a valid regular expression that contains at least one capturing group. The expression is used to match the data in the incoming event (_raw) and cannot use key specifiers such as 'source::'. Try these transforms

[set_new_sourcetype] SOURCE_KEY = MetaData:Source REGEX = (/var/log/path/tofile.log) FORMAT = sourcetype::new_sourcetype_with_new_timeformat DEST_KEY = MetaData:Sourcetype

Re: Reassigning / replacing a sourcetype based on source

PickleRick — Thu, 24 Oct 2024 12:44:51 GMT

OK. Sourcetype rewriting is one thing but I have a hunch that you're trying to not only get a particular metadata field rewritten but would also like to reprocess whole event with the new sourcetype. And that's (depending on how much "back" you would want to go) is either much more difficult or plain impossible.

See here - https://community.splunk.com/t5/Getting-Data-In/Diagrams-of-how-indexing-works-in-the-Splunk-platform-the-Masa/m-p/590774

The event comes through all stages of the ingestion pipeline in sequence and there is no way for an event to "go back". Even if you rewrite the sourcetype, it will still get processed further "down the street" according to the props/transforms from the original sourcetype. The only difference will be that it will be written into the index with the new sourcetype and when you search your data the search-time props for the new sourcetype will be applied.

There is one "exception" - if you use CLONE_SOURCETYPE, the event will be "forked" and a new copy with the new sourcetype will be ingested again into the pipeline but it will still be after linebreaking and - as far as I remember - after timestamp parsing.