Splunk Search

How to configure source_type to regex value?

psyched4splunk
Explorer

I'm using a Universal Forwarder and want Splunk to return source_type as what's defined for source within the monitor path(/.../).
Please see my configuration files below.

Inputs.conf

#apache
[monitor:///apps/web/test/sfagent/.../*.log.*]
sourcetype = replace_sourcetype_with_segment_5_from_source
blacklist = \.(zip|gz)$
index = web

Transforms.conf

[replace_sourcetype_with_segment_5_from_source]
SOURCE_KEY = MetaData:Source
REGEX = ^source::(?:/[^/]+){4}/([^/]+)/
FORMAT = sourcetype::replace_sourcetype_with_segment_5_from_source
DEST_KEY = MetaData:Sourcetype   

Props.conf

[replace_sourcetype_with_segment_5_from_source]
TRANSFORMS-replaceSourcetype = replace_sourcetype_with_segment_5_from_source

After some digging I discovered I can't use transforms.conf on a Universal Forwarder. Is this absolutely true?
If yes, is there a way to get Splunk to grab the source_type from the metadata and display it within the monitor path without using a heavy forwarder?

0 Karma
1 Solution

woodcock
Esteemed Legend

You cannot do this on the UF but you can on HF/Indexer like this:

props.conf:

[source:///apps/web/test/sfagent/.../*.log.*]
TRANSFORMS-replaceSourcetype = replace_sourcetype_with_segment_5_from_source

transforms.conf:

[replace_sourcetype_with_segment_5_from_source]
SOURCE_KEY = MetaData:Source
REGEX = source::(?:/[^/]+){4}/([^/]+)/
FORMAT = sourcetype::$1
DEST_KEY = MetaData:Sourcetype

If you are doing a sourcetype override/overwrite, you must use the ORIGINAL values NOT the new value, then you must deploy this to the first full instance(s) of Splunk that handles the events (usually either the HF-tier, if you use this, or your Indexer tier), restart all Splunk instances there, send in new events (old events will stay broken), then test using _index_earliest=-5m to be absolutely certain that you are only examining the newly indexed events.

View solution in original post

woodcock
Esteemed Legend

You cannot do this on the UF but you can on HF/Indexer like this:

props.conf:

[source:///apps/web/test/sfagent/.../*.log.*]
TRANSFORMS-replaceSourcetype = replace_sourcetype_with_segment_5_from_source

transforms.conf:

[replace_sourcetype_with_segment_5_from_source]
SOURCE_KEY = MetaData:Source
REGEX = source::(?:/[^/]+){4}/([^/]+)/
FORMAT = sourcetype::$1
DEST_KEY = MetaData:Sourcetype

If you are doing a sourcetype override/overwrite, you must use the ORIGINAL values NOT the new value, then you must deploy this to the first full instance(s) of Splunk that handles the events (usually either the HF-tier, if you use this, or your Indexer tier), restart all Splunk instances there, send in new events (old events will stay broken), then test using _index_earliest=-5m to be absolutely certain that you are only examining the newly indexed events.

amitm05
Builder

Can you mention whats the use case here. Knowing your end goal, the community can try to suggest you a better way around.

The intent of sourcetype is for it to identify the data structure of an event. And it determines how Splunk Enterprise should format the data during its indexing process. Trying to set a dynamic sourcetype defeats this purpose.

I'd rather suggest you use a search time extraction OR "tags" if you want to associate some meaningful info with your events based on source.

psyched4splunk
Explorer

My end goal is to extract the sourcetype and index with a regex from the monitor path at runtime based on a lookup from the directory structure.

For example in the case of apache
actual monitor path will look like:
/apps/apache/http/access/http-access.log
OR
/apps/nginx/http/access/http-error.log

input.conf

#apache or nginx
 [monitor:///apps/.../.../.../*.log.*]
 sourcetype = ( REGEX = ^source::(?:/[^/]+){1}/([^/]+)/ 😞 ( REGEX = ^source::(?:/[^/]+){2}/([^/]+)/ )
 index =  (REGEX = ^source::(?:/[^/]+){0}/([^/]+)/ )
 blacklist = \.(zip|gz)$

Desired output:

Splunk sends all apache access logs from /app/apache/http/access/http-access.log with index=apache and sourcetype = http:access
and splunk also sends all nginx error logs from /apps/nginx/http/error/http-error.log with index=nginx and sourcetype=http:error

0 Karma

kmorris_splunk
Splunk Employee
Splunk Employee

You would probably be better off explicitly setting the index and sourcetype for each path as a separate input.

0 Karma

cpetterborg
SplunkTrust
SplunkTrust

Universal forwarders DO NOT parse the data, so there can be no transforms on the UF.

As to the second part of your question, I don't think that you can, particularly on the UF. If you want to do such a think on the indexer, then that is where you should do it (or on an intermediate heavy forwarder). The UF's are not particularly helpful in looking at the data and performing any sorts of manipulation (except in the Windows Event Logs where they have made a particular use case for doing some parsing of that data).

Get Updates on the Splunk Community!

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...

Ready, Set, SOAR: How Utility Apps Can Up Level Your Playbooks!

 WATCH NOW Powering your capabilities has never been so easy with ready-made Splunk® SOAR Utility Apps. Parse ...

DevSecOps: Why You Should Care and How To Get Started

 WATCH NOW In this Tech Talk we will talk about what people mean by DevSecOps and deep dive into the different ...