Getting Data In

Associating index file to a sourcetype, regardless of source?

NK
Explorer

I want any logfile (local, or remote via a UniversalForwarder) with the filename "xyz.log" to have a sourcetype of XYZ, and get indexed in my xyz index (not the main index).
What do I need to put in props.conf?
Do I also need to configure transforms.conf?

I'm using Splunk Enterprise v8 on Windows.

current props.conf:

[source::...\\xyz.log]
sourcetype = XYZ 

 

Labels (2)
0 Karma

NK
Explorer

For the record,

remote UF inputs.conf:

 

[monitor://C:\pathname\xyz.log]
sourcetype = XYZ
index = xyz
disabled = 0

 

and Splunk Server props.conf:

 

[source::...\\xyz.log]
sourcetype = XYZ

 


doesn't do it; the data from remote UF gets the right sourcetype, but still gets indexed into main not xyz.
(The local xyz.log data coming from the Splunk Server directory gets the right sourcetype and goes into index xyz)
Something on the Splunk Server is over-riding the explicit index specification on the remote UF?

Going to try the other recommendation for props.conf and transforms.conf

0 Karma

jotne
Builder

In your input.conf on your UF, set a fixed sourcetype for your data, example "my_logdata"

Then on your Index server or HF add the following props and transforms add the following.

 

props.conf

[my_logdata]
TRANSFORMS-changesourcetype = changesourcetype-my_logdata

 

 

transforms.conf

[changesourcetype-my_logdata]
SOURCE_KEY = MetaData:Source
REGEX = ([^\\]+).log
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::$1

 

We do use some like this to set sourcetype based on foldername

 

jotne
Builder

Fixed error in regex that did not work correctly
An even better regex to get just the file name (especially for long path name)

(?:[^\\]+\\)+([^.]+).log$
Tags (1)
0 Karma

PickleRick
SplunkTrust
SplunkTrust

You could do a source entry in props.conf (source has higher priority than host and sourcetype) so that stanza should (with all additional remarks on config file priorities) get applied to all files for this source. But.

1) You would have to have this config on all your parsing components (which means first "heavy" - based on a full Splunk Enterprise package - instance in your event path. So probably all indexers and all HFs.

2) After initial ingestion the source field doesn't have to have anything to do with actual source file name. You can easily overwrite it with inputs.conf and vice versa - your syslog-received events can have a /tmp/whatever.log source because why not.

0 Karma

NK
Explorer

@PickleRick , it's a small system (mostly for dev work): everything on 1 Wintel Splunk Server; not distributed/clustered.

A couple of remote machines that use a UF.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @NK,

did you tried to set up your inputs.conf:

[monitor:\\c:\your_path\xyz.log]
sourcetype = XYZ
index = xyz
disabled = 0

?

You can set up index and sourcetype in all the stanzas you want.

Ciao.

Giuseppe

0 Karma

NK
Explorer

@gcusello , that looks like a setup for local files.

(I have that set up via Splunk's "Data inputs" --> "Files & directories")

It does not seem to work for remote files coming in via the Splunk Universal Forwarder.

They may have different drive letters and pathnames, but the same filename (xyz.log).

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @NK,

for each input you can define index and sourcetype.

and anyway, you can override index and sourcetype on Indexers or (if present) on Heavy Forwarders.

Ciao.

Giuseppe

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...