Getting Data In

Associating index file to a sourcetype, regardless of source?

NK
Explorer

I want any logfile (local, or remote via a UniversalForwarder) with the filename "xyz.log" to have a sourcetype of XYZ, and get indexed in my xyz index (not the main index).
What do I need to put in props.conf?
Do I also need to configure transforms.conf?

I'm using Splunk Enterprise v8 on Windows.

current props.conf:

[source::...\\xyz.log]
sourcetype = XYZ 

 

Labels (2)
0 Karma

NK
Explorer

For the record,

remote UF inputs.conf:

 

[monitor://C:\pathname\xyz.log]
sourcetype = XYZ
index = xyz
disabled = 0

 

and Splunk Server props.conf:

 

[source::...\\xyz.log]
sourcetype = XYZ

 


doesn't do it; the data from remote UF gets the right sourcetype, but still gets indexed into main not xyz.
(The local xyz.log data coming from the Splunk Server directory gets the right sourcetype and goes into index xyz)
Something on the Splunk Server is over-riding the explicit index specification on the remote UF?

Going to try the other recommendation for props.conf and transforms.conf

0 Karma

jotne
Contributor

In your input.conf on your UF, set a fixed sourcetype for your data, example "my_logdata"

Then on your Index server or HF add the following props and transforms add the following.

 

props.conf

[my_logdata]
TRANSFORMS-changesourcetype = changesourcetype-my_logdata

 

 

transforms.conf

[changesourcetype-my_logdata]
SOURCE_KEY = MetaData:Source
REGEX = ([^\\]+).log
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::$1

 

We do use some like this to set sourcetype based on foldername

 

jotne
Contributor

Fixed error in regex that did not work correctly
An even better regex to get just the file name (especially for long path name)

(?:[^\\]+\\)+([^.]+).log$
Tags (1)
0 Karma

PickleRick
SplunkTrust
SplunkTrust

You could do a source entry in props.conf (source has higher priority than host and sourcetype) so that stanza should (with all additional remarks on config file priorities) get applied to all files for this source. But.

1) You would have to have this config on all your parsing components (which means first "heavy" - based on a full Splunk Enterprise package - instance in your event path. So probably all indexers and all HFs.

2) After initial ingestion the source field doesn't have to have anything to do with actual source file name. You can easily overwrite it with inputs.conf and vice versa - your syslog-received events can have a /tmp/whatever.log source because why not.

0 Karma

NK
Explorer

@PickleRick , it's a small system (mostly for dev work): everything on 1 Wintel Splunk Server; not distributed/clustered.

A couple of remote machines that use a UF.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @NK,

did you tried to set up your inputs.conf:

[monitor:\\c:\your_path\xyz.log]
sourcetype = XYZ
index = xyz
disabled = 0

?

You can set up index and sourcetype in all the stanzas you want.

Ciao.

Giuseppe

0 Karma

NK
Explorer

@gcusello , that looks like a setup for local files.

(I have that set up via Splunk's "Data inputs" --> "Files & directories")

It does not seem to work for remote files coming in via the Splunk Universal Forwarder.

They may have different drive letters and pathnames, but the same filename (xyz.log).

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @NK,

for each input you can define index and sourcetype.

and anyway, you can override index and sourcetype on Indexers or (if present) on Heavy Forwarders.

Ciao.

Giuseppe

0 Karma
Get Updates on the Splunk Community!

Streamline Data Ingestion With Deployment Server Essentials

REGISTER NOW!Every day the list of sources Admins are responsible for gets bigger and bigger, often making the ...

Remediate Threats Faster and Simplify Investigations With Splunk Enterprise Security ...

REGISTER NOW!Join us for a Tech Talk around our latest release of Splunk Enterprise Security 7.2! We’ll walk ...

Introduction to Splunk AI

WATCH NOWHow are you using AI in Splunk? Whether you see AI as a threat or opportunity, AI is here to stay. ...