Getting Data In

Associating index file to a sourcetype, regardless of source?

NK
Explorer

I want any logfile (local, or remote via a UniversalForwarder) with the filename "xyz.log" to have a sourcetype of XYZ, and get indexed in my xyz index (not the main index).
What do I need to put in props.conf?
Do I also need to configure transforms.conf?

I'm using Splunk Enterprise v8 on Windows.

current props.conf:

[source::...\\xyz.log]
sourcetype = XYZ 

 

Labels (2)
0 Karma

NK
Explorer

For the record,

remote UF inputs.conf:

 

[monitor://C:\pathname\xyz.log]
sourcetype = XYZ
index = xyz
disabled = 0

 

and Splunk Server props.conf:

 

[source::...\\xyz.log]
sourcetype = XYZ

 


doesn't do it; the data from remote UF gets the right sourcetype, but still gets indexed into main not xyz.
(The local xyz.log data coming from the Splunk Server directory gets the right sourcetype and goes into index xyz)
Something on the Splunk Server is over-riding the explicit index specification on the remote UF?

Going to try the other recommendation for props.conf and transforms.conf

0 Karma

jotne
Builder

In your input.conf on your UF, set a fixed sourcetype for your data, example "my_logdata"

Then on your Index server or HF add the following props and transforms add the following.

 

props.conf

[my_logdata]
TRANSFORMS-changesourcetype = changesourcetype-my_logdata

 

 

transforms.conf

[changesourcetype-my_logdata]
SOURCE_KEY = MetaData:Source
REGEX = ([^\\]+).log
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::$1

 

We do use some like this to set sourcetype based on foldername

 

jotne
Builder

Fixed error in regex that did not work correctly
An even better regex to get just the file name (especially for long path name)

(?:[^\\]+\\)+([^.]+).log$
Tags (1)
0 Karma

PickleRick
SplunkTrust
SplunkTrust

You could do a source entry in props.conf (source has higher priority than host and sourcetype) so that stanza should (with all additional remarks on config file priorities) get applied to all files for this source. But.

1) You would have to have this config on all your parsing components (which means first "heavy" - based on a full Splunk Enterprise package - instance in your event path. So probably all indexers and all HFs.

2) After initial ingestion the source field doesn't have to have anything to do with actual source file name. You can easily overwrite it with inputs.conf and vice versa - your syslog-received events can have a /tmp/whatever.log source because why not.

0 Karma

NK
Explorer

@PickleRick , it's a small system (mostly for dev work): everything on 1 Wintel Splunk Server; not distributed/clustered.

A couple of remote machines that use a UF.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @NK,

did you tried to set up your inputs.conf:

[monitor:\\c:\your_path\xyz.log]
sourcetype = XYZ
index = xyz
disabled = 0

?

You can set up index and sourcetype in all the stanzas you want.

Ciao.

Giuseppe

0 Karma

NK
Explorer

@gcusello , that looks like a setup for local files.

(I have that set up via Splunk's "Data inputs" --> "Files & directories")

It does not seem to work for remote files coming in via the Splunk Universal Forwarder.

They may have different drive letters and pathnames, but the same filename (xyz.log).

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @NK,

for each input you can define index and sourcetype.

and anyway, you can override index and sourcetype on Indexers or (if present) on Heavy Forwarders.

Ciao.

Giuseppe

0 Karma
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In September, the Splunk Threat Research Team had two releases of new security content via the Enterprise ...

New in Observability - Improvements to Custom Metrics SLOs, Log Observer Connect & ...

The latest enhancements to the Splunk observability portfolio deliver improved SLO management accuracy, better ...

Improve Data Pipelines Using Splunk Data Management

  Register Now   This Tech Talk will explore the pipeline management offerings Edge Processor and Ingest ...