Getting Data In

Is it possible to have separate indexes within a single monitored directory?

cmeyers
Explorer

I have a single directory being monitored. Via Splunk GUI, you can only select a single index for the logs to be outputted to. There are dozens of different types of logs, but there is a specific kind of log (.log files) that I would like indexed into a separate index.

The help I have seen so far shows to explicitly list all file types and direct accordingly. I would like to only have to list the ones I would like redirected.

Any help would be appreciated! Thanks!

0 Karma
1 Solution

koshyk
Super Champion

In inputs.conf file
Option1: Monitor specific files only

[monitor:///var/log/filetype1*.log]
sourcetype = mysourcetype1
index = myindex1

[monitor:///var/log/filetype2*.log]
sourcetype = mysourcetype2
index = myindex2

Option2: Use combination of Blacklist if the first filter is too generic

    [monitor:///var/log/filetype*.log]
    sourcetype = mysourcetype1
    index = myindex1
   # Below will blacklist if you want to protect some files that need not to be indexed
    [blacklist:///var/log/filetype2*.log]  

Option3: Use combination of inputs, props & transforms
inputs.conf

 [monitor:///var/log/filetype*.log]
sourcetype = mysourcetype1

In props.conf

[mysourcetype1]
 TRANSFORMS-set_index_myindex = set_index_myindex

Which references this in transforms.conf:

 [set_index_myindex]
 SOURCE_KEY = MetaData:Host
 REGEX = ^host::192\.168\.1\.1$
 FORMAT = index::myindex1
 DEST_KEY = _MetaData:Index

View solution in original post

lukejadamec
Super Champion

I have input directories with 5 different index destinations based on the file name of the log file in the directory. To do this I use the Splunk file directory monitor UI and introduce wildcards in the source to differentiate different log names types.
Can you post the names of the log files in the directory and how you want to separate them to different indexes?

0 Karma

koshyk
Super Champion

In inputs.conf file
Option1: Monitor specific files only

[monitor:///var/log/filetype1*.log]
sourcetype = mysourcetype1
index = myindex1

[monitor:///var/log/filetype2*.log]
sourcetype = mysourcetype2
index = myindex2

Option2: Use combination of Blacklist if the first filter is too generic

    [monitor:///var/log/filetype*.log]
    sourcetype = mysourcetype1
    index = myindex1
   # Below will blacklist if you want to protect some files that need not to be indexed
    [blacklist:///var/log/filetype2*.log]  

Option3: Use combination of inputs, props & transforms
inputs.conf

 [monitor:///var/log/filetype*.log]
sourcetype = mysourcetype1

In props.conf

[mysourcetype1]
 TRANSFORMS-set_index_myindex = set_index_myindex

Which references this in transforms.conf:

 [set_index_myindex]
 SOURCE_KEY = MetaData:Host
 REGEX = ^host::192\.168\.1\.1$
 FORMAT = index::myindex1
 DEST_KEY = _MetaData:Index

riqbal
Communicator

1- I have one syslog server where all the network devices sending logs and that logs are saving at
/central/$hostname$/$hostname$.log
2- I install UF on that syslog server and configure it to send logs to HF.

whats should be my regex here

0 Karma

koshyk
Super Champion

personally, i feel you need to do filtering at syslog level, so your syslog will log it in better way so you can identify the source/ip etc. as early as possible.

Please try changing your syslog template so it will log as
/central/$host-ip/$port/$facility.$severity.log

This will ensure, the data from each source-ip , from each facility & port is logged separately as possible. (better to use host-ip than hostname as hostname is not guaranteed)

You can do via regex in splunk, but that's harder work than splitting at source if possible

0 Karma
Get Updates on the Splunk Community!

Monitoring MariaDB and MySQL

In a previous post, we explored monitoring PostgreSQL and general best practices around which metrics to ...

Financial Services Industry Use Cases, ITSI Best Practices, and More New Articles ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Splunk Federated Analytics for Amazon Security Lake

Thursday, November 21, 2024  |  11AM PT / 2PM ET Register Now Join our session to see the technical ...