topic Re: Is it possible to have separate indexes within a single monitored directory? in Getting Data In

Is it possible to have separate indexes within a single monitored directory?

cmeyers — Mon, 24 Oct 2016 18:40:16 GMT

I have a single directory being monitored. Via Splunk GUI, you can only select a single index for the logs to be outputted to. There are dozens of different types of logs, but there is a specific kind of log (.log files) that I would like indexed into a separate index.

The help I have seen so far shows to explicitly list all file types and direct accordingly. I would like to only have to list the ones I would like redirected.

Any help would be appreciated! Thanks!

Re: Is it possible to have separate indexes within a single monitored directory?

koshyk — Mon, 24 Oct 2016 19:11:48 GMT

In inputs.conf file
Option1: Monitor specific files only

[monitor:///var/log/filetype1*.log]
sourcetype = mysourcetype1
index = myindex1

[monitor:///var/log/filetype2*.log]
sourcetype = mysourcetype2
index = myindex2

Option2: Use combination of Blacklist if the first filter is too generic

    [monitor:///var/log/filetype*.log]
    sourcetype = mysourcetype1
    index = myindex1
   # Below will blacklist if you want to protect some files that need not to be indexed
    [blacklist:///var/log/filetype2*.log]

Option3: Use combination of inputs, props & transforms
inputs.conf

 [monitor:///var/log/filetype*.log]
sourcetype = mysourcetype1

In props.conf

[mysourcetype1]
 TRANSFORMS-set_index_myindex = set_index_myindex

Which references this in transforms.conf:

 [set_index_myindex]
 SOURCE_KEY = MetaData:Host
 REGEX = ^host::192\.168\.1\.1$
 FORMAT = index::myindex1
 DEST_KEY = _MetaData:Index

Re: Is it possible to have separate indexes within a single monitored directory?

lukejadamec — Mon, 24 Oct 2016 19:12:33 GMT

I have input directories with 5 different index destinations based on the file name of the log file in the directory. To do this I use the Splunk file directory monitor UI and introduce wildcards in the source to differentiate different log names types.
Can you post the names of the log files in the directory and how you want to separate them to different indexes?

Re: Is it possible to have separate indexes within a single monitored directory?

riqbal — Mon, 21 May 2018 12:46:36 GMT

1- I have one syslog server where all the network devices sending logs and that logs are saving at
/central/$hostname$/$hostname$.log
2- I install UF on that syslog server and configure it to send logs to HF.

whats should be my regex here

Re: Is it possible to have separate indexes within a single monitored directory?

koshyk — Fri, 25 May 2018 14:25:11 GMT

personally, i feel you need to do filtering at syslog level, so your syslog will log it in better way so you can identify the source/ip etc. as early as possible.

Please try changing your syslog template so it will log as
/central/$host-ip/$port/$facility.$severity.log

This will ensure, the data from each source-ip , from each facility & port is logged separately as possible. (better to use host-ip than hostname as hostname is not guaranteed)

You can do via regex in splunk, but that's harder work than splitting at source if possible