topic Re: fully reindexing a file every time the datestamp changes in Getting Data In

fully reindexing a file every time the datestamp changes

mjones414 — Tue, 29 Sep 2020 23:50:23 GMT

I've a few different automated pulls of data into directories of files I want splunk to index. These files get completely overwritten every night at least, but sometimes more often than that depending on different operational conditions out of my control. I need splunk to reindex these files every time the datestamp changes and that doesn't appear to be working. current props configurations:

[source:: /data/ridiculi/all_group/ridiculi.*]
CHECK_METHOD = modtime

[ridiculi:group]
DATETIME_CONFIG = CURRENT
FIELD_DELIMITER = ":"
FIELD_NAMES = gid,status,gidnumber,members
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
category = Operating System
description = Ridiculi
disabled = false
pulldown_type = true

inputs.conf:

[monitor:///data/ridiculi/all_group/ridiculi.wanker]
disabled = false
sourcetype = ridiculi:group
index=ridiculi

Re: fully reindexing a file every time the datestamp changes

somesoni2 — Fri, 29 Mar 2019 15:08:14 GMT

Could you also post first few lines from the file?
Also, The props.conf with [source:..., did you place it in the forwarder (same host as where your inputs.conf lives)?

Re: fully reindexing a file every time the datestamp changes

adonio — Fri, 29 Mar 2019 15:08:27 GMT

where is your props.conf? iirc the top portion has to be on the forwarder:

[source:: /data/ridiculi/all_group/ridiculi.*]
CHECK_METHOD = modtime

the rest will be on the indexer

Re: fully reindexing a file every time the datestamp changes

harsmarvania57 — Fri, 29 Mar 2019 15:11:37 GMT

Are you running Universal Forwarder to read /data/ridiculi/all_group/ridiculi.wanker file ? If yes then below props.conf will not work on Universal Forwarder.

[ridiculi:group]
DATETIME_CONFIG = CURRENT
FIELD_DELIMITER = ":"
FIELD_NAMES = gid,status,gidnumber,members
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
category = Operating System
description = Ridiculi
disabled = false
pulldown_type = true

You need to configure above props.conf configuration on first Splunk Enterprise instance from Universal Forwarder because parsing happens on full splunk instance not on UF.

Re: fully reindexing a file every time the datestamp changes

mjones414 — Fri, 29 Mar 2019 15:54:53 GMT

I'm actually running this on the splunk distributed search head in an app context. The props.conf should be ina distribution bundle that goes to the indexers. The splunk distributed search head does output to an output queue of indexers. Is there something else missing from that config that I need?

Re: fully reindexing a file every time the datestamp changes

harsmarvania57 — Fri, 29 Mar 2019 15:57:51 GMT

On which instance you are monitoring /data/ridiculi/all_group/ridiculi.wanker logfile ? Search Head or Universal Forwarder ?

Re: fully reindexing a file every time the datestamp changes

mjones414 — Fri, 29 Mar 2019 15:57:54 GMT

sure.

First few lines from a file:
Admins:NISG:123123:jim,bob,joe
Users:NISG:456456:alpha,whiskey,tango

Re: fully reindexing a file every time the datestamp changes

mjones414 — Fri, 29 Mar 2019 16:06:32 GMT

Its in etc/apps/ridiculi/local/props.conf and inputs.conf respectively on the distributed search head. the /data path is an autofs mount point that the splunk search head can read (other files are being indexed over /data both from this search head and from indexers as required.)

Re: fully reindexing a file every time the datestamp changes

mjones414 — Fri, 29 Mar 2019 16:07:52 GMT

Permissions on all files are 644. permissions on directory are 2644. Filesystem is NFSv3.

Re: fully reindexing a file every time the datestamp changes

mjones414 — Tue, 02 Apr 2019 12:41:52 GMT

Just a bump on this to see if there were any more ideas? About to open a case with Splunk support -- as it seems whats here should be sufficient.

Re: fully reindexing a file every time the datestamp changes

mjones414 — Tue, 02 Apr 2019 12:42:07 GMT

Just a bump on this to see if there were any more ideas? About to open a case with Splunk support -- as it seems whats here should be sufficient.

Re: fully reindexing a file every time the datestamp changes

adonio — Wed, 30 Sep 2020 00:00:53 GMT

the answer is between the lines ... where is your props.conf that has the:
[source:: /data/ridiculi/all_group/ridiculi.*]
CHECK_METHOD = modtime
it supposed to be on the instance that collects the data

Re: fully reindexing a file every time the datestamp changes

mjones414 — Tue, 02 Apr 2019 12:51:25 GMT

In my situation that is true. the /data directory is monitored by the splunk search head and the props.conf is also on the splunk search head. There is no forwarder involved in this particular data input.

Re: fully reindexing a file every time the datestamp changes

pdaigle_splunk — Tue, 29 Sep 2020 23:57:11 GMT

This is a log file rotation setup where you need to use the crcSalt bit configuration and/or the initCrcLength attribute:

https://docs.splunk.com/Documentation/Splunk/7.2.5/Data/Howlogfilerotationishandled

The winning combination was CHECK_METHOD combined with setting crcSalt to something like REINDEX_ALWAYS. Because the file has similar or almost the same data, more than likely the CRC Checksum value and size is the same and Splunk will skip the log file even if the time and date of the file has changed.

Re: fully reindexing a file every time the datestamp changes

mjones414 — Thu, 04 Apr 2019 19:19:19 GMT

Thank you for posting this Paul!

Re: fully reindexing a file every time the datestamp changes

pdaigle_splunk — Thu, 04 Apr 2019 19:21:59 GMT

You're welcome! Just sharing in case others in the community run across the same issue. 🙂