Deployment Architecture

Search heads re-indexing files continously

alekksi
Communicator

Hi all,

I'm having an issue with files in the /etc partition being indexed continuously. Yesterday morning a file was automatically deployed using ansible, which caused all the clustered search heads on site 3 to start re-indexing this file (/etc/krb5.conf). When I checked on site 2, it turns out that it had been going on even longer and had a wider scope of files.

Restarting Splunk and the servers yields no results. When Splunk is shut down, there are no remaining handles on the file. Nothing appears to be wrong, except that Splunk keeps reindexing the files.

Example logs from one of the worst-affected servers:

02-22-2017 09:34:23.844 +0000 INFO  WatchedFile - Will begin reading at offset=0 for file='/etc/avahi/avahi-daemon.conf'.
02-22-2017 09:34:23.844 +0000 INFO  WatchedFile - Will begin reading at offset=0 for file='/etc/security/namespace.conf'.
02-22-2017 09:34:23.846 +0000 INFO  WatchedFile - Will begin reading at offset=0 for file='/etc/security/namespace.conf'.
02-22-2017 09:34:23.846 +0000 INFO  WatchedFile - Will begin reading at offset=0 for file='/etc/security/time.conf'.
02-22-2017 09:34:23.848 +0000 INFO  WatchedFile - Will begin reading at offset=0 for file='/etc/security/time.conf'.
02-22-2017 09:34:23.848 +0000 INFO  WatchedFile - Will begin reading at offset=0 for file='/etc/security/group.conf'.
02-22-2017 09:34:23.850 +0000 INFO  WatchedFile - Will begin reading at offset=0 for file='/etc/security/group.conf'.
02-22-2017 09:34:23.850 +0000 INFO  WatchedFile - Will begin reading at offset=0 for file='/etc/security/access.conf'.
02-22-2017 09:34:23.851 +0000 INFO  WatchedFile - Will begin reading at offset=0 for file='/etc/security/access.conf'.
02-22-2017 09:34:23.851 +0000 INFO  WatchedFile - Will begin reading at offset=0 for file='/etc/security/sepermit.conf'.
02-22-2017 09:34:23.853 +0000 INFO  WatchedFile - Will begin reading at offset=0 for file='/etc/security/sepermit.conf'.
02-22-2017 09:34:23.853 +0000 INFO  WatchedFile - Will begin reading at offset=0 for file='/etc/security/pam_env.conf'.
02-22-2017 09:34:23.855 +0000 INFO  WatchedFile - Will begin reading at offset=0 for file='/etc/security/pam_env.conf'.
02-22-2017 09:34:23.855 +0000 INFO  WatchedFile - Will begin reading at offset=0 for file='/etc/security/limits.conf'.
02-22-2017 09:34:23.856 +0000 INFO  WatchedFile - Will begin reading at offset=0 for file='/etc/security/limits.conf'.
02-22-2017 09:34:23.857 +0000 INFO  WatchedFile - Will begin reading at offset=0 for file='/etc/mc/filehighlight.ini'.
02-22-2017 09:34:23.859 +0000 INFO  WatchedFile - Will begin reading at offset=0 for file='/etc/mc/filehighlight.ini'.
02-22-2017 09:34:23.859 +0000 INFO  WatchedFile - Will begin reading at offset=0 for file='/etc/mcelog/mcelog.conf'.
02-22-2017 09:34:23.861 +0000 INFO  WatchedFile - Will begin reading at offset=0 for file='/etc/mcelog/mcelog.conf'.
02-22-2017 09:34:23.861 +0000 INFO  WatchedFile - Will begin reading at offset=0 for file='/etc/abrt/abrt.conf'.
02-22-2017 09:34:23.862 +0000 INFO  WatchedFile - Will begin reading at offset=0 for file='/etc/abrt/abrt.conf'.
02-22-2017 09:34:23.862 +0000 INFO  WatchedFile - Will begin reading at offset=0 for file='/etc/abrt/abrt-action-save-package-data.conf'.
02-22-2017 09:34:23.864 +0000 INFO  WatchedFile - Will begin reading at offset=0 for file='/etc/abrt/abrt-action-save-package-data.conf'.
02-22-2017 09:34:23.865 +0000 INFO  WatchedFile - Will begin reading at offset=0 for file='/etc/foomatic/filter.conf'.
02-22-2017 09:34:23.866 +0000 INFO  WatchedFile - Will begin reading at offset=0 for file='/etc/foomatic/filter.conf'.
02-22-2017 09:34:23.866 +0000 INFO  WatchedFile - Will begin reading at offset=0 for file='/etc/nfsmount.conf'.
02-22-2017 09:34:23.868 +0000 INFO  WatchedFile - Will begin reading at offset=0 for file='/etc/nfsmount.conf'.
02-22-2017 09:34:23.868 +0000 INFO  WatchedFile - Will begin reading at offset=0 for file='/etc/sos.conf'.
02-22-2017 09:34:23.869 +0000 INFO  WatchedFile - Will begin reading at offset=0 for file='/etc/sos.conf'.
02-22-2017 09:34:23.869 +0000 INFO  WatchedFile - Will begin reading at offset=0 for file='/etc/kdump.conf'.
02-22-2017 09:34:23.871 +0000 INFO  WatchedFile - Will begin reading at offset=0 for file='/etc/kdump.conf'.
02-22-2017 09:34:23.871 +0000 INFO  WatchedFile - Will begin reading at offset=0 for file='/etc/dracut.conf'.
02-22-2017 09:34:23.872 +0000 INFO  WatchedFile - Will begin reading at offset=0 for file='/etc/dracut.conf'.
02-22-2017 09:34:23.872 +0000 INFO  WatchedFile - Will begin reading at offset=0 for file='/etc/krb5.conf'.
02-22-2017 09:34:23.874 +0000 INFO  WatchedFile - Will begin reading at offset=0 for file='/etc/krb5.conf'.
02-22-2017 09:34:23.874 +0000 INFO  WatchedFile - Will begin reading at offset=0 for file='/etc/yum.conf'.
02-22-2017 09:34:23.875 +0000 INFO  WatchedFile - Will begin reading at offset=0 for file='/etc/yum.conf'.

Some DEBUG logs:

02-21-2017 17:18:46.202 +0000 INFO  WatchedFile - Will begin reading at offset=0 for file='/etc/krb5.conf'.
02-21-2017 17:18:46.203 +0000 INFO  WatchedFile - Will begin reading at offset=0 for file='/etc/krb5.conf'.
02-21-2017 17:19:42.979 +0000 DEBUG PropertiesMapConfig - Pattern 'krb5.conf-too_small' matches with priority 100
02-21-2017 17:19:43.885 +0000 DEBUG PropertiesMapConfig - Pattern 'krb5.conf-too_small' matches with priority 100
02-21-2017 17:19:43.990 +0000 DEBUG PropertiesMapConfig - Pattern 'krb5.conf-too_small' matches with priority 100
02-21-2017 17:19:44.164 +0000 DEBUG PropertiesMapConfig - Pattern 'krb5.conf-too_small' matches with priority 100
02-21-2017 17:19:44.196 +0000 DEBUG TailingProcessor - File state notification for path='/etc/krb5.conf' (first time).
02-21-2017 17:19:44.197 +0000 DEBUG TailingProcessor - File state notification for path='/etc/krb5.conf.2017-02-21-09:01:42' (first time).
02-21-2017 17:19:44.205 +0000 DEBUG TailReader - Enqueued file=/etc/krb5.conf in tailreader0
02-21-2017 17:19:44.205 +0000 DEBUG TailReader - Enqueued file=/etc/krb5.conf.2017-02-21-09:01:42 in tailreader0
02-21-2017 17:19:44.418 +0000 DEBUG TailReader - Start reading file="/etc/krb5.conf" in tailreader0 thread
02-21-2017 17:19:44.418 +0000 DEBUG PropertiesMapConfig - Performing pattern matching for: source::/etc/krb5.conf
02-21-2017 17:19:44.418 +0000 INFO  WatchedFile - Will use tracking rule=modtime for file='/etc/krb5.conf'.
02-21-2017 17:19:44.418 +0000 DEBUG TailingProcessor -   Item '/etc/krb5.conf' matches stanza: /etc.
02-21-2017 17:19:44.418 +0000 DEBUG TailingProcessor -   Skipping itemPath='/etc/krb5.conf', does not match path='/opt/splunk-sh/etc/splunk.version' :Not a directory :Not a symlink
02-21-2017 17:19:44.418 +0000 DEBUG TailingProcessor -   Skipping itemPath='/etc/krb5.conf', does not match path='/opt/splunk-sh/var/log/introspection' :Not a directory :Not a symlink
02-21-2017 17:19:44.418 +0000 DEBUG TailingProcessor -   Skipping itemPath='/etc/krb5.conf', does not match path='/opt/splunk-sh/var/log/splunk' :Not a directory :Not a symlink
02-21-2017 17:19:44.419 +0000 DEBUG TailingProcessor -   Skipping itemPath='/etc/krb5.conf', does not match path='/opt/splunk-sh/var/log/splunk/license_usage_summary.log' :Not a directory :Not a symlink
02-21-2017 17:19:44.419 +0000 DEBUG TailingProcessor -   Skipping itemPath='/etc/krb5.conf', does not match path='/opt/splunk-sh/var/spool/splunk' :Not a directory :Not a symlink
02-21-2017 17:19:44.419 +0000 DEBUG TailingProcessor -   Skipping itemPath='/etc/krb5.conf', does not match path='/opt/splunk-sh/var/spool/splunk' :Not a directory :Not a symlink
02-21-2017 17:19:44.419 +0000 DEBUG TailingProcessor -   Skipping itemPath='/etc/krb5.conf', does not match path='/var/log' :Not a directory :Not a symlink
02-21-2017 17:19:44.419 +0000 DEBUG FilesystemFilter - Testing path=/etc/krb5.conf(real=/etc/krb5.conf) with global blacklisted paths
02-21-2017 17:19:44.419 +0000 DEBUG TailReader -   Will attempt to read file: /etc/krb5.conf.
02-21-2017 17:19:44.419 +0000 DEBUG PropertiesMapConfig - Performing pattern matching for: source::/etc/krb5.conf
02-21-2017 17:19:44.419 +0000 DEBUG FileClassifierManager - Finding type for file: /etc/krb5.conf
02-21-2017 17:19:44.419 +0000 DEBUG PropertiesMapConfig - Performing pattern matching for: source::/etc/krb5.conf
02-21-2017 17:19:44.419 +0000 DEBUG PropertiesMapConfig - Performing pattern matching for: source::/etc/krb5.conf|config_file
02-21-2017 17:19:44.419 +0000 DEBUG PropertiesMapConfig - Performing pattern matching for: source::/etc/krb5.conf|config_file
02-21-2017 17:19:44.419 +0000 DEBUG FileClassifierManager - filename="/etc/krb5.conf" invalidCharCount="0" TotalCharCount="301" PercentInvalid="0.000000"
02-21-2017 17:19:44.419 +0000 DEBUG WatchedFile - Storing pending metadata for file=/etc/krb5.conf, sourcetype=config_file, charset=UTF-8
02-21-2017 17:19:44.419 +0000 DEBUG PropertiesMapConfig - Performing pattern matching for: source::/etc/krb5.conf|host::splunk-site3-001.local|config_file|82
02-21-2017 17:19:44.420 +0000 DEBUG WatchedFile -   Attempting to load indexed extractions config from conf=source::/etc/krb5.conf|host::splunk-site3-001.local|config_file|82 ...
02-21-2017 17:19:44.420 +0000 DEBUG WatchedFile - /etc/krb5.conf requested modtime-based check.
02-21-2017 17:19:44.420 +0000 DEBUG PropertiesMapConfig - Performing pattern matching for: source::/etc/krb5.conf|host::splunk-site3-001.local|config_file|83
02-21-2017 17:19:44.420 +0000 DEBUG WatchedFile -   Attempting to load indexed extractions config from conf=source::/etc/krb5.conf|host::splunk-site3-001.local|config_file|83 ...
02-21-2017 17:19:44.420 +0000 DEBUG TailReader - About to read data (Opening file: /etc/krb5.conf).
02-21-2017 17:19:44.420 +0000 DEBUG WatchedFile - seeking /etc/krb5.conf to off=0
02-21-2017 17:19:44.421 +0000 DEBUG PropertiesMapConfig - Performing pattern matching for: source::/etc/krb5.conf|host::splunk-site3-001.local|config_file|83
02-21-2017 17:19:44.421 +0000 DEBUG WatchedFile - Reached EOF: fname=/etc/krb5.conf fishstate=key=0xaaeb7ddeef141e73 sptr=301 scrc=0x0 fnamecrc=0x8e77b1ba24e13795 modtime=1487697423
02-21-2017 17:19:44.421 +0000 INFO  UTF8Processor - Converting using CHARSET="UTF-8" for conf "source::/etc/krb5.conf|host::splunk-site3-001.local|config_file|83"
02-21-2017 17:19:44.421 +0000 INFO  LineBreakingProcessor - Using truncation length 1000000 for conf "source::/etc/krb5.conf|host::splunk-site3-001.local|config_file|83"
02-21-2017 17:19:44.421 +0000 INFO  LineBreakingProcessor - Using lookbehind 100 for conf "source::/etc/krb5.conf|host::splunk-site3-001.local|config_file|83"
02-21-2017 17:19:44.421 +0000 DEBUG TailReader - Finished reading file='/etc/krb5.conf' in tailreader0 thread, disposition=NO_DISPOSITION, deferredBy=3000
02-21-2017 17:19:44.421 +0000 DEBUG TailReader - Defering notification for file=/etc/krb5.conf by 3000ms
02-21-2017 17:19:44.421 +0000 DEBUG UTF8Processor - Done key received for: source::/etc/krb5.conf|host::splunk-site3-001.local|config_file|83
02-21-2017 17:19:44.421 +0000 INFO  AggregatorMiningProcessor - Setting up line merging apparatus for: source::/etc/krb5.conf|host::splunk-site3-001.local|config_file|83
02-21-2017 17:19:44.421 +0000 DEBUG AggregatorMiningProcessor - Failed to parse timestamp. Defaulting to time specified by data input. - data_source="/etc/krb5.conf", data_host="splunk-site3-001.local", data_sourcetype="config_file"
02-21-2017 17:19:44.421 +0000 INFO  AggregatorMiningProcessor - Got done message for: source::/etc/krb5.conf|host::splunk-site3-001.local|config_file|83
02-21-2017 17:19:44.427 +0000 DEBUG TcpOutputProc - Pushed eventId=253 on chanId=51 to back of tcp client (tcp output) queue. source:source::/etc/krb5.conf|host::splunk-site3-001.local|config_file|83
02-21-2017 17:19:44.427 +0000 DEBUG TcpOutputProc - Pushed eventId=254 on chanId=51 to back of tcp client (tcp output) queue. source:source::/etc/krb5.conf|host::splunk-site3-001.local|config_file|83
02-21-2017 17:19:44.427 +0000 DEBUG S2SSender - Created new channel_code=10 for conf="source::/etc/krb5.conf|host::splunk-site3-001.local|config_file|83", unique_id=51
02-21-2017 17:19:44.519 +0000 DEBUG TailReader - Start reading file="/etc/krb5.conf.2017-02-21-09:01:42" in tailreader0 thread
02-21-2017 17:19:44.519 +0000 DEBUG PropertiesMapConfig - Performing pattern matching for: source::/etc/krb5.conf.2017-02-21-09:01:42
02-21-2017 17:19:44.519 +0000 DEBUG TailingProcessor -   Item '/etc/krb5.conf.2017-02-21-09:01:42' matches stanza: /etc.
02-21-2017 17:19:44.519 +0000 DEBUG TailingProcessor -   Skipping itemPath='/etc/krb5.conf.2017-02-21-09:01:42', does not match path='/opt/splunk-sh/etc/splunk.version' :Not a directory :Not a symlink
02-21-2017 17:19:44.519 +0000 DEBUG TailingProcessor -   Skipping itemPath='/etc/krb5.conf.2017-02-21-09:01:42', does not match path='/opt/splunk-sh/var/log/introspection' :Not a directory :Not a symlink
02-21-2017 17:19:44.519 +0000 DEBUG TailingProcessor -   Skipping itemPath='/etc/krb5.conf.2017-02-21-09:01:42', does not match path='/opt/splunk-sh/var/log/splunk' :Not a directory :Not a symlink
02-21-2017 17:19:44.519 +0000 DEBUG TailingProcessor -   Skipping itemPath='/etc/krb5.conf.2017-02-21-09:01:42', does not match path='/opt/splunk-sh/var/log/splunk/license_usage_summary.log' :Not a directory :Not a symlink
02-21-2017 17:19:44.519 +0000 DEBUG TailingProcessor -   Skipping itemPath='/etc/krb5.conf.2017-02-21-09:01:42', does not match path='/opt/splunk-sh/var/spool/splunk' :Not a directory :Not a symlink
02-21-2017 17:19:44.519 +0000 DEBUG TailingProcessor -   Skipping itemPath='/etc/krb5.conf.2017-02-21-09:01:42', does not match path='/opt/splunk-sh/var/spool/splunk' :Not a directory :Not a symlink
02-21-2017 17:19:44.519 +0000 DEBUG TailingProcessor -   Skipping itemPath='/etc/krb5.conf.2017-02-21-09:01:42', does not match path='/var/log' :Not a directory :Not a symlink
02-21-2017 17:19:44.519 +0000 DEBUG FilesystemFilter - Testing path=/etc/krb5.conf.2017-02-21-09:01:42(real=/etc/krb5.conf.2017-02-21-09:01:42) with global blacklisted paths
02-21-2017 17:19:44.519 +0000 DEBUG TailReader -   Will attempt to read file: /etc/krb5.conf.2017-02-21-09:01:42.
02-21-2017 17:19:44.519 +0000 DEBUG PropertiesMapConfig - Performing pattern matching for: source::/etc/krb5.conf.2017-02-21-09:01:42
02-21-2017 17:19:44.520 +0000 DEBUG FileClassifierManager - Finding type for file: /etc/krb5.conf.2017-02-21-09:01:42
02-21-2017 17:19:44.520 +0000 DEBUG PropertiesMapConfig - Performing pattern matching for: source::/etc/krb5.conf.2017-02-21-09:01:42
02-21-2017 17:19:44.520 +0000 DEBUG PropertiesMapConfig - Performing pattern matching for: source::/etc/krb5.conf.2017-02-21-09:01:42
02-21-2017 17:19:44.520 +0000 DEBUG PropertiesMapConfig - Performing pattern matching for: source::/etc/krb5.conf.2017-02-21-09:01:42
02-21-2017 17:19:44.520 +0000 DEBUG FileClassifierManager - filename="/etc/krb5.conf.2017-02-21-09:01:42" invalidCharCount="0" TotalCharCount="241" PercentInvalid="0.000000"
02-21-2017 17:19:44.520 +0000 DEBUG PropertiesMapConfig - Pattern 'krb5.conf-too_small' matches with priority 100
02-21-2017 17:19:44.520 +0000 DEBUG TailReader -   Got classified_sourcetype='krb5.conf-too_small' and classified_charset='UTF-8'.
02-21-2017 17:19:44.520 +0000 DEBUG WatchedFile - Storing pending metadata for file=/etc/krb5.conf.2017-02-21-09:01:42, sourcetype=krb5.conf-too_small, charset=UTF-8
02-21-2017 17:19:44.520 +0000 DEBUG PropertiesMapConfig - Performing pattern matching for: krb5.conf-too_small
02-21-2017 17:19:44.521 +0000 DEBUG PropertiesMapConfig - Pattern 'krb5.conf-too_small' matches with priority 100
02-21-2017 17:19:44.521 +0000 DEBUG PropertiesMapConfig - Performing pattern matching for: source::/etc/krb5.conf.2017-02-21-09:01:42|host::splunk-site3-001.local|krb5.conf-too_small|110
02-21-2017 17:19:44.521 +0000 DEBUG PropertiesMapConfig - Pattern 'krb5.conf-too_small' matches with priority 100
02-21-2017 17:19:44.521 +0000 DEBUG WatchedFile -   Attempting to load indexed extractions config from conf=source::/etc/krb5.conf.2017-02-21-09:01:42|host::splunk-site3-001.local|krb5.conf-too_small|110 ...
02-21-2017 17:19:44.521 +0000 DEBUG WatchedFile - /etc/krb5.conf.2017-02-21-09:01:42 is a small file (size=241b).
02-21-2017 17:19:44.522 +0000 DEBUG PropertiesMapConfig - Performing pattern matching for: source::/etc/krb5.conf.2017-02-21-09:01:42|host::splunk-site3-001.local|krb5.conf-too_small|111
02-21-2017 17:19:44.522 +0000 DEBUG PropertiesMapConfig - Pattern 'krb5.conf-too_small' matches with priority 100
02-21-2017 17:19:44.522 +0000 DEBUG WatchedFile -   Attempting to load indexed extractions config from conf=source::/etc/krb5.conf.2017-02-21-09:01:42|host::splunk-site3-001.local|krb5.conf-too_small|111 ...
02-21-2017 17:19:44.522 +0000 DEBUG TailReader - About to read data (Opening file: /etc/krb5.conf.2017-02-21-09:01:42).
02-21-2017 17:19:44.522 +0000 DEBUG WatchedFile - seeking /etc/krb5.conf.2017-02-21-09:01:42 to off=241
02-21-2017 17:19:44.522 +0000 DEBUG WatchedFile - Reached EOF: /etc/krb5.conf.2017-02-21-09:01:42 (read 0 bytes)
02-21-2017 17:19:44.522 +0000 DEBUG TailReader - Finished reading file='/etc/krb5.conf.2017-02-21-09:01:42' in tailreader0 thread, disposition=ACKNOWLEDGE_CHANGE, deferredBy=0
02-21-2017 17:19:44.522 +0000 DEBUG TailReader - Returning disposition=ACKNOWLEDGE_CHANGE for file=/etc/krb5.conf.2017-02-21-09:01:42
02-21-2017 17:19:44.522 +0000 DEBUG FilesystemChangeWatcher - inotify doing infrequent backup polling for healthy path="/etc/krb5.conf.2017-02-21-09:01:42"
02-21-2017 17:19:44.540 +0000 DEBUG TcpOutputQ - Out of order ACK for id=253 looking for id=254 channel=source::/etc/krb5.conf|host::splunk-site3-001.local|config_file|83
02-21-2017 17:19:47.251 +0000 DEBUG TailingProcessor - Deferred notification for path='/etc/krb5.conf'.
02-21-2017 17:19:47.251 +0000 DEBUG TailReader - Enqueued file=/etc/krb5.conf in tailreader0
02-21-2017 17:19:54.395 +0000 DEBUG TcpOutputQ - Deleting in order ACK id=253 chanId=51 channel=source::/etc/krb5.conf|host::splunk-site3-001.local|config_file|83
02-21-2017 17:19:54.395 +0000 DEBUG TcpOutputQ - Deleting out of order ACK id=254 chanId=51 channel=source::/etc/krb5.conf|host::splunk-site3-001.local|config_file|83

Does anyone have any ideas on what could be the issue?

Thanks!!
Alex

0 Karma
1 Solution

TStrauch
Communicator

Hi,

@martin_mueller and me find a solution for our problem alekksi.

It seems to be a bug in the Splunk_TA_nix. We have a small workaround for the problem.

Go into the default/props.conf of the Splunk_TA_nix and remove "CHECK_METHOD = modtime" from the following two stanzas.

[source::(....(config|conf|cfg|inii|cfg|emacs|ini|license|lng|plist|presets|properties|props|vim|wsdl))]
sourcetype = config_file
CHECK_METHOD = modtime

[config_file]
LINE_BREAKER = ^((?!))$
TRUNCATE = 1000000
SHOULD_LINEMERGE = false
DATETIME_CONFIG = NONE
CHECK_METHOD = modtime
KV_MODE = none
pulldown_type = true
SEGMENTATION-all      = whitespace-only
SEGMENTATION-inner    = whitespace-only
SEGMENTATION-outer    = whitespace-only
SEGMENTATION-standard = whitespace-only
LEARN_MODEL = false
LEARN_SOURCETYPE = false

Restart your Splunk Servers and the crazy indexing should stop. We tested it a little bit and it seems to work very well.

Kind regards

View solution in original post

TStrauch
Communicator

Hi,

@martin_mueller and me find a solution for our problem alekksi.

It seems to be a bug in the Splunk_TA_nix. We have a small workaround for the problem.

Go into the default/props.conf of the Splunk_TA_nix and remove "CHECK_METHOD = modtime" from the following two stanzas.

[source::(....(config|conf|cfg|inii|cfg|emacs|ini|license|lng|plist|presets|properties|props|vim|wsdl))]
sourcetype = config_file
CHECK_METHOD = modtime

[config_file]
LINE_BREAKER = ^((?!))$
TRUNCATE = 1000000
SHOULD_LINEMERGE = false
DATETIME_CONFIG = NONE
CHECK_METHOD = modtime
KV_MODE = none
pulldown_type = true
SEGMENTATION-all      = whitespace-only
SEGMENTATION-inner    = whitespace-only
SEGMENTATION-outer    = whitespace-only
SEGMENTATION-standard = whitespace-only
LEARN_MODEL = false
LEARN_SOURCETYPE = false

Restart your Splunk Servers and the crazy indexing should stop. We tested it a little bit and it seems to work very well.

Kind regards

View solution in original post

martin_mueller
SplunkTrust
SplunkTrust

Note, do not edit the props.conf in default - make a props.conf in local and overwrite the setting there.

TStrauch
Communicator

Of course, Martin is right. Do it in local.

alekksi
Communicator

Thanks guys -- in production and up and running, confirmed working.

Just for other people's benefit, here are the config changes I put in local/props.conf for this app:

[source::(....(config|conf|cfg|inii|cfg|emacs|ini|license|lng|plist|presets|properties|props|vim|wsdl))]
CHECK_METHOD = endpoint_md5

[config_file]
CHECK_METHOD = endpoint_md5

alekksi
Communicator

Awesome, will update when I've had an opportunity to test and will accept the answer then.

Thank you very, very much!!! 😄

0 Karma

TStrauch
Communicator

Hi,

whats your outputs.conf configuration for the searchheads? Did you configure the following in outputs.conf? Your searchheads try to index data. Something they should not do.

To disable the indexing on your searchheads. Add the following in outputs.conf.
[indexAndForward]
index = false

Then configure the Searchheads to send the data to your indexers via outputs.conf

kind regards

0 Karma

alekksi
Communicator

Hi, I did have the following set:

[tcpout]
indexAndForward=false
[tcpout:prod_cluster_indexers]
server = splunk-indexer-1.local:9997,splunk-indexer-2.local:9997

I've added what you are suggesting, but I see no change in behaviour for this.

That said, I can confirm that the only index on the search heads is kvstore (summary has data but was moved away over a year ago and so just needs to be manually deleted)

 du -sh /opt/splunk/var/lib/splunk/* | grep M
7.0M    /opt/splunk/var/lib/splunk/audit
138M    /opt/splunk/var/lib/splunk/fishbucket
8.7M    /opt/splunk/var/lib/splunk/persistentstorage
269M    /opt/splunk/var/lib/splunk/summarydb

du -sh /opt/splunk/var/lib/splunk/* | grep G
1.9G    /opt/splunk/var/lib/splunk/kvstore
0 Karma

TStrauch
Communicator

I have read it again... Ok not a problem with indexing on your searchheads. The Splunk TA for *nix monitors the conf files in /etc if you have enabled the input.

Check if the input is enabled in inputs.conf of the Unix TA (Splunk_TA_nix). Just search the file for "/etc"

0 Karma

alekksi
Communicator

Well, I've turned it off on most of the search heads because it's just re-indexing the files unnecessarily. I've kept it enabled on one search head to enable me to investigate the issue. Where possible, I want to understand why this is happening in case this happens to other monitor stanzas.

Just to clarify, I have been reading files from /etc on my search heads since they've been put into a cluster. Only since one of the files was modified on site 3 (/etc/krb5.conf), it has started to re-index the files continuously. I now have hundreds of events with the same data in from each server.

To answer your question, this is my current config (extracted using btool) for the /etc monitor stanza on the search head where I have left this on.

[monitor:///etc]
_rcvbuf = 1572864
_whitelist = (\.conf|\.cfg|config$|\.ini|\.init|\.cf|\.cnf|shrc$|^ifcfg|\.profile|\.rc|\.rules|\.tab|tab$|\.login|policy$)
disabled = 0
host = splunk-site3-001.local
index = xxxx
0 Karma

lycollicott
Motivator

@alekksi The Splunk Add-on for Unix and Linux continuously re-read /etc on every server I deployed it on, so I eventually turned off that monitor, because it was using over 40GB of my license everyday. It sounds like a good idea to index /etc to track config changes, but the add-on does it quite poorly.

0 Karma

TStrauch
Communicator

Yes i agree, i tested it on some machines this morning. Everywhere the same behavior. Continously re-indexing the files.

I will try to find a solution for this, because i need this feature in near future.

0 Karma
Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.