Hi All, We have been experiencing intermittent indexing delays on our Splunk environment, which consists of three standalone (non-clustered) indexers. On certain days, one or more indexers begin queuing for extended durations (7–8 hours, sometimes up to 12–14 hours). During these periods: Indexing throughput degrades significantly Event ingestion is delayed (in some cases by more than 1 hour) Downstream processes such as summary indexing jobs receive incomplete data Upstream reporting jobs show incorrect or partial counts Observations CPU and Memory utilization on all indexers remain within normal limits Each indexer has 24 CPU cores and 256 GB RAM No obvious resource saturation observed at the OS level (CPU/Memory) However, during these queueing periods: Disk I/O latency spikes significantly FlashArray storage reports high read/write activity and elevated latency From preliminary analysis, we suspect: Sudden ingestion bursts from specific hosts or Kubernetes pods In some cases, we observed extreme spikes (e.g., ~24 million events in a single minute from a single source) We believe that short-lived but intense ingestion bursts are overwhelming the storage subsystem, causing: Increased disk I/O wait on indexers Backpressure in the indexing pipeline Prolonged queue buildup and delayed ingestion We are trying to identify the exact sources responsible for these bursts through Splunk queries.   Identify bursty data sources (hosts / Kubernetes pods) Determine which hosts or Kubernetes pods generate sudden spikes in event volume over short time intervals (e.g., 1-minute or 5-minute windows), relative to their typical baseline activity. Analyze parsing overhead on indexers Identify which indexers are spending the most time processing (parsing) incoming data, and determine the sourcetypes contributing most to this parsing load.     ... View more

Environment Splunk Enterprise (single-instance: indexing + monitoring on same host) OS: Linux Log directory mounted via SMB/CIFS (smb2) Input type: monitor:// Files are created by an external application on the CIFS share Problem Description Splunk does not automatically detect or ingest newly created log files under a CIFS-mounted directory when using a monitor:// input. Newly created files are only picked up after a manual Splunk restart. Existing files are indexed correctly at startup. Observed Behavior New log files are created in the monitored directory and are visible via ls Files have correct permissions and ownership Splunk does not ingest or log any monitor activity for new files while running After restarting Splunk, all previously missed files are detected and indexed Expected Behavior Splunk should detect and ingest newly created files in the monitored directory without requiring a restart. Key Finding The monitored path is mounted via CIFS/SMB, and it appears that filesystem change notifications (inotify) are not being triggered for new files created on the share. As a result, Splunk is not notified of new files while running. Troubleshooting Performed Verified monitor stanza is loaded using splunk btool inputs list --debug Verified file permissions and ownership (readable by Splunk user) Tested with simple locally created files (echo > test.log) Confirmed files appear in directory but are not indexed Verified whitelist/blacklist regex does not exclude files Adjusted CRC-related settings (crcSalt, initCrcLength) Confirmed that polling_interval is not a valid or supported setting Observed that Splunk only detects files during startup full directory scan Can someone help to answer below so that it helps to understand if monitoring of files is possible for logs configured over mount. Confirm whether monitor:// inputs are supported and expected to work reliably on CIFS/SMB-mounted directories Clarify any documented or undocumented limitations regarding inotify on network-mounted filesystems Advise on supported configurations or recommended alternatives (e.g., NFS, local landing directory, or forwarder-based architecture)   ... View more

We are seeing a large discrepancy in field extraction counts between our Prod and Dev environments for sourcetype=xxx. In Prod, a search returns ~600+ fields. In Dev, the same search returns ~100 fields. We confirmed that KV_MODE=auto is set on both environments, but Dev still does not extract as many fields. Raw events in Dev do contain certain keys (e.g., PreStore), but these fields do not consistently appear in the sidebar unless explicitly searched. Prod has ~58 field extractions defined for this sourcetype, while Dev only has ~6. A large number of the extractions in Prod appear as Private in the UI. We are unclear whether these “Private” extractions are also being applied to other users, or only to the owners. Questions How do “Private” field extractions behave — are they ever applied to users other than the owner, or should they only affect the owner’s searches? Could differences in data verbosity (more key=value pairs in Prod logs) be compounding the discrepancy, even with the same KV_MODE setting? What is the best way to identify all active field extractions (including private/app-scoped) that are being applied to a sourcetype, so we can reconcile between environments? How can we ensure consistent field discovery behavior between Dev and Prod?   Steps taken so far   Checked props.conf and transforms.conf on the search app in both environments — only a few extractions found in Dev vs many in Prod. Verified KV_MODE settings using REST and btool. Confirmed Prod SH shows auto, Dev was updated to auto, but discrepancy remains. Compared number of field extractions and in PROD it is 58 for sourcetype and in Dev it is 6.   ... View more

Hi Folks, We have thousands of universal forwarders that are currently running on old version (7.0.2). We are planning to upgrade universal forwarders to most recent version but before we do that we would like to reduce the overall footprint of universal forwarders by uninstalling them from the servers that are no longer sending logs.  Logs for few applications and infrastructure are migrated to Azure so they are no longer sending it to splunk. Need to find a list of such servers so i can uninstall them before i do mass upgrade. Is there a query that can give me the list of hostname along with timestamp of last log that it sent. Thanks in advance ... View more

Hi Folks, We have a complaint from stakeholders that they are seeing duplicate events in Splunk. they shared few examples where same events were indexed hundreds of times, sometimes in thousands. I can confirm that there are no duplicate stanzas to monitor log files in inputs.conf. I checked the actual log files and could see some events were duplicated in source file itself. count of it was 12 however in splunk same event was indexed 524 times. We see there are lot of inconsistencies in how the logs are being written at source. we see timestamps are either missing or partial. could that be the reason splunk is kind of going in bad state and re reading the log files?  This is how my inputs.conf is configured.  [monitor://\\hostname\log] recursive = false disabled = false followTail = 0 host = host_regex = ^.*\d-(.+?)-\d+-\d{8}_?\d*\. ignoreOlderThan = 8d index = index sourcetype = sourcetype whitelist = \.log|\.out time_before_close = 1 initCrcLength = 4096 ... View more

Hi Folks, We have a Splunk instance which comprises of 1 SH, 2 Indexers in cluster (Replication enabled), 1 Cluster Master and 1 Heavy forwarder.   We need to migrate these servers to a new set of servers. I would like to know the steps involved in migrating the already indexed clustered data. I am little newbie in Splunk Administration so would appreciate if someone can help me with detailed steps and points that i need to take care of while migrating the data. Thanks in advance. ... View more