Aight so, an admittal if you will. I have been ignoring the fact that all the timestamps are in Zulu time. Therefor, some of what I've claimed is incorrect. So taking the time conversion into account in the below response:   ./splunk list monitor   Shows all the files that should be monitored, are.   ./splunk list inputstatus   Shows all the files as before.. the small, less active ones are type = finished reading. The larger, problematic ones are type = open file. All are 100%. Re-running the Query, again for the past 24 hours I now have 4 empty hours (Z -> CDT; 2AM, 3AM, 4AM and 7AM), grepping the log file as before for a timestamp that's missing from SC. So far the first three missing hours: 2, 3, and 4AM are indeed empty from the log file. 7am however has events after searching randomly for, 07:10, 07:34, 07:45..many events.  So I have a question out to the networking team to verify there are events/alerts in the local log storage on a device from the group that sourced the missing data.   Still waiting for a response from networking. ... View more

Actually stumbled across that earlier this morning. :thumbsup: ... View more

(Sorry, Didnt mean to ignore that part of the question. Was also looking at a different Splunk issue with a co-worker)   Performing a query: index=<problem_index>  'Last 24 hours' which resulted in 4248841 Events There are two entire hours in the graph that are empty. The first prior to the first empty one contains 770,044 events. Then you have the empty hour, the next hour has 236,000, then another empty then the next 12 hours continue on more or less as usual averaging 200k. Finally the last three; 9, 10 and 11PM hours are all empty.  If I grep thru the log file this stanza is set to monitor, there are a lot of events with a timestamp of `06:xx:xx` (for example).   I tried querying the same index for the specific timestamp: 23-07-20T06:10:00Z No results, on a whim I changed the timeframe to 'Last 7 days', nothing.  So there are definitely events in the log file that are not making it into SplunkCloud.   ... View more

Hey, I was unaware of pipelines, I will try to look those up.  You mention the default limits.. the suggestion to set maxkbps to 0 is network through, yes? Or did I misinterpret that? ... View more

I was able to narrow down the primary input that's causing 42 out of the 47.646077M events this host if trying to process.  Rsyslog is setup to receive data on 12 custom ports. Rulesets send that data to individual log files using individual queues.  Then I have a custom 'app' with an inputs.conf monitoring each "rsyslog' file. The inputs file is pretty generic; index name, sourcetype, file/directory and that's it. To be honest I don't know enough to know what I should use here that'd help a high traffic server.  which is why I was looking for troubleshooting help.  ... View more

So just to confirm, I have no queues appearing in the search query you suggested. But I should still apply the limits.conf change that appears to remove all throughput limits on the host in question, right? To be fair and since you know more than I do I will give it a shot. But.. this completely ignores the idea of monitoring/troubleshooting to determine an actual cause. In essence you are suggesting to release the Kraken on the network when I cannot confirm that it's the network throughput limitations issue.   ... View more

Oh I don't doubt the query, except in my case it returned no matching data on the host in question. Actually querying the past 4 hours it found 37M events for one of our UF hosts in Japan. Still though, nothing for the one I'm working on. But aside from that, a query is not a 'how' to configure an inputs.conf file's ingest rates. That's what I'm trying to discover. ... View more

ingestion queues? Yes that's what Im trtying to find information on.. the 'how-to' is escaping me so far. Splunks Troubleshooting Universal Forwarder page mentions just that.  But they offer no links to further info and the search so far turns up nada. ... View more

Thanks @gcusello  Yes I'm experiencing delays in SplunkClouds indexing.. I did find a mention of possibly shortening the ingestion interval on the UF but I have not yet found information on how to do this. One of my forwarders in particular has a pretty high usage on receipt of log data. Rsyslog on this host is processing upwards of 50M events in a 24hr period. The CPU util is below 5%, memory below 20% and nic usage around 45KBps (average) with 0 transmit errors.  Also found a document explaining the parts of metrics.log, to be fair I'm not familiar enough with this yet to identify potential problems.  ... View more

This post led me to the /etc/systemd/system/SplunkForwarder.service file. Except now, on version 9.1.0.1 , this line: ExecStartPost=/bin/bash -c "chown -R <userid>:<groupid> /sys/fs/cgroup/system.slice/%n" already exists. Since all objects under /sys/fs/cgroup is owned by root:root, I added the user `splunkfwd` to the `adm` group then rebooted. Then I can use systemd to start/stop splunkforwarder. ... View more

Thank you Sanjay,  so yet another known issue huh? Upgrading/installing version 9 has a few it seems. So, being as though I'm sure this is low priority is there any ETA on it at all? I have automation that handles the installing of the UF. Now when checking the returns/results of a service restart I need to make sure to include bits to ensure the 'Warning' generated doesn't cause me problems. ... View more

Getting this error when either: installing fresh 9.1.0.1 or upgrading 8.x to 9.1.  This is just sad.. I mean how could Splunk have NOT fixed this in over a year?? Obviously the syntax changed.. can't be that hard to figure out why. ... View more

So a known issue and will not cause problems, that's great.  Any idea on a cause? Every time I call the splunk binary whether is starting/restarting the app or invoking btool. ... View more