Finally got around to reading about SC4S and I'm fairly certain this is the route forward. I got the ok to replace the pair of UF's with a Heavy if needed. But I have a good feeling that the SC4S will be able to handle things very well. ... View more

No not confusing the two.. I'm well aware of the differences. My scenario is that I have about a hundred devices all sending syslog data between two receivers in two sites currently. Which is then picked up by two UF's and then forwarded to Splunk Cloud. When those two sites get rolled up together into a single colo I'll need to combine them (for lack of better words). Hence the load balancing.  HA would be fine, if I could determine if both UF and rsyslog could be operated in a high availability setting. Pretty sure the former is possible.. the latter though not from what I understand. ... View more

So Im understanding here that yeah, syslog itself doesn't lend well to being balanced. Splunk Universal forwarder should not be used to receive Syslog traffic, but a Heavy forwarder "can". And if I wanted to load balance a pair of Splunk UF's then I should setup a real load balancer.. And if I want to LB syslog,  should also setup a real LB and have the syslog and splunk functions on dedicated hosts.  this look accurate? ... View more

Yup, And these Splunk instances aren't listening on tcp/udp ports so that's good. ... View more

Ive read that too and it was kinda ambiguous to me. The same linux server is running an instance of Splunk UF as well as rsyslog. The latter is writing to log files that are listed in the formers inputs.conf. If this is the exact scenario that Splunk says is not good to, they should word it a bit better. The syslog daemon would have to write to logs files across the network.. or the UF would have to reach out to read the log files remotely. Either of those ideas seemed like a bad thing. In any case.. with our impending colo changes I'm going to have to refactor these hosts. So if separating the services is the best path forward that's what we'll do.   But I will still need to LB a pair of forwarders.. unless someone knows how well the UF and/or HF scale hardware-wise?  Like if I throw 8 cores, a fistfull of memory and 25gigs of network throughput, can a single Splunk host process enough data to keep up with say 100k events per day? ... View more

I knew HF's could handle custom ports but our team has been limited to UFs for over a year now. I'd be tickled pink to ditch rsyslog entirely...  I should have perhaps asked a different question. Replacing syslog with a Heavy Forwarder. ... View more

Hey @scelikok  Thanks for the pointer.. I realize my question might fall into a grey area, I hope others who may have insight or experience still chime in. Learning things about load balancing just the UF will also be super helpful! ... View more

Aight so, an admittal if you will. I have been ignoring the fact that all the timestamps are in Zulu time. Therefor, some of what I've claimed is incorrect. So taking the time conversion into account in the below response:   ./splunk list monitor   Shows all the files that should be monitored, are.   ./splunk list inputstatus   Shows all the files as before.. the small, less active ones are type = finished reading. The larger, problematic ones are type = open file. All are 100%. Re-running the Query, again for the past 24 hours I now have 4 empty hours (Z -> CDT; 2AM, 3AM, 4AM and 7AM), grepping the log file as before for a timestamp that's missing from SC. So far the first three missing hours: 2, 3, and 4AM are indeed empty from the log file. 7am however has events after searching randomly for, 07:10, 07:34, 07:45..many events.  So I have a question out to the networking team to verify there are events/alerts in the local log storage on a device from the group that sourced the missing data.   Still waiting for a response from networking. ... View more

Actually stumbled across that earlier this morning. :thumbsup: ... View more

(Sorry, Didnt mean to ignore that part of the question. Was also looking at a different Splunk issue with a co-worker)   Performing a query: index=<problem_index>  'Last 24 hours' which resulted in 4248841 Events There are two entire hours in the graph that are empty. The first prior to the first empty one contains 770,044 events. Then you have the empty hour, the next hour has 236,000, then another empty then the next 12 hours continue on more or less as usual averaging 200k. Finally the last three; 9, 10 and 11PM hours are all empty.  If I grep thru the log file this stanza is set to monitor, there are a lot of events with a timestamp of `06:xx:xx` (for example).   I tried querying the same index for the specific timestamp: 23-07-20T06:10:00Z No results, on a whim I changed the timeframe to 'Last 7 days', nothing.  So there are definitely events in the log file that are not making it into SplunkCloud.   ... View more

Hey, I was unaware of pipelines, I will try to look those up.  You mention the default limits.. the suggestion to set maxkbps to 0 is network through, yes? Or did I misinterpret that? ... View more

I was able to narrow down the primary input that's causing 42 out of the 47.646077M events this host if trying to process.  Rsyslog is setup to receive data on 12 custom ports. Rulesets send that data to individual log files using individual queues.  Then I have a custom 'app' with an inputs.conf monitoring each "rsyslog' file. The inputs file is pretty generic; index name, sourcetype, file/directory and that's it. To be honest I don't know enough to know what I should use here that'd help a high traffic server.  which is why I was looking for troubleshooting help.  ... View more

So just to confirm, I have no queues appearing in the search query you suggested. But I should still apply the limits.conf change that appears to remove all throughput limits on the host in question, right? To be fair and since you know more than I do I will give it a shot. But.. this completely ignores the idea of monitoring/troubleshooting to determine an actual cause. In essence you are suggesting to release the Kraken on the network when I cannot confirm that it's the network throughput limitations issue.   ... View more

Oh I don't doubt the query, except in my case it returned no matching data on the host in question. Actually querying the past 4 hours it found 37M events for one of our UF hosts in Japan. Still though, nothing for the one I'm working on. But aside from that, a query is not a 'how' to configure an inputs.conf file's ingest rates. That's what I'm trying to discover. ... View more

ingestion queues? Yes that's what Im trtying to find information on.. the 'how-to' is escaping me so far. Splunks Troubleshooting Universal Forwarder page mentions just that.  But they offer no links to further info and the search so far turns up nada. ... View more

Thanks @gcusello  Yes I'm experiencing delays in SplunkClouds indexing.. I did find a mention of possibly shortening the ingestion interval on the UF but I have not yet found information on how to do this. One of my forwarders in particular has a pretty high usage on receipt of log data. Rsyslog on this host is processing upwards of 50M events in a 24hr period. The CPU util is below 5%, memory below 20% and nic usage around 45KBps (average) with 0 transmit errors.  Also found a document explaining the parts of metrics.log, to be fair I'm not familiar enough with this yet to identify potential problems.  ... View more

This post led me to the /etc/systemd/system/SplunkForwarder.service file. Except now, on version 9.1.0.1 , this line: ExecStartPost=/bin/bash -c "chown -R <userid>:<groupid> /sys/fs/cgroup/system.slice/%n" already exists. Since all objects under /sys/fs/cgroup is owned by root:root, I added the user `splunkfwd` to the `adm` group then rebooted. Then I can use systemd to start/stop splunkforwarder. ... View more

Thank you Sanjay,  so yet another known issue huh? Upgrading/installing version 9 has a few it seems. So, being as though I'm sure this is low priority is there any ETA on it at all? I have automation that handles the installing of the UF. Now when checking the returns/results of a service restart I need to make sure to include bits to ensure the 'Warning' generated doesn't cause me problems. ... View more

Getting this error when either: installing fresh 9.1.0.1 or upgrading 8.x to 9.1.  This is just sad.. I mean how could Splunk have NOT fixed this in over a year?? Obviously the syntax changed.. can't be that hard to figure out why. ... View more

So a known issue and will not cause problems, that's great.  Any idea on a cause? Every time I call the splunk binary whether is starting/restarting the app or invoking btool. ... View more

Should I be able to see references to those two bracketed 'variables' in transforms.conf in splunkd.log after restarting the service? Or maybe they are endpoint arguments? If so, I do not see them.  ... View more

@yeahnah  So both Props and Transforms are required to effect the change you describe right? I created both files under $SPLUNKHOME/etc/system/local and bounced the service, Im now waiting for SC to update. By search head I assume you mean either the Heavy Forwarder, or teh Universal? In this case, it's the Universal as I do not have that sort of access to Splunk Cloud. ... View more