Facepalm moment... The initial solution actually worked props.conf EVAL-severity = lower(severity) Only problem was, there was a # comment made inline which I did not include. Removing this and only using the correct eval works like a charm. Thank you @gcusello! ... View more

Hello again @gcusello  I may be a bit low on caffein so maybe I'm not seeing this correct. The suggested solution in my props.conf file is the same solution I initially tried, which did not work. I tried moving the EVAL to my transforms.conf with no luck. The SPL search <base search> | eval severity = lower(severity) | stats count by severity Works just fine, I get lower case values just as expected. But I cannot get this to work in props.conf and/or transforms.conf. Asking the all-knowing AI it tries to tell me that EVAL lower in props.conf only works on indexed fields. So am I doing or reading something incorrectly, or is there some other issue I am missing?   ... View more

Hello @gcusello  No, I am unsure where in all the evals the need arises. I was hoping for a "quick workaround" 🙂 Just to make sure, should this be in the transforms.conf file or should I go back to only the props.conf eval? ... View more

It seems like if your deployment server is moonlighting as something else, the agent management page becomes unavailable. Found a rouge server.conf file setting mode=search and enabling distributed search though search peers. Removing this file and restarting the service, the agent management page loaded and is fully functional. So this solved the issue, though I suppose this is only an option if you have a dedicated deployment server and can disable distributed search functionality. ... View more

Thank you @PickleRick, so much to unpack.... I'm not reading that I'm wrong regarding the field extraction, so where do you submit an issue to correct this? Even if it is a single field it should at least work as intended. I also noticed that both the auditd and linux_audit sourcetypes of the TA_nix app are also in a section following this: # Stanzas in this section are legacy configuration stanzas. So I'm guessing that there is no "current" way to collect audit log. Neither scripted through ausearch or by reading the logfile. The solution therefore seems straightforward, I need another app to deal with audit logs. I cannot use any scripted solution relying on ausearch but must read a local audit log file being dumped. To make the suggested app https://splunkbase.splunk.com/app/4232 (which does look good , thank you) "backwards compatible" I'd need to perform "several minor (unsupported) changes". Or I can just "live with it" as it is not working correctly at the moment anyway and switch over to another sourcetype. This manual entry https://docs.splunk.com/Documentation/AddOns/released/Linux/Configure4 points to the same sourcetype as the documentation for the AuditD TA, "linux:audit". But also indicates the use of yet another app https://splunkbase.splunk.com/app/3412 which I will be unable to use as it relies on changes to auditd on the host and making use of HEC traffic rather than a filewatch. While I am leaning towards the "minor but unsupported changes", what would be the recommended path forward from someone with a deeper understanding of the issue? a) Performing the "minor but unsupported changes" to the app, i.e. including the deprecated/legacy linux_audit sourcetype and installing the recommended TA in the SH cluster (as well as HF and IX as per the documentation)? b) Switch the defined sourcetype (linux_audit) under the audit filewatch stanza in the inputs.conf of the Splunk_TA_nix app deployed to all universal forwarders over to linux:audit and installing the recommended TA in the SH cluster (as well as HF and IX as per the documentation)? c) Building a new or heavily modifying an existing or merging several TAs to deal with audit logs in a manner closer to what is expected? I apologize if any cynicism or irritation is not fully tucked away and that this has become a bit of a snowball. I do appreciate all the help, feedback and suggestions. ... View more

I'll mark this down as the solution and figure out how to push the modified settings from a manager. ... View more

Ah, yes. Crible. Looked at their product earlier for other issues. Seems like a good product and solution but as you point out, another product to manage. Supposedly a logstash in front of our heavy forwarders could also do the trick. But that also includes another product and I assume that log shipping would then also become HEC traffic and in my (very limited but still) experience also mean having to deal with a new log format and sourcetype for a standard Linux audit log. ... View more

That's a shame, but reasonable I suppose. This feature should be resource heavy as it would need to keep a bunch of data in memory running "searches" with a high frequency, but it would have been a nice fix for my problem. 'pruned' is an understatement. Not to give it all away (in case someone happens to google their way here) but I've repeatedly pointed out that "you do not need to check if your running on a virtual machine/VPS once every hour, every day, forever". Especially if you are the service provider and know that you will never ever ever ever run on "Alibaba Cloud" and most likely never use QEMU. But that's a problem related to "drop-in solutions" for metric collection which no one has any interest of optimizing and over which I have no control. Hence, insanely detailed grep solutions and allowing some completely pointles audit log to trickle in is just the way it has to be for now. Thank you for your feedback, much appreciated.   ... View more

Yes, "should" seems correct 😊 However, I did change these levels and there was no effect. This led me to the documentation pointing at local changes on the indexers. The warnings are presented for indexers under the "Health of Distributed Splunk Deployment" pointing at indexers, though I am not sure where warnings are generated, to where they are "collected" before being presented in the search head cluster. I also cannot figure out where in the healtch.conf file these levels could/should be modified (health.conf | Splunk Docs) so I'm guessing it is likely somewhere else. ... View more

Thank you So the acceptable solution to these issues is to adjust thresholds to not trigger under "normal operation". The follow-up regarding thresholds settings, from what I understand these alerts are generated locally on the indexers in the indexer cluster. The health.conf settings are apparently not synced in an indexer cluster, only in the search head cluster where any changes has no effect (already tried). If thresholds are to be modified in the indexer cluster, what file and values are of interest to push from the manager to change relevant thresholds? I have not been able to identify these in the documentation. If not in the indexer cluster, then where? ... View more

I removed all expired ingest based licenses and restarted the license manager, this removed all current warnings/alerts. So far I don't see any new warnings so (fingers crossed) everything is fine. Feels good to hear that there were no obvious mistakes during install either. Yes, I can monitor resource usage from the monitoring console, hopefully I can get the "conversion equation" which Splunk uses to translate load towards our allowed allocation. Would be nice to be able to check before onboarding new sources. Thank you for the feedback, much appreciated. ... View more

I am tempted to switch this post to the "solution", it is not "exactly" what I asked for but it did achiecve the effect I was looking for. As per Use the deployer to distribute apps and configuration updates - Splunk Documentation. After an initial push of PSC and MLTK from the deployment server we added a local/app.conf file containing only  [shclustering] deployer_push_mode = local_only The "push-time" dropped from 167 seconds to 47 seconds with push mode set in the PSC app, then to 24 seconds when also changing push mode in the MLTK app. So this did have the effect I was after, even though it is not strictly speaking a "local only install" of an app. All the best ... View more

Did not know that you could do this on an "app level", but it might be worth looking into different push modes! Thanx ... View more

"Email recipients groups" | Ideas ... View more

https://ideas.splunk.com/ideas/APPSID-I-989 ... View more

Well crud... The PSC os something like 2.3 GB, so even if the entire app is not redistributed every push I suspect it severly affects the check and push process. No, it is a fresh and first time install so no inherited problems. As far as I can see, there are no errors or problems associated with the install itself and MLTK is running smoothly. So this is another "improvement" to wish for then, something like a ".gitignore" option where you can manually add apps to your cluster without having them "managed" by the SHD. My guess is that the only way around this would be a standalone SH where you'd basically only install PSC and MLTK and only use it for these purposes. In any case, thank you for your feedback and have a nice weekend ... View more

I have similar issues poppping up as of late. But how does one isolate the affected forwarder? The error message reads Forwarder Ingestion Latency   Root Cause(s): Indicator 'ingestion_latency_gap_multiplier' exceeded configured value. The observed value is 89. Message from <UUID>:<ip-addrs>:54246 Unhealthy Instances: indexer1 indexer2   The "message from" section just lists the UUID, an IP adress and a port. Which part here would help me find the actual forwarder? The UUID does not match any "Client name" under forwarder management on the deployment server. The IP adress does not match a server on which I have a forwarder installed. One or a few of the indexers are listed as "unhealthy instances" each time. But the actual error sounds like it lives in the forwarder end and not on the indexer. With the available information in this warning/error. How can I figure out which forwarder is either experiencing latency issues OR need to have that log file mentioned flushed. ... View more

So, how does one isolate the affected forwarder? The error message reads Forwarder Ingestion Latency   Root Cause(s): Indicator 'ingestion_latency_gap_multiplier' exceeded configured value. The observed value is 89. Message from <UUID>:<ip-addrs>:54246 Unhealthy Instances: indexer1 indexer2   The "message from" section just lists the UUID, an IP adress and a port. Which part here would help me find the actual forwarder? The UUID does not match any "Client name" under forwarder management on the deployment server. The IP adress does not match a server on which I have a forwarder installed. One or a few of the indexers are listed as "unhealthy instances" each time. But the actual error sounds like it lives in the forwarder end and not on the indexer. With the available information in this warning/error. How can I figure out which forwarder is either experiencing latency issues OR need to have that log file mentioned flushed.   ... View more

Sure can 🙂 So, PickleRick and you basically made the same argument which helped a lot. I'm not going to duplicate the whole reply, hope that OK. I can still split the same "disk" into warm and cold defining different amounts of storage, or just dump everything in the same volume and let Splunk figure it out. Thanks for the feedback ... View more