About fatsug

fatsug · ‎07-16-2025

Thank you @PickleRick, so much to unpack.... I'm not reading that I'm wrong regarding the field extraction, so where do you submit an issue to correct this? Even if it is a single field it should at least work as intended. I also noticed that both the auditd and linux_audit sourcetypes of the TA_nix app are also in a section following this: # Stanzas in this section are legacy configuration stanzas. So I'm guessing that there is no "current" way to collect audit log. Neither scripted through ausearch or by reading the logfile. The solution therefore seems straightforward, I need another app to deal with audit logs. I cannot use any scripted solution relying on ausearch but must read a local audit log file being dumped. To make the suggested app https://splunkbase.splunk.com/app/4232 (which does look good , thank you) "backwards compatible" I'd need to perform "several minor (unsupported) changes". Or I can just "live with it" as it is not working correctly at the moment anyway and switch over to another sourcetype. This manual entry https://docs.splunk.com/Documentation/AddOns/released/Linux/Configure4 points to the same sourcetype as the documentation for the AuditD TA, "linux:audit". But also indicates the use of yet another app https://splunkbase.splunk.com/app/3412 which I will be unable to use as it relies on changes to auditd on the host and making use of HEC traffic rather than a filewatch. While I am leaning towards the "minor but unsupported changes", what would be the recommended path forward from someone with a deeper understanding of the issue? a) Performing the "minor but unsupported changes" to the app, i.e. including the deprecated/legacy linux_audit sourcetype and installing the recommended TA in the SH cluster (as well as HF and IX as per the documentation)? b) Switch the defined sourcetype (linux_audit) under the audit filewatch stanza in the inputs.conf of the Splunk_TA_nix app deployed to all universal forwarders over to linux:audit and installing the recommended TA in the SH cluster (as well as HF and IX as per the documentation)? c) Building a new or heavily modifying an existing or merging several TAs to deal with audit logs in a manner closer to what is expected? I apologize if any cynicism or irritation is not fully tucked away and that this has become a bit of a snowball. I do appreciate all the help, feedback and suggestions.

fatsug · ‎07-15-2025

Hi community I've been pulling my hair for quite some time regarding field extraction using the Splunk_TA_nix app. One thing that has been annoying me is the absence of a field which contains the full command executed. My question/comment which I seek to get some feedback on. While trying to figure out why I am not seeing the expected/desired content I noticed something. Splunk_TA_nix/default/props.conf [linux_audit] REPORT-command = command_for_linux_audit Splunk_TA_nix/default/transforms.conf [command_for_linux_audit] REGEX = exe=.*\/(\S+)\" FORMAT = command::$1 This regex only applies to the "type=SYSCALL" audit log entry which is the only one containing "exe=" and it does not work in our environment. There is no trailing quotation mark in our log so this field is not properly extracted with this regex. So to work as intended this would need to be changed to [command_for_linux_audit] REGEX = exe=.*\/(\S+) FORMAT = command::$1 This would generate a field called "command" with the executed command (binary) only. Is this just in our environment where we have a make-shift solution to generate a second audit log file for collection, or is this a general issue? And the rant: It seems that if not defined elsewhere the default field separator is space. This means that most <field>=<value> entries in the audit log are extracted . The sourcetype=linux_audit type= PROCTITLE events actually has a field called "proctitle" which contains the full command executed. While a field called "proctitle" is extracted the value of this field is cut short after the first space, meaning only the command (binary) is available. Assuming this is expected behaviour, I suppose that I have to define a field extraction overwriding the "default" behaviour to get a field "proctitle" with the desired content.

fatsug · ‎07-11-2025

I'll mark this down as the solution and figure out how to push the modified settings from a manager.

fatsug · ‎07-09-2025

Ah, yes. Crible. Looked at their product earlier for other issues. Seems like a good product and solution but as you point out, another product to manage. Supposedly a logstash in front of our heavy forwarders could also do the trick. But that also includes another product and I assume that log shipping would then also become HEC traffic and in my (very limited but still) experience also mean having to deal with a new log format and sourcetype for a standard Linux audit log.

fatsug · ‎07-08-2025

That's a shame, but reasonable I suppose. This feature should be resource heavy as it would need to keep a bunch of data in memory running "searches" with a high frequency, but it would have been a nice fix for my problem. 'pruned' is an understatement. Not to give it all away (in case someone happens to google their way here) but I've repeatedly pointed out that "you do not need to check if your running on a virtual machine/VPS once every hour, every day, forever". Especially if you are the service provider and know that you will never ever ever ever run on "Alibaba Cloud" and most likely never use QEMU. But that's a problem related to "drop-in solutions" for metric collection which no one has any interest of optimizing and over which I have no control. Hence, insanely detailed grep solutions and allowing some completely pointles audit log to trickle in is just the way it has to be for now. Thank you for your feedback, much appreciated.

fatsug · ‎07-08-2025

Yes, "should" seems correct 😊 However, I did change these levels and there was no effect. This led me to the documentation pointing at local changes on the indexers. The warnings are presented for indexers under the "Health of Distributed Splunk Deployment" pointing at indexers, though I am not sure where warnings are generated, to where they are "collected" before being presented in the search head cluster. I also cannot figure out where in the healtch.conf file these levels could/should be modified (health.conf | Splunk Docs) so I'm guessing it is likely somewhere else.

fatsug · ‎07-08-2025

When collecting Linux logs using a Universal Forwarder we are collecting a lot of unnecessary audit log from cronjobs collecting server status and other information. In particular the I/O from a little script. To build a grep based logic on a Heavy Forwarder there would have to be a long list of very particular "grep" strings not to loose ALL grep attempts. In a similar manner, commands like 'uname' and 'id' are even harder to filter out. The logic needed o reliably filter out only I/O generated by the script would be to find events with comm="script-name", get the pid value from that initial event and drop all events for the next say 10 seconds with a ppid that matches the pid. To make things complicated there is no control over the log/files on the endpoints, only what the universal forwarder is able to do then the heavy forwarder before the log in indexed. Is there any way to accomplish this kind of adaptive filtering/stateful cross-event logic in transit and under these conditions? Is this something that may be possible using the new and shiny Splunk Edge Processor once it is generally available?

fatsug · ‎07-08-2025

Thank you So the acceptable solution to these issues is to adjust thresholds to not trigger under "normal operation". The follow-up regarding thresholds settings, from what I understand these alerts are generated locally on the indexers in the indexer cluster. The health.conf settings are apparently not synced in an indexer cluster, only in the search head cluster where any changes has no effect (already tried). If thresholds are to be modified in the indexer cluster, what file and values are of interest to push from the manager to change relevant thresholds? I have not been able to identify these in the documentation. If not in the indexer cluster, then where?

fatsug · ‎07-08-2025

Hello community, I have a question which has been floating around here for quite some time and though I've seen quite a few conversations and tips, I have not found a "single definitive source of truth". At some point, some time ago, when bumping Splunk from v8 to v9 we started noting Iowait alerts from the health monitor. I've checked our resource usage on our indexers (which are generating the alerts) and the cause of the alert seem to be spikes in resource usage. 3 out of x indexers have spikes in resource usage within 10 minutes which triggers an alert. Most of the time these alert seem wound really tight and the alerts somewhat overblown, on the other hand they should be there for a reason and I am not sure of tuning the alert levels is the right way to go. I have gone through the following threads: https://community.splunk.com/t5/Monitoring-Splunk/Why-is-IOWait-red-after-upgrade/m-p/600262#M8968 https://community.splunk.com/t5/Deployment-Architecture/IOWAIT-alert/m-p/666536#M27634 https://community.splunk.com/t5/Splunk-Enterprise/Why-am-I-receiving-this-error-message-IOWait-Resource-usage/m-p/578077#M10932 https://community.splunk.com/t5/Splunk-Search/Configure-a-Vsphere-VM-for-Splunk/td-p/409840 https://community.splunk.com/t5/Monitoring-Splunk/Running-Splunk-on-a-VM-CPU-contention/m-p/107582 Some recommendations exist to either ignore or adjust thresholds. Continuously ignoring seems like a slippery slope to desensitization and continuously monitoring add to the risk of alert fatigue. Other recommends ensuring adequate resources to solve the core issue, which seems logical though I am unsure regarding how. I am left with two questions 1) What are concrete actions could be taken to minimize the chance of these alerts/issues in a deployment based on VMWare Linux servers. In other words, what can/should I forward to the server group that they can work with, check and confirm in order to minimize the chance of these alerts? 2) What recommendations if any exists regarding modifying default thresholds? I could set thresholds high enough to not alert on "normal activity", is this the recommended adjustment or are there any concrete recommended modifications?

fatsug · ‎07-04-2025

I removed all expired ingest based licenses and restarted the license manager, this removed all current warnings/alerts. So far I don't see any new warnings so (fingers crossed) everything is fine. Feels good to hear that there were no obvious mistakes during install either. Yes, I can monitor resource usage from the monitoring console, hopefully I can get the "conversion equation" which Splunk uses to translate load towards our allowed allocation. Would be nice to be able to check before onboarding new sources. Thank you for the feedback, much appreciated.

fatsug · ‎07-03-2025

We recently switched over from an ingest based license to a resource based (vCPU) license model in our deployment. The license was successfully installed in the (dedicated) license manager however after the old license expired I noticed a bunch of warnings that the allowed volume had been exceeded. Our manually specified pools have not exceeded their allocation. Though when checking the "Usage report" the available total pool license is now the "free" 500 MB/day. This is not very surprising as we no longer have a "max per day". But should the available not be "infinite" now rather then drop down to default? I deleted all expired licenses and restarted the license manager and the warnings seem to have disappeared, at least for now. But the "total available license" still pushes a varning at 500 MB and up with the gauge screaming red in the "usage report" My first question, how can I modify the "total available license" from the ingest based GB per day to "infinite" or any other higher number than 500 MB per day? Did I miss some step when switching from ingest based license to resource based license in the configuration of the license manager? My second related question, how can I now monitor available license? There is no resource based license usage report available on the license manager? All the best

fatsug · ‎05-07-2025

I am tempted to switch this post to the "solution", it is not "exactly" what I asked for but it did achiecve the effect I was looking for. As per Use the deployer to distribute apps and configuration updates - Splunk Documentation. After an initial push of PSC and MLTK from the deployment server we added a local/app.conf file containing only [shclustering] deployer_push_mode = local_only The "push-time" dropped from 167 seconds to 47 seconds with push mode set in the PSC app, then to 24 seconds when also changing push mode in the MLTK app. So this did have the effect I was after, even though it is not strictly speaking a "local only install" of an app. All the best

fatsug · ‎05-07-2025

Did not know that you could do this on an "app level", but it might be worth looking into different push modes! Thanx

fatsug · ‎05-02-2025

"Email recipients groups" | Ideas

fatsug · ‎05-02-2025

https://ideas.splunk.com/ideas/APPSID-I-989

fatsug · ‎05-02-2025

Well crud... The PSC os something like 2.3 GB, so even if the entire app is not redistributed every push I suspect it severly affects the check and push process. No, it is a fresh and first time install so no inherited problems. As far as I can see, there are no errors or problems associated with the install itself and MLTK is running smoothly. So this is another "improvement" to wish for then, something like a ".gitignore" option where you can manually add apps to your cluster without having them "managed" by the SHD. My guess is that the only way around this would be a standalone SH where you'd basically only install PSC and MLTK and only use it for these purposes. In any case, thank you for your feedback and have a nice weekend

fatsug · ‎05-02-2025

This may be a "dumb" question, but I'll just throw it out there while I try to work it out. The Python for Scientific Computing (PSC) app is HUGE. We have a clustered environment and Search Heads (SH) receive configuration from our deployer. During initial setup the maximum bundle size was increased to allow pushing the PSC app from the deployer to the SHs. While it worked we've noticed that any push after adding the PSC app to the deployer now takes around 2 minutes to complete, regardless of how small of a change even if no restart is needed. I was hoping there was a way to install the PSC app locally in the search head cluster without going through the deployer. To option to "install from file" is not present in the web UI, assumably since we have a deployer for managing apps. Removing the app from the deployer and consequentially the SH cluster I tried to unpack the app into the /opt/splunk/etc/apps folder but it is then removed/deleted automatically as the cluster is restarted. Presumably since it is not available on the deployment server. So, how should we install and use PSC in a clustered environment? Is the only/correct way to to push the giant app from the deployer or is there another way to distribute the app? All feedback and/or suggestions are welcome

fatsug · ‎04-07-2025

I have similar issues poppping up as of late. But how does one isolate the affected forwarder? The error message reads Forwarder Ingestion Latency Root Cause(s): Indicator 'ingestion_latency_gap_multiplier' exceeded configured value. The observed value is 89. Message from <UUID>:<ip-addrs>:54246 Unhealthy Instances: indexer1 indexer2 The "message from" section just lists the UUID, an IP adress and a port. Which part here would help me find the actual forwarder? The UUID does not match any "Client name" under forwarder management on the deployment server. The IP adress does not match a server on which I have a forwarder installed. One or a few of the indexers are listed as "unhealthy instances" each time. But the actual error sounds like it lives in the forwarder end and not on the indexer. With the available information in this warning/error. How can I figure out which forwarder is either experiencing latency issues OR need to have that log file mentioned flushed.

fatsug · ‎04-07-2025

So, how does one isolate the affected forwarder? The error message reads Forwarder Ingestion Latency Root Cause(s): Indicator 'ingestion_latency_gap_multiplier' exceeded configured value. The observed value is 89. Message from <UUID>:<ip-addrs>:54246 Unhealthy Instances: indexer1 indexer2 The "message from" section just lists the UUID, an IP adress and a port. Which part here would help me find the actual forwarder? The UUID does not match any "Client name" under forwarder management on the deployment server. The IP adress does not match a server on which I have a forwarder installed. One or a few of the indexers are listed as "unhealthy instances" each time. But the actual error sounds like it lives in the forwarder end and not on the indexer. With the available information in this warning/error. How can I figure out which forwarder is either experiencing latency issues OR need to have that log file mentioned flushed.

fatsug · ‎03-24-2025

Sure can 🙂 So, PickleRick and you basically made the same argument which helped a lot. I'm not going to duplicate the whole reply, hope that OK. I can still split the same "disk" into warm and cold defining different amounts of storage, or just dump everything in the same volume and let Splunk figure it out. Thanks for the feedback

fatsug · ‎03-24-2025

Thank you for the feedback. The long answer to @PickleRick should cover most if my reply, just to clarify. There is no disc tier, only SSD all the way at a flat rate. This is why I was curious if there was any point in keeping any or just a minimum amount of cold storage available. Retention time control seems to be the best argument so far 🙂

fatsug · ‎03-24-2025

That is probably the strongest argument right now, as there is only SSD storage at a flat rate. So if i configure nothing and just set a path to "storage" data [volume:storage] path = /data/splunk/warm/ # adjust when correct disk is mounted maxVolumeDataSizeMB = 2800000 Then use "storage" for both warm and cold , I assume that there is a "default" balance beteween home (hot/warm) and cold with regards to space allocated. So I then have no control over the sizes of home and cold. However, I can set the forezenTimePeriodInSecs and delete based on time from cold with higher accuracy. I assume I could just add another definition for the same volume mounted and divide the space between hot and cold [volume:varm] path = /data/splunk/warm/ # adjust when correct disk is mounted maxVolumeDataSizeMB = 1000000 [volume:cold] path = /data/splunk/warm/ # adjust when correct disk is mounted maxVolumeDataSizeMB = 1800000 Should produce similar results as just keeping the defaults (as indicated by other replies). The question remaining then is, should I just separate volumes into warm and cold so that I can more easily expand cold storage when needed. As well as some possible efficiency gains. Thank you for your feedback!

fatsug · ‎03-21-2025

Without a tiered storage model it seems like there would be little argument for using cold/frozen storage. Except potentially if additional compression helps save space. If not, using only a homePath in indexes.conf would seem like it would make all data readily accesible as hot/warm. However, checking the documentation there seems to be three paths for indexes that are reqired for Splunkd to start being homePath, coldPatch and thawedPath. (indexes.conf - Splunk Documentation) So using a single disk/volume/mount, what does the inputs.conf look like? Should the same path just be set for all three? Making sure that maxVolumeDataSizeMB adds up to the total volume available on /data/splunk/warm. [volume:storage] path = /data/splunk/warm/ # adjust when correct disk is mounted maxVolumeDataSizeMB = 2800000 ... ... [volume:_splunk_summaries] path = /data/splunk/warm/ # ~ 200GB maxVolumeDataSizeMB = 200000 ... ... [main] homePath = volume:storage/defaultdb/db coldPath = volume:storage/defaultdb/colddb thawedPath = $SPLUNK_DB/defaultdb/thaweddb [history] homePath = volume:storage/historydb/db coldPath = volume:storage/historydb/colddb thawedPath = $SPLUNK_DB/historydb/thaweddb [summary] homePath = volume:storage/summarydb/db coldPath = volume:storage/summarydb/colddb thawedPath = $SPLUNK_DB/summarydb/thaweddb ... ... [windows] homePath = volume:storage/windows/db coldPath = volume:storage/windows/colddb summaryHomePath = volume:storage/windows/summary thawedPath = $SPLUNK_DB/windows/thaweddb tstatsHomePath = volume:_splunk_summaries/windows/datamodel_summary frozenTimePeriodInSecs = 63072000 [linux] homePath = volume:storage/linux/db coldPath = volume:storage/linux/colddb summaryHomePath = volume:storage/linux/summary thawedPath = $SPLUNK_DB/linux/thaweddb tstatsHomePath = volume:_splunk_summaries/linux/datamodel_summary frozenTimePeriodInSecs = 63072000 I'm assuming this would work, right? Though as it seems that Splunk requires, and does make use of "cold" and "thawed" anyway, does it make more sence to just partition mounts for warm and cold separately anyway? [volume:warm] path = /data/splunk/warm/ # adjust when correct disk is mounted maxVolumeDataSizeMB = 500000 [volume:cold] path = /data/splunk/warm/ # adjust when correct disk is mounted maxVolumeDataSizeMB = 2500000 ... ... [volume:_splunk_summaries] path = /data/splunk/warm/ # ~ 200GB maxVolumeDataSizeMB = 200000 ... ... [main] homePath = volume:warm/defaultdb/db coldPath = volume:cold/defaultdb/colddb thawedPath = $SPLUNK_DB/defaultdb/thaweddb [history] homePath = volume:warm/historydb/db coldPath = volume:cold/historydb/colddb thawedPath = $SPLUNK_DB/historydb/thaweddb [summary] homePath = volume:warm/summarydb/db coldPath = volume:cold/summarydb/colddb thawedPath = $SPLUNK_DB/summarydb/thaweddb ... ... [windows] homePath = volume:warm/windows/db coldPath = volume:cold/windows/colddb summaryHomePath = volume:warm/windows/summary thawedPath = $SPLUNK_DB/windows/thaweddb tstatsHomePath = volume:_splunk_summaries/windows/datamodel_summary frozenTimePeriodInSecs = 63072000 [linux] homePath = volume:warm/linux/db coldPath = volume:warm/linux/colddb summaryHomePath = volume:warm/linux/summary thawedPath = $SPLUNK_DB/linux/thaweddb tstatsHomePath = volume:_splunk_summaries/linux/datamodel_summary frozenTimePeriodInSecs = 63072000 Does it matter and what would be "best praxis".

fatsug · ‎12-19-2024

"There's an app for that" 😄 Thank you!

fatsug · ‎12-18-2024

I feel content accepting that bundle checksums are by design present to visualise the different states and not to detect modifications between validation and application. Thank you for the link though, I will definetly have a look at the presentation. With a little luck it might even be available online somewhere. All the best

Posts	152
Solutions	5
Karma Given	69
Karma Received	9
Member Since	‎12-17-2021

Online Status	Offline
Date Last Visited	‎07-17-2025 02:24 AM

Incorrect regex for command_for_linux_audit?

Adaptive filtering/Stateful cross-event logic to f...

Iowait: Sum of 3 highest per-cpu iowaits reached x...

Why is the License manager reporting ingest based ...

Local, non distributed app install in clustered se...

How to handle homePath, coldPath and thawedPath on...

Why is the checksum for active_bundle not matching...

Possible lineabreaker issue with lastlog in Splunk...

URA reports incompatanility with JQuery 3.5

Why is Splunk_TA_nix generating a lot of DateParse...

Re: Incorrect regex for command_for_linux_audit?

Incorrect regex for command_for_linux_audit?

Re: Iowait: Sum of 3 highest per-cpu iowaits reach...

Re: Adaptive filtering/Stateful cross-event logic ...

Re: Adaptive filtering/Stateful cross-event logic ...

Re: Iowait: Sum of 3 highest per-cpu iowaits reach...

Adaptive filtering/Stateful cross-event logic to f...

Re: Iowait: Sum of 3 highest per-cpu iowaits reach...

Iowait: Sum of 3 highest per-cpu iowaits reached x...

Re: Why is the License manager reporting ingest ba...

Why is the License manager reporting ingest based ...

Re: Local, non distributed app install in clustere...

Re: Local, non distributed app install in clustere...

Re: Defining a global token for alert recipients

Re: Local, non distributed app install in clustere...

Re: Local, non distributed app install in clustere...

Local, non distributed app install in clustered se...

Re: How to see Root cause for Ingestion Latency

Re: Forwarder Ingestion Latency

Re: How to handle homePath, coldPath and thawedPat...

Re: How to handle homePath, coldPath and thawedPat...

Re: How to handle homePath, coldPath and thawedPat...

How to handle homePath, coldPath and thawedPath on...

Re: Why is the checksum for active_bundle not matc...

Re: Why is the checksum for active_bundle not matc...

Are you a member of the Splunk Community?