About fatsug

fatsug

I am tempted to switch this post to the "solution", it is not "exactly" what I asked for but it did achiecve the effect I was looking for. As per Use the deployer to distribute apps and configuration updates - Splunk Documentation. After an initial push of PSC and MLTK from the deployment server we added a local/app.conf file containing only [shclustering] deployer_push_mode = local_only The "push-time" dropped from 167 seconds to 47 seconds with push mode set in the PSC app, then to 24 seconds when also changing push mode in the MLTK app. So this did have the effect I was after, even though it is not strictly speaking a "local only install" of an app. All the best

fatsug

Did not know that you could do this on an "app level", but it might be worth looking into different push modes! Thanx

fatsug

"Email recipients groups" | Ideas

fatsug

https://ideas.splunk.com/ideas/APPSID-I-989

fatsug

Well crud... The PSC os something like 2.3 GB, so even if the entire app is not redistributed every push I suspect it severly affects the check and push process. No, it is a fresh and first time install so no inherited problems. As far as I can see, there are no errors or problems associated with the install itself and MLTK is running smoothly. So this is another "improvement" to wish for then, something like a ".gitignore" option where you can manually add apps to your cluster without having them "managed" by the SHD. My guess is that the only way around this would be a standalone SH where you'd basically only install PSC and MLTK and only use it for these purposes. In any case, thank you for your feedback and have a nice weekend

fatsug

This may be a "dumb" question, but I'll just throw it out there while I try to work it out. The Python for Scientific Computing (PSC) app is HUGE. We have a clustered environment and Search Heads (SH) receive configuration from our deployer. During initial setup the maximum bundle size was increased to allow pushing the PSC app from the deployer to the SHs. While it worked we've noticed that any push after adding the PSC app to the deployer now takes around 2 minutes to complete, regardless of how small of a change even if no restart is needed. I was hoping there was a way to install the PSC app locally in the search head cluster without going through the deployer. To option to "install from file" is not present in the web UI, assumably since we have a deployer for managing apps. Removing the app from the deployer and consequentially the SH cluster I tried to unpack the app into the /opt/splunk/etc/apps folder but it is then removed/deleted automatically as the cluster is restarted. Presumably since it is not available on the deployment server. So, how should we install and use PSC in a clustered environment? Is the only/correct way to to push the giant app from the deployer or is there another way to distribute the app? All feedback and/or suggestions are welcome

fatsug · ‎04-07-2025

I have similar issues poppping up as of late. But how does one isolate the affected forwarder? The error message reads Forwarder Ingestion Latency Root Cause(s): Indicator 'ingestion_latency_gap_multiplier' exceeded configured value. The observed value is 89. Message from <UUID>:<ip-addrs>:54246 Unhealthy Instances: indexer1 indexer2 The "message from" section just lists the UUID, an IP adress and a port. Which part here would help me find the actual forwarder? The UUID does not match any "Client name" under forwarder management on the deployment server. The IP adress does not match a server on which I have a forwarder installed. One or a few of the indexers are listed as "unhealthy instances" each time. But the actual error sounds like it lives in the forwarder end and not on the indexer. With the available information in this warning/error. How can I figure out which forwarder is either experiencing latency issues OR need to have that log file mentioned flushed.

fatsug · ‎04-07-2025

So, how does one isolate the affected forwarder? The error message reads Forwarder Ingestion Latency Root Cause(s): Indicator 'ingestion_latency_gap_multiplier' exceeded configured value. The observed value is 89. Message from <UUID>:<ip-addrs>:54246 Unhealthy Instances: indexer1 indexer2 The "message from" section just lists the UUID, an IP adress and a port. Which part here would help me find the actual forwarder? The UUID does not match any "Client name" under forwarder management on the deployment server. The IP adress does not match a server on which I have a forwarder installed. One or a few of the indexers are listed as "unhealthy instances" each time. But the actual error sounds like it lives in the forwarder end and not on the indexer. With the available information in this warning/error. How can I figure out which forwarder is either experiencing latency issues OR need to have that log file mentioned flushed.

fatsug · ‎03-24-2025

Sure can 🙂 So, PickleRick and you basically made the same argument which helped a lot. I'm not going to duplicate the whole reply, hope that OK. I can still split the same "disk" into warm and cold defining different amounts of storage, or just dump everything in the same volume and let Splunk figure it out. Thanks for the feedback

fatsug · ‎03-24-2025

Thank you for the feedback. The long answer to @PickleRick should cover most if my reply, just to clarify. There is no disc tier, only SSD all the way at a flat rate. This is why I was curious if there was any point in keeping any or just a minimum amount of cold storage available. Retention time control seems to be the best argument so far 🙂

fatsug · ‎03-24-2025

That is probably the strongest argument right now, as there is only SSD storage at a flat rate. So if i configure nothing and just set a path to "storage" data [volume:storage] path = /data/splunk/warm/ # adjust when correct disk is mounted maxVolumeDataSizeMB = 2800000 Then use "storage" for both warm and cold , I assume that there is a "default" balance beteween home (hot/warm) and cold with regards to space allocated. So I then have no control over the sizes of home and cold. However, I can set the forezenTimePeriodInSecs and delete based on time from cold with higher accuracy. I assume I could just add another definition for the same volume mounted and divide the space between hot and cold [volume:varm] path = /data/splunk/warm/ # adjust when correct disk is mounted maxVolumeDataSizeMB = 1000000 [volume:cold] path = /data/splunk/warm/ # adjust when correct disk is mounted maxVolumeDataSizeMB = 1800000 Should produce similar results as just keeping the defaults (as indicated by other replies). The question remaining then is, should I just separate volumes into warm and cold so that I can more easily expand cold storage when needed. As well as some possible efficiency gains. Thank you for your feedback!

fatsug · ‎03-21-2025

Without a tiered storage model it seems like there would be little argument for using cold/frozen storage. Except potentially if additional compression helps save space. If not, using only a homePath in indexes.conf would seem like it would make all data readily accesible as hot/warm. However, checking the documentation there seems to be three paths for indexes that are reqired for Splunkd to start being homePath, coldPatch and thawedPath. (indexes.conf - Splunk Documentation) So using a single disk/volume/mount, what does the inputs.conf look like? Should the same path just be set for all three? Making sure that maxVolumeDataSizeMB adds up to the total volume available on /data/splunk/warm. [volume:storage] path = /data/splunk/warm/ # adjust when correct disk is mounted maxVolumeDataSizeMB = 2800000 ... ... [volume:_splunk_summaries] path = /data/splunk/warm/ # ~ 200GB maxVolumeDataSizeMB = 200000 ... ... [main] homePath = volume:storage/defaultdb/db coldPath = volume:storage/defaultdb/colddb thawedPath = $SPLUNK_DB/defaultdb/thaweddb [history] homePath = volume:storage/historydb/db coldPath = volume:storage/historydb/colddb thawedPath = $SPLUNK_DB/historydb/thaweddb [summary] homePath = volume:storage/summarydb/db coldPath = volume:storage/summarydb/colddb thawedPath = $SPLUNK_DB/summarydb/thaweddb ... ... [windows] homePath = volume:storage/windows/db coldPath = volume:storage/windows/colddb summaryHomePath = volume:storage/windows/summary thawedPath = $SPLUNK_DB/windows/thaweddb tstatsHomePath = volume:_splunk_summaries/windows/datamodel_summary frozenTimePeriodInSecs = 63072000 [linux] homePath = volume:storage/linux/db coldPath = volume:storage/linux/colddb summaryHomePath = volume:storage/linux/summary thawedPath = $SPLUNK_DB/linux/thaweddb tstatsHomePath = volume:_splunk_summaries/linux/datamodel_summary frozenTimePeriodInSecs = 63072000 I'm assuming this would work, right? Though as it seems that Splunk requires, and does make use of "cold" and "thawed" anyway, does it make more sence to just partition mounts for warm and cold separately anyway? [volume:warm] path = /data/splunk/warm/ # adjust when correct disk is mounted maxVolumeDataSizeMB = 500000 [volume:cold] path = /data/splunk/warm/ # adjust when correct disk is mounted maxVolumeDataSizeMB = 2500000 ... ... [volume:_splunk_summaries] path = /data/splunk/warm/ # ~ 200GB maxVolumeDataSizeMB = 200000 ... ... [main] homePath = volume:warm/defaultdb/db coldPath = volume:cold/defaultdb/colddb thawedPath = $SPLUNK_DB/defaultdb/thaweddb [history] homePath = volume:warm/historydb/db coldPath = volume:cold/historydb/colddb thawedPath = $SPLUNK_DB/historydb/thaweddb [summary] homePath = volume:warm/summarydb/db coldPath = volume:cold/summarydb/colddb thawedPath = $SPLUNK_DB/summarydb/thaweddb ... ... [windows] homePath = volume:warm/windows/db coldPath = volume:cold/windows/colddb summaryHomePath = volume:warm/windows/summary thawedPath = $SPLUNK_DB/windows/thaweddb tstatsHomePath = volume:_splunk_summaries/windows/datamodel_summary frozenTimePeriodInSecs = 63072000 [linux] homePath = volume:warm/linux/db coldPath = volume:warm/linux/colddb summaryHomePath = volume:warm/linux/summary thawedPath = $SPLUNK_DB/linux/thaweddb tstatsHomePath = volume:_splunk_summaries/linux/datamodel_summary frozenTimePeriodInSecs = 63072000 Does it matter and what would be "best praxis".

fatsug · ‎12-19-2024

"There's an app for that" 😄 Thank you!

fatsug · ‎12-18-2024

I feel content accepting that bundle checksums are by design present to visualise the different states and not to detect modifications between validation and application. Thank you for the link though, I will definetly have a look at the presentation. With a little luck it might even be available online somewhere. All the best

fatsug · ‎12-18-2024

While I still find this "counter intuitive" speaking of checksums, I understand that this is by design. The validated bundle is modified upon application and hence the checksum changes. This is not to prevent pushing bundles that have been modified after validation but rather to keep track of states of changes/bundles. Thank you for the explanation

fatsug · ‎12-17-2024

Thank you @sainag_splunk I do understand the "how" so to speak, but I do not understand the "why". If I validate a bundle and then push that bundle I would assume that the checksum for the validated bundle after a push should be the same as the new and current one. Otherwise I cannot validate that the last validated bundle is exactly what has been pushed and applied. But maybe I'm missing the point? If the checksums are not there to validate changes but rather just to track the presence of unapplied changes, then its fine. I suspect that at this point the logical follow-up question is, how are bundles modified after validation but before being applied? Or are the checksums only for the changed files/settings and the final cheksum is for the entire bundle? All the best

fatsug · ‎12-13-2024

This is not a particulary crucial question but it has been nagging me for a while. When applying changes to indexes.conf on the manager node I usually do a calidate -> show -> apply round. But I am somewhout confused about a detail. When you validate a bundle after making modifications you get an updated "last_validated_bundle" checksum. In my mind, when you then apply this bundle the now "last_validated_bundle" checksum should become the new "latest_bundle" and "active_bundle". But this is not the case. $ splunk apply cluster-bundle Creating new bundle with checksum=<does_not_match_last_validated_bundle> Applying new bundle. The peers may restart depending on the configurations in applied bundle. Please run 'splunk show cluster-bundle-status' for checking the status of the applied bundle. OK After the new bundle has been applied active_bundle, latest_bundle and last_validated_bundle checksums all match again. But why is the checksum produced after bundle validation not matching the checksum for the applied bundle? Have a great weekend!

fatsug · ‎10-30-2024

Hi all After installing Splunk_TA_nix with no local/inputs on heavy forwarders the error I was seeing in this post went away. So that one was actually solved. However, the issue with missing linebreaks in the output mentionen by @PickleRick remains. "1) Breaks the whole lastlog output into separate events on the default LINE_BREAKER (which means every line is treated as separate event)" So I thought I'd see if I could get that one confirmed and/or fixed as well When searching for "source=lastlog" right now I get get a list of events from each host like so: > user2 10.0.0.1 Wed Oct 30 11:20 > another_user 10.0.0.1 Wed Oct 30 11:21 > discovery 10.0.0.2 Tue Oct 29 22:19 > scanner 10.0.0.3 Mon Oct 28 21:39 > admin_user 10.0.0.4 Mon Oct 21 11:19 > root 10.0.0.1 Tue Oct 1 08:57 Before placing the TA on the HFs I would see output only containing the header > USERNAME FROM LATEST Which is completely useless 😄 After adding the TA to the HFs this "header" line is no longer present, at all, in any events from any server. While Field names are correct and fully searchable with IP adresses, usernames etc. My question at this point is probably best formulated as "am I alright now"? 😁 Based on the feedback in the previous post I was sort of assuming that the expected output/events should be the same as the screen output when running the script locally, i.e. one event with the entire output, like so USERNAME FROM LATEST user2 10.0.0.1 Wed Oct 30 11:20 another_user 10.0.0.1 Wed Oct 30 11:21 discovery 10.0.0.2 Tue Oct 29 22:19 scanner 10.0.0.3 Mon Oct 28 21:39 admin_user 10.0.0.4 Mon Oct 21 11:19 root 10.0.0.1 Tue Oct 1 08:57 While I can see this as being easier on the eyes and easier to interpret when found, it could make processing individual filed:value pairs more problematic in searches. So what I am wondering, is everything "OK" now? Or am I still getting events with incorrect linebreaks? I don't know what the expected/correct output should be. Best regards

fatsug · ‎09-26-2024

So it is now flagged to include "z" versions of OpenSSL, meaning that all prior and current versions were and are indeed affected? Could you provide a link to the supporting information?

fatsug · ‎09-23-2024

Sorry for the delay Exporting the scan results did provide additional information, as with most other apps the problem is with "backups" of older versions of the app ".../default.old.20240828…i/views/attribution.xml" So URA is triggering on "old" folders which are no longer active. The remaining questions hence is "to delete or not to delete"? I know I've participated in these discussions before. For "private" apps I could normally just ignore a specific search path for an app, this is not possible for the "splunk base app". So either I have to ingore the "failing" (false positives) apps completely, or manually delete "old" folders. What is the "best praxis" here?

fatsug · ‎09-17-2024

I get weekly email updates with results from weekly URA scans. After noticing that we had outdated apps we rolled out updates for three public apps, Sankey Diagram, Scalable Vector Graphics and Splunk Dashboard Examples. In our testing environment URA is now content and all apps pass jQuery scans without issues. However, in our production environment URA scan still fails in all three apps. It does not specify which files or of there is a problem om one or all instances so I don’t know what is causing the results. I have double and triple checked the apps comparing hash values for every file both on the deployment server and on all individual test and production search heads. Everything except for the “install hash” in “meta.local” is identical in both test and production environment. Apps are all identical between cluster members in test and production environment respectively. There are not additional files present on any search head in the production environment. Why is URA still failing these apps only in the production environment? How can I identify the reason for the scan failures as I they should all pass in both environments, being identical and all. Any and all suggestions are most welcome All the best

fatsug · ‎08-26-2024

Reasonable enough, and yes I do get incorrect linebreaker. So in this case the "fix" is to install the TA on the HF layer. Maybe it's just me bur this could be indicated with a bit more "urgency" in the documentation? The first table indicates "This add-on supports forwarders of any type for data collection. The host must run a supported version of *nix." which is not really the same as "needed for correct parsing of logs". The "Distributed deployment feature compatibility" does not even list HF, so while it is logical it is not really intuitive based on the documentation 🙂 Thanks and all the best

fatsug · ‎08-26-2024

So the reason for the observed error, only reported by the heavy forwarders, seems clear. The TA is not installed on the heavy forwarders. I might be missing some dependency but I fail to see where this date would fit? The HF complains about the incorrectly genereated dates as being "out of bounds", but the output from the script does not contain "year" or a reasonable "timestamp" (as the year is missing). PRINTF='{printf "%-30s %-30.30s %-s\n", username, from, latest}' if [ "$KERNEL" = "Linux" ] ; then CMD='last -iw' # shellcheck disable=SC2016 FILTER='{if ($0 == "") exit; if ($1 ~ /reboot|shutdown/ || $1 in users) next; users[$1]=1}' # shellcheck disable=SC2016 FORMAT='{username = $1; from = (NF==10) ? $3 : "<console>"; latest = $(NF-6) " " $(NF-5) " " $(NF-4) " " $(NF-3)}' The incorrect years are not included in the indexed events, so it seems that there is little to gain by trying to "fix" this on the HF layer? The set-up is Universal Forwarder (TA nix) --> Heavy Forwarder () --> Indexer () --> Search Head (TA nix). The documentations seems to indicate installing the plugin on HF as "optional" (Install the Splunk Add-on for Unix and Linux - Splunk Documentation)? The intention would not be "data collection" from the HF server and I don't see that this would necessarily fix the issue? If these events from lastlog.sh should not contain years and do not contain a readable timestamp, then should this error not be automatically supressed by the HF?

fatsug · ‎08-25-2024

Checking the "host" details these complaints are all coming from the heavy forwarders, so I assumed this is where I should be checking. Running the command on a heavy forwarder produced the following output: /opt/splunk/etc/system/default/props.conf DATETIME_CONFIG = /etc/datetime.xml While on any search head or indexer the output is as you suggested: /opt/splunk/etc/apps/Splunk_TA_nix/default/props.conf DATETIME_CONFIG = CURRENT There is no '/etc/datetime.xml', however there is an '/opt/splunk/etc/datetime.xml' file. I have no idea of the configuration is reffering to a non-existing file or the built in Splunk one. I don't know if or why anyone would have modified this setting, or the consequences of doing such. I'll do some reasearch on my own but any feedback and/or suggestions are more than welcome

fatsug · ‎08-23-2024

I'm using the Splunk TA for linux to collect serverlogs. Some background Looking in the "_internal" log I am seing a lot of these errors: 08-23-2024 15:52:39.910 +0200 WARN DateParserVerbose [6460 merging_0] - A possible timestamp match (Wed Aug 19 15:39:00 2015) is outside of the acceptable time window. If this timestamp is correct, consider adjusting MAX_DAYS_AGO and MAX_DAYS_HENCE. Context: source=lastlog|host=<hostname>|lastlog|13275 08-23-2024 15:52:39.646 +0200 WARN DateParserVerbose [6460 merging_0] - A possible timestamp match (Fri Aug 7 09:08:00 2009) is outside of the acceptable time window. If this timestamp is correct, consider adjusting MAX_DAYS_AGO and MAX_DAYS_HENCE. Context: source=lastlog|host=<hostname>|lastlog|13418 08-23-2024 15:52:32.378 +0200 WARN DateParserVerbose [6506 merging_1] - A possible timestamp match (Fri Aug 7 09:09:00 2009) is outside of the acceptable time window. If this timestamp is correct, consider adjusting MAX_DAYS_AGO and MAX_DAYS_HENCE. Context: source=lastlog|host=<hostname>|lastlog|13338 This is slightly confusing and somewhat problematic as the "lastlog" is collected not through a filewatch but from scripted output. The "lastlog" file is not collected/read and a stats-check on the file confirms accurate dates. However, this is not the source of the problem. I cannot se anything in the output from the commands in the script (Splunk_TA_nix/bin/lastlog.sh) which would indicate the precense of a "year"/timestamp. The indexed log does not contain "year" and the actual _time timestamp is correct. These "years" in "_internal" are also from a time when the server was not running/present, so they are not collected from any actual source "on the server". And the questions - Why am I seeing these errors - From where are these problematic "timestamps" generated - How do I fix the issue All the best

Posts	141
Solutions	5
Karma Given	64
Karma Received	9
Member Since	‎12-17-2021

Online Status	Offline
Date Last Visited	Wednesday

Local, non distributed app install in clustered se...

How to handle homePath, coldPath and thawedPath on...

Why is the checksum for active_bundle not matching...

Possible lineabreaker issue with lastlog in Splunk...

URA reports incompatanility with JQuery 3.5

Why is Splunk_TA_nix generating a lot of DateParse...

Using basesearch with subsearch

Defining a global token for alert recipients

"Custom" maps via geojson/json

Priority and editing of local.meta/default.meta

Re: Local, non distributed app install in clustere...

Re: Local, non distributed app install in clustere...

Re: Defining a global token for alert recipients

Re: Local, non distributed app install in clustere...

Re: Local, non distributed app install in clustere...

Local, non distributed app install in clustered se...

Re: How to see Root cause for Ingestion Latency

Re: Forwarder Ingestion Latency

Re: How to handle homePath, coldPath and thawedPat...

Re: How to handle homePath, coldPath and thawedPat...

Re: How to handle homePath, coldPath and thawedPat...

How to handle homePath, coldPath and thawedPath on...

Re: Why is the checksum for active_bundle not matc...

Re: Why is the checksum for active_bundle not matc...

Re: Why is the checksum for active_bundle not matc...

Re: Why is the checksum for active_bundle not matc...

Why is the checksum for active_bundle not matching...

Possible lineabreaker issue with lastlog in Splunk...

Re: Why is OPENSSL vulnerability still showing in ...

Re: URA reports incompatanility with JQuery 3.5

URA reports incompatanility with JQuery 3.5

Re: Why is Splunk_TA_nix generating a lot of DateP...

Re: Why is Splunk_TA_nix generating a lot of DateP...

Re: Why is Splunk_TA_nix generating a lot of DateP...

Why is Splunk_TA_nix generating a lot of DateParse...

Are you a member of the Splunk Community?