About nunoaragao

nunoaragao · ‎04-16-2025

Old thread, but leaving it here in hope other people might chip in. Also seen the same WARNs, at a time we suffered from KV Store consuming a whole lot of CPU and wiredTigerCacheSizeGB, over 15 times the sum of any collection. Opening a Splunk Support case we were told: Splunk does not use KV Store to manage or store the history of scheduled searches. Scheduled searches are managed and tracked via internal logs, dispatch directories, and internal indexes, not KV Store However, we occasionally see those events on Search Heads, and frequently on Heavy Forwarders on role of parsing and routing, not connected to License Manager and disabled KV Store.

nunoaragao · ‎07-09-2024

Dislike to reply to my own comment, but I got an answer from Splunk Support. HTTP Event Collector does NOT log metrics from UF sending data over HTTP, and this is reported on internal ticket SPL-239230 : "No metrics are sent to the http_event_collector_metrics.log" which has been in backlog since 2023.

nunoaragao · ‎07-03-2024

Hi @edoardo_vicendo , thanks for the reply. Yeah, no issue with the sending of data. Like you, we managed to crack it. But the HEC that receives the data is also receiving from an appliance and a AWS Firehose, on two other input tokens. Using the Splunk search I sent, I'm able to see metrics for connections, bytes ingested and parsing errors for those two other tokens, but NONE from the token used by the UF using S2S over HTTP.

nunoaragao · ‎07-03-2024

Hi @edoardo_vicendo , Do you still have your working setup? Do you find that introspection logs from the HEC receiver instances report metrics for tokens used by "/services/collector/raw" and "/services/collector/event", but not "/services/collector/s2s" ? index="_introspection" component=HttpEventCollector data.token_name=*

nunoaragao · ‎10-23-2023

Did you check the values of Report_Id I mentioned earlier @mdsnmss ? Are they repeating or all unique?

nunoaragao · ‎10-06-2023

.. but that is your own post. sorry.

nunoaragao · ‎10-06-2023

Oh! If the issue is tied toWinEventLog://ForwardedEvents maybe you'd like to also comment on splunk-winevtlog.exe keeps crashing post. I also stumbled of this input stanza on my investigation, but on our estate, we're not enabling it.

nunoaragao · ‎09-28-2023

So, the result of our troubleshoot, was that all of the hundreds of crash events we were seeing related to winevtlog or perfmon, were not crashes happening but re-attempts of sending past crashes to Microsoft, and failing, because these were on an air-gaped subset of the estate. The clue was, apart from ceasing the activity by clearing all the WER folder, that Report_Id just kept repeating ... sourcetype="WinEventLog:Application" AppCrash | regex Message=".*(?<splunk>splunk[-\w]+)" | timechart span=30m dc(Report_Id) This search showed a constant value of 12 across several days Then we found there is a GPO that tells servers to log AppCrash events, but not send them to Microsoft

nunoaragao · ‎09-14-2023

Hi ! We're having similar issue as well, with splunk-winevtlog and splunk-perfmon being the culprits, but Signature P4 is not KERNELBASE but the splunk process itself. It only affects Server 2019. Bursts of AppCrash show up every 5 minutes or so. We've tried version 9.0.6 and 9.1.1 We opened a support case, turned on DEBUG logging and sent them diags, etc .. the works .. What we've found was that, clearing the WER folder of AppCrash files C:\ProgramData\Microsoft\Windows\WER ( Settings > Storage > Free Space > Windows Error Report - check delete Files ) .. would cease the issue on that single server .. at least until the issue gets triggered again. Did you get your issue fixed with release version 9.0.6 @PeterBoard ? Thanks

nunoaragao · ‎12-13-2022

To @potnuru and others that might fall here .. there is a setting on transforms.conf called replicate to: Indicates whether to replicate CSV lookups to indexers

nunoaragao · ‎07-21-2022

Agree @chadmedeiros there is no event log for the export of a diag. The issue was made worse when Splunk replaced their Support Portal, and the new portal doesn't show customers their own uploaded files. They claim the old one never did, so, until I can prove them they'll just log that as a suggestion. Does anyone keep a screenshot of the old portal with the uploaded files section right below the description? Thanks.

nunoaragao · ‎01-27-2022

.. and to make chrony output similar to ntpdate, replaced the final section like this (switching the last lines of code to ntpdate condition) ( ... ) else CMD2="chronyc" assertHaveCommand $CMD2 chronyc -n sources | awk -F"[] []+" '/\^/ {printf "server %s, stratum %d, offset %s, delay %s\n",$2,$3,$8,$10}' Attempted to also use variable $AWK like most of the other scripts but kept having double quotes escaping errors .. gave up. Maybe not efficient but does the trick

nunoaragao · ‎01-26-2022

Regarding TA nix >8.1 with support for Chrony, I'm deflated that any recent version does not attempt at having similar log events from systems with Chrony or systems with ntpdate. As most other scripts do with smart awk expressions. Also, on systems with ntpdate BUT that run Chrony for time sync (hint: RHEL7), the script bypasses Chrony configuration. But this I managed to fix it myself by including lines 2 and 3: if [ $FOUND_NTPDATE -eq 0 ] ; then echo "Found ntpdate command" >> $TEE_DEST if [ -f /etc/chrony.conf ] ; then # Special CONFIG=/etc/chrony.conf elif [ -f /etc/ntp.conf ] ; then # Linux; FreeBSD; AIX; Mac OS X maybe CONFIG=/etc/ntp.conf elif [ -f /etc/inet/ntp.conf ] ; then # Solaris CONFIG=/etc/inet/ntp.conf elif [ -f /private/etc/ntp.conf ] ; then # Mac OS X CONFIG=/private/etc/ntp.conf else CONFIG= fi

nunoaragao · ‎01-18-2022

While most Warn and Errors show up on the Job dropdown (1) some are also displayed in an area right below the search bar (2). Looking at HTML this placeholder is named search-searchflashmessages What's the name for this area ( to discuss this with Support ) ? Is it possible to config which messages to show/not show in here ? Looking at documentation this area is not detailed: https://docs.splunk.com/Documentation/Splunk/8.2.4/Search/WhatsinSplunkSearch

nunoaragao · ‎11-19-2021

Hi @samjenk_2 , These are the events used to toggle health colours in Splunk's Health Report. So we get to know what are the typical values, and in which servers. index=_introspection sourcetype=splunk_resource_usage component=IOWait

nunoaragao · ‎11-11-2021

Just got an answer from Splunk Support, @kisstian / @samjenk_2 / everyone Splunk have now updated their documentation regarding disable health report features. It states in a box: If distributed health reporting is enabled for your deployment, disabling a feature on the local instance will not be reflected in the health report. It seems, the workaround to disable a feature in +8.2 has just became a feature. The old behavior in +8.1 in which you could disable a single feature regardless of distributed health report has been "improved" If anyone submits an Idea to bring the old behavior back, let me know. You get my vote.

nunoaragao · ‎11-10-2021

Hi @samjenk_2 , my case(s) with Splunk Support were #2733102 and #2737559 .. SPL-213405 is Splunk's internal JIRA to track this issue. It may, or not, then show up on Splunk's release notes as known issue. It's still being investigated. If you deal with Support you can ask to link with it. My issue is that Docs say "You can disable any feature (...) for example, if you want to exclude a feature's status from the health report". So we expect to be able to disable a specific feature (i.e. Buckets) without requiring to disable distributed_health_reporter, which would also disable/hide a lot of other features if we're on a typical topology where we have search head clusters and clustered indexers. In other words, tell the Search Head to gray out a Indexer peer feature even if that peer is reporting health. It matches your scenario ?

nunoaragao · ‎10-27-2021

I also found this issue on 8.2.2.1, and Splunk is tracking this on SPL-213405. My issue is that I disable IOWait and Buckets health check on the Search Heads, but as long as search peers report issues with these features, they'll still show up. Except if we disable distributed health report, but then we lose visibility on a lot of other health checks. And apparently, the documentation now reports that distributed health report is enabled by default.

nunoaragao · ‎10-21-2021

I created two Splunk Support cases related to another item on Splunk Health, and it would be simpler if these had better support documentation. The item was actually Buckets, which is listed on health.conf as feature:buckets stanza with two indicators. One of them percent_small_buckets_created_last_24h What I found was, on app splunk_instrumentation there is a savedsearch called [instrumentation.deployment.index] that writes these Metrics to _internal based on querying rest /services/data/indexes endpoint. 10-21-2021 12:02:06.956 +0100 INFO PeriodicHealthReporter - feature="Buckets" color=red indicator="percent_small_buckets_created_last_24h" And then there is a second savedsearch with stanza [instrumentation.usage.healthMonitor.currentState] that joins a table returned from rest endpoints with these logs read from _internal. The search returns a JSON structure with the overall Health Report stats. Is this structure used to populate the Health Report on the Web Interface ? Unsure. As I said .. two tickets ..

nunoaragao · ‎10-08-2021

Where does the License Manager fit this picture ? Is License Manager 8.0 compatible with 8.2 Indexing layer ? Or does it need to be upgraded at the same time as Indexers ?

nunoaragao · ‎10-04-2021

Hi Splunkers, Long time ago we setup a SH cluster, and added search peers using CLI Some time later we changed the setup and began setting the search peers via an App pushed from the deployer. All $SPLUNK/etc/system/local distsearch.conf files were purged, and the app contains some peers on distsearch.conf and clustered indexers on server.conf Recently we found one cluster member stubbornly kept re-creating a distsearch.conf on system/local that overrided the cluster's configs ( pushed via the App ). Removing the file and doing a rolling restart, the file showed up again. Cleaning the KV Store and adding that member to the cluster restored the rogue distsearch.conf again. We also deleted *bundle files on $SPLUNK/var/run. Following the steps described as "Add a member that was both removed and disabled" here: https://docs.splunk.com/Documentation/Splunk/8.2.2/DistSearch/Addaclustermember We found when calling "splunk add shcluster-member" on the Captain, the "new" member's system/local path was purged from some files and re-created .. with the rogue distsearch.conf we were trying to get rid. And this event showed up on _internal ERROR ApplicationUpdater - Error reloading : handler for distsearch (access_endpoints /search/distributed/bundle-replication-files, /search/distributed/peers) Any ideas ? The final solution was to nuke the VM and re-create from scratch.

nunoaragao · ‎05-26-2021

I can see the exact same issue with forwarders +8, but only on Win Servers 2019. Has anyone found a fix ?

nunoaragao · ‎05-12-2021

Maybe not working because SOURCE_KEY = MetaData:Host will return host:: at the begging of the string. Maybe change REGEX to either REGEX = ^host\:\:8\.\d{1,3}\.\d{1,3}\.\d{1,3} REGEX = (?<!\d)8\.\d{1,3}\.\d{1,3}\.\d{1,3}

nunoaragao · ‎12-24-2019

I've hit the same snag on some AMI on AWS. Could not figure out an elegant solution, so I've resorted to: DUPLEX= cat /sys/class/net/$iface/duplex 2>/dev/null | sed 's/./\u&/' ... SPEED= cat /sys/class/net/$iface/speed 2>/dev/null Which redirects the errors to dev null. That was Splunk's implementation back on version 5 of the add-on.

nunoaragao · ‎10-30-2019

cat: /sys/class/net/eth7/speed: will return Invalid argument if the NIC is disconnected. Same for duplex. The way I fixed this, was to previously check if the state is "Up", like this if [ -r /sys/class/net/$iface/duplex ]; then if grep -Fq "up" /sys/class/net/$iface/operstate; then DUPLEX=`cat /sys/class/net/$iface/duplex | sed 's/./\u&/'` [ .... more original code here .... ] fi Other lines of code might also do the trick

Posts	29
Solutions	0
Karma Given	4
Karma Received	2
Member Since	‎03-29-2017

Online Status	Offline
Date Last Visited	‎04-16-2025 02:30 AM

Warn and Error messages below search bar

distsearch.conf on Search Head cluster

Re: KV store issue for collection SavedSearchHisto...

Re: Configure Universal forwarder 8.1.0 to send da...

Re: How do I Configure Universal forwarder 8.1.0 t...

Re: Configure Universal forwarder 8.1.0 to send da...

Re: Getting crash events in windows application lo...

Re: Getting crash events in windows application lo...

Re: Getting crash events in windows application lo...

Re: How to resolve crash events in windows applica...

Re: How to resolve crash events in windows applica...

Re: Using Local=true in Automatic Lookups

Re: Log showing when Diags were generated?

Re: When will Chrony be supported?

Re: When will Chrony be supported?

Warn and Error messages below search bar

Re: Cannot Disable Health Report Features in 8.2.2

Re: Cannot Disable Health Report Features in 8.2.2

Re: Cannot Disable Health Report Features in 8.2.2

Re: Cannot Disable Health Report Features in 8.2.2

Re: Dokumentation for "Health of Splunk Deployment...

Re: Upgrade Splunk 8.0.0 to 8.2

distsearch.conf on Search Head cluster

Re: Splunk-Perfmon crash errors in forwarder 7.2.9

Re: REGEX in transform.conf using "^" to specify t...

Re: ERROR ExecProcessor - message from "/opt/splun...

Re: ERROR ExecProcessor - message from "/opt/splun...

Join the Conversation