We are building a small Splunk installation in Azure and I'm not sure what the architecture should look like. The client came up with the idea based on the following link - Deploying Splunk on Microsoft Azure. They created an indexer, a search head, and a license server/cluster master. We do need to ingest syslog data from Meraki devices, so I wonder whether we need a heavy forwarder. Any thoughts? ... View more

When ingesting Microsoft Azure data, we see different time formats for different Azure categories, and I wonder how to parse it correctly? Both timezones seem to be UTC. Is the proper approach to set  TZ=UTC and specify in datetime.xml the two formats? { category: NonInteractiveUserSignInLogs time: 2024-07-30T18:02:42.0324621Z . . . } { category: RiskyUsers time: 7/30/2024 1:48:56 PM . . . } ... View more

We see cases where warm buckets are not being moved to cold storage for six weeks, and we wonder how to set it up correctly so they move within two or three weeks. ... View more

We apparently have the StreamWeaver integration in place, but we are not sure how it was implemented as the folks who did it are no longer around. How is it done usually?  Is it a REST API integration? as I see at Connect: Splunk Enterprise  ... View more

We want to add a TA (app) to our indexers at the path /opt/splunk/etc/master-apps by running the command /opt/splunk/bin/splunk apply cluster-bundle My question is if we can deploy an indexer app without a restart of the indexer? The TA we want to deploy is an extension to the nix TA, and all it does is run some simple bash scripted inputs.     ... View more

We are in the midst of a migration from physical servers to virtual servers, and we wonder if stopping Splunk is mandatory in order to perform the Cold Data migration or if there’s a workaround to this and this can be safely done without stopping Splunk. ... View more

Since we are in early stages of using Splunk cloud, we don't define props.conf as part of the onboarding process, and we introduce props for a certain sourcetype only when parsing goes off. So for a certain sourcetype, line-breaking is off, and when looking for the sourcetype on the indexers (via support) and on the on-prem HF where the data is ingested and I cannot find the props for this sourcetype. So I wonder is it possible to have no sourcetype anywhere for a particular source? ... View more

With syslog-ng we hit all kinds of limitations from the inability to support TCP, to the inability to write fast enough to disk, and therefore losing vast amounts of UDP data, struggling to have the various F5 LBs to distribute the data evenly to the syslog servers behind the F5s... and all of this led to Kafka as a potential solution. Does anybody use Kafka effectively instead of syslog-ng? By the way, we did look at Splunk Connect for Syslog (SC4S) without much luck. ... View more

I just realized that the NIX TA is being deployed to our forwarders via the deployment apps, to the indexers via the master apps and to the SHs via the SH apps. It was a surprise for me to realize that the TA is not being deployed to the deployment and deployer servers, the license master and the cluster master. And therefore, how can the TA be deployed to all Splunk servers? ... View more

We are in the midst of a virtualization project, and we are looking for a way to sanity check all the different components. I know that the MC does some of it, but I’m not sure if it covers all aspects. I’m thinking about scripted input, and a dedicated dashboard to monitor and verify all the settings. Do you have any other suggestions, by any chance? ... View more

We use the free version of syslog-ng, and recently we had a requirement to have TLS on top of TCP, and we don't have the knowledge to implement it. Therefore we wonder whether anybody has migrated from syslog-ng to SC4S to assist with more advanced requirements such as TCP/TLS? https://splunkbase.splunk.com/app/4740 If so, what lessons learned and motivations to do that? ... View more

We are in the process of virtualizing our environments and then we are facing the question of whether to use multiple cluster masters or to have fewer cluster masters that serve more indexers each. However, we don’t know how to go about it. Therefore the question, what are the scalability rules for a cluster master? ... View more

We are planning for a migration of hot drives to faster disk for all indexers. The current plan is below, and I wonder if it makes sense, because the plan applies maintenance mode for each indexer, and I grew up with using maintenance mode for the entire operation. Any thoughts? The current plan -   Event monitoring Suspend monitoring of alert on the indexers (Repeat following for each indexer, one at a time) Splunk Ops: Put Splunk Cluster in Maintenance Mode Stop Splunk service on One Indexer VM Ops vMotion the existing 2.5TB disk to any Unity datastores Provision new 2.5TB VM disk from the VSAN datastore Linux Ops Rename the existing hot data logical volume "/opt/splunk/hot-data" to "/opt/splunk/hot-data-old" Create a new volume group and mount the new 2.5TB disk as "/opt/splunk/hot-data" Splunk Ops Restart Splunk service on indexer Take Indexer Cluster out of Maintenance Mode Review Cluster Master to confirm indexer is processing and rebalancing has started as expected Wait a few minutes to allow for Splunk to rebalance across all indexers (Return to top and repeat steps for next indexer) Splunk Ops: Validate service and perform test searches Check CM Panel - -> Resources/usage/machine (bottom panel - IOWait Times) and monitor changes in IOWait Event monitoring Enable monitoring of alert on the indexers     In addition, Splunk PS suggested to use -   splunk offline --enforce-counts     Not sure if it's the right way since it might need to migrate the ~40TB of cold data, and would slow the entire operation ... View more

We ran into this known issue with the AD servers having indexing delays of a couple of days when enabling evt_resolve_ad_obj. What confuses us is the fact that a UF restart backfills days of missing security data, and since the restart, we can have a week where there are no delays. Why does the restart manage to do this backfill? ... View more

With MLTK, when looking at accumulated runtime, the outliers are detected cleanly (three out of three spikes), whereas with the anomaly detection app, only two of the three spikes are detected (along with one false positive, even at medium sensitivity). The code generated by the MLTK is as follows -   index=_audit host=XXXXXXXX action=search info=completed | table _time host total_run_time savedsearch_name | eval total_run_time_mins=total_run_time/60 | convert ctime(search_*) | eval savedsearch_name=if(savedsearch_name="","Ad-hoc",savedsearch_name) | search savedsearch_name!="_ACCEL*" AND savedsearch_name!="Ad-hoc" | timechart span=30m median(total_run_time_mins) | eval "atf_hour_of_day"=strftime(_time, "%H"), "atf_day_of_week"=strftime(_time, "%w-%A"), "atf_day_of_month"=strftime(_time, "%e"), "atf_month" = strftime(_time, "%m-%B") | eventstats dc("atf_hour_of_day"),dc("atf_day_of_week"),dc("atf_day_of_month"),dc("atf_month") | eval "atf_hour_of_day"=if('dc(atf_hour_of_day)'<2, null(), 'atf_hour_of_day'),"atf_day_of_week"=if('dc(atf_day_of_week)'<2, null(), 'atf_day_of_week'),"atf_day_of_month"=if('dc(atf_day_of_month)'<2, null(), 'atf_day_of_month'),"atf_month"=if('dc(atf_month)'<2, null(), 'atf_month') | fields - "dc(atf_hour_of_day)","dc(atf_day_of_week)","dc(atf_day_of_month)","dc(atf_month)" | eval "_atf_hour_of_day_copy"=atf_hour_of_day,"_atf_day_of_week_copy"=atf_day_of_week,"_atf_day_of_month_copy"=atf_day_of_month,"_atf_month_copy"=atf_month | fields - "atf_hour_of_day","atf_day_of_week","atf_day_of_month","atf_month" | rename "_atf_hour_of_day_copy" as "atf_hour_of_day","_atf_day_of_week_copy" as "atf_day_of_week","_atf_day_of_month_copy" as "atf_day_of_month","_atf_month_copy" as "atf_month" | fit DensityFunction "median(total_run_time_mins)" by "atf_hour_of_day" dist=expon threshold=0.01 show_density=true show_options="feature_variables,split_by,params" into "_exp_draft_ca4283816029483bb0ebe68319e5c3e7"   And the code generated by the anomaly detection app -   ``` Same data as above ``` | dedup _time | sort 0 _time | table _time XXXX | interpolatemissingvalues value_field="XXXX" | fit AutoAnomalyDetection XXXX job_name=test sensitivity=1 | table _time, XXXX, isOutlier, anomConf     The major code difference is that with MLTK, we use -   | fit DensityFunction "median(total_run_time_mins)" by "atf_hour_of_day" dist=expon threshold=0.01 show_density=true show_options="feature_variables,split_by,params" into "_exp_draft_ca4283816029483bb0ebe68319e5c3e7"   whereas with the anomaly detection app, we use -   | fit AutoAnomalyDetection XXXX job_name=test sensitivity=1 | table _time, XXXX, isOutlier, anomConf     Any ideas why the fit function uses DensityFunction vs AutoAnomalyDetection parameters, and why the results are different? ... View more

We have a case of a delay of an hour for a certain index that happened last week, while the indexing delays are normally up to half a minute. I'm struggling with the parameters for the MLTK to capture these specific cases as outliers. Any ideas how to set it up correctly? It’s the tolerance that seems to be affected by the spike itself. ... View more

In Step 2 "Add the Dataset" of "Create Anomaly Job" within the Splunk App for Anomaly Detection, when running the following SPL, we get the warning-        index=wineventlog_security | timechart count "Could not load lookup=LOOKUP-HTTP_STATUS No matching fields exist."         What can it be? We use the following versions - Splunk App for Anomaly Detection - 1.1.0 Python for Scientific Computing  - 4.1.2  Splunk Machine Learning Toolkit  - 5.4.0 ... View more