We are planning for a migration of hot drives to faster disk for all indexers. The current plan is below, and I wonder if it makes sense, because the plan applies maintenance mode for each indexer, and I grew up with using maintenance mode for the entire operation. Any thoughts? The current plan -   Event monitoring Suspend monitoring of alert on the indexers (Repeat following for each indexer, one at a time) Splunk Ops: Put Splunk Cluster in Maintenance Mode Stop Splunk service on One Indexer VM Ops vMotion the existing 2.5TB disk to any Unity datastores Provision new 2.5TB VM disk from the VSAN datastore Linux Ops Rename the existing hot data logical volume "/opt/splunk/hot-data" to "/opt/splunk/hot-data-old" Create a new volume group and mount the new 2.5TB disk as "/opt/splunk/hot-data" Splunk Ops Restart Splunk service on indexer Take Indexer Cluster out of Maintenance Mode Review Cluster Master to confirm indexer is processing and rebalancing has started as expected Wait a few minutes to allow for Splunk to rebalance across all indexers (Return to top and repeat steps for next indexer) Splunk Ops: Validate service and perform test searches Check CM Panel - -> Resources/usage/machine (bottom panel - IOWait Times) and monitor changes in IOWait Event monitoring Enable monitoring of alert on the indexers     In addition, Splunk PS suggested to use -   splunk offline --enforce-counts     Not sure if it's the right way since it might need to migrate the ~40TB of cold data, and would slow the entire operation ... View more

We ran into this known issue with the AD servers having indexing delays of a couple of days when enabling evt_resolve_ad_obj. What confuses us is the fact that a UF restart backfills days of missing security data, and since the restart, we can have a week where there are no delays. Why does the restart manage to do this backfill? ... View more

With MLTK, when looking at accumulated runtime, the outliers are detected cleanly (three out of three spikes), whereas with the anomaly detection app, only two of the three spikes are detected (along with one false positive, even at medium sensitivity). The code generated by the MLTK is as follows -   index=_audit host=XXXXXXXX action=search info=completed | table _time host total_run_time savedsearch_name | eval total_run_time_mins=total_run_time/60 | convert ctime(search_*) | eval savedsearch_name=if(savedsearch_name="","Ad-hoc",savedsearch_name) | search savedsearch_name!="_ACCEL*" AND savedsearch_name!="Ad-hoc" | timechart span=30m median(total_run_time_mins) | eval "atf_hour_of_day"=strftime(_time, "%H"), "atf_day_of_week"=strftime(_time, "%w-%A"), "atf_day_of_month"=strftime(_time, "%e"), "atf_month" = strftime(_time, "%m-%B") | eventstats dc("atf_hour_of_day"),dc("atf_day_of_week"),dc("atf_day_of_month"),dc("atf_month") | eval "atf_hour_of_day"=if('dc(atf_hour_of_day)'<2, null(), 'atf_hour_of_day'),"atf_day_of_week"=if('dc(atf_day_of_week)'<2, null(), 'atf_day_of_week'),"atf_day_of_month"=if('dc(atf_day_of_month)'<2, null(), 'atf_day_of_month'),"atf_month"=if('dc(atf_month)'<2, null(), 'atf_month') | fields - "dc(atf_hour_of_day)","dc(atf_day_of_week)","dc(atf_day_of_month)","dc(atf_month)" | eval "_atf_hour_of_day_copy"=atf_hour_of_day,"_atf_day_of_week_copy"=atf_day_of_week,"_atf_day_of_month_copy"=atf_day_of_month,"_atf_month_copy"=atf_month | fields - "atf_hour_of_day","atf_day_of_week","atf_day_of_month","atf_month" | rename "_atf_hour_of_day_copy" as "atf_hour_of_day","_atf_day_of_week_copy" as "atf_day_of_week","_atf_day_of_month_copy" as "atf_day_of_month","_atf_month_copy" as "atf_month" | fit DensityFunction "median(total_run_time_mins)" by "atf_hour_of_day" dist=expon threshold=0.01 show_density=true show_options="feature_variables,split_by,params" into "_exp_draft_ca4283816029483bb0ebe68319e5c3e7"   And the code generated by the anomaly detection app -   ``` Same data as above ``` | dedup _time | sort 0 _time | table _time XXXX | interpolatemissingvalues value_field="XXXX" | fit AutoAnomalyDetection XXXX job_name=test sensitivity=1 | table _time, XXXX, isOutlier, anomConf     The major code difference is that with MLTK, we use -   | fit DensityFunction "median(total_run_time_mins)" by "atf_hour_of_day" dist=expon threshold=0.01 show_density=true show_options="feature_variables,split_by,params" into "_exp_draft_ca4283816029483bb0ebe68319e5c3e7"   whereas with the anomaly detection app, we use -   | fit AutoAnomalyDetection XXXX job_name=test sensitivity=1 | table _time, XXXX, isOutlier, anomConf     Any ideas why the fit function uses DensityFunction vs AutoAnomalyDetection parameters, and why the results are different? ... View more

We have a case of a delay of an hour for a certain index that happened last week, while the indexing delays are normally up to half a minute. I'm struggling with the parameters for the MLTK to capture these specific cases as outliers. Any ideas how to set it up correctly? It’s the tolerance that seems to be affected by the spike itself. ... View more

In Step 2 "Add the Dataset" of "Create Anomaly Job" within the Splunk App for Anomaly Detection, when running the following SPL, we get the warning-        index=wineventlog_security | timechart count "Could not load lookup=LOOKUP-HTTP_STATUS No matching fields exist."         What can it be? We use the following versions - Splunk App for Anomaly Detection - 1.1.0 Python for Scientific Computing  - 4.1.2  Splunk Machine Learning Toolkit  - 5.4.0 ... View more

We have a large (~500 line) report being used to calculate CVE scores and fill a summary index daily, with vulnerabilities from Qualys as the initial input that gets enriched. One of the fields in the ouput is called ICT_ID, and this is supposed to match on the vulnerabilities from Qualys according to a lookup CSV file. If the vulnerability matches, it gets a corresponding ICT_ID, otherwise this field is NULL. The issue is that for our lookup file, there are no unique primary keys. QID (Qualys vuln ID) is the closest thing to a PK in the lookup, but there are multiple rows with the same QID and other fields like IP and host which differ. The requirement for matching a vulnerability to the ICT list is two-fold: 1) the QID must match, but also must match 2) *any* of the following (host, IP, app) *in that order of precedence* ----------------- The following code was implemented, but it seems to only match on the first matching instance of QID in the lookup, which usually breaks the rest of the logic for the other fields which should match.       ``` precedence: (host -> ip -> app -> assetType) && QID ``` | join type=left QID [ | inputlookup ICT_LOOKUP.csv | rename "Exp Date" as Exp_Date | rename * as ICT_LOOKUP-* | rename ICT_LOOKUP-QID as QID ] ```| rename fields from lookup as VM_ICT_Tracker-<field>``` | eval ICTNumber = case( like(entity,'ICT_LOOKUP-host'), 'ICT_LOOKUP-ICTNumber', like(IP,'ICT_LOOKUP-ip'), 'ICT_LOOKUP-ICTNumber', like(app,'ICT_LOOKUP-app'), 'ICT_LOOKUP-ICTNumber', like(assetType,'ICT_LOOKUP-assetType'), 'ICT_LOOKUP-ICTNumber', 1=1,"NULL" ) | rename ICTNumber as ICT_ID       ... View more

We need to call a search via the API and return a link to a report, produced by this call. Is it doable? So, I have something like the following that returns the result set as json, and the requirement is to return it as a link to a report -   curl -k -u 'moogsoft_smart_triage_user:xxxxxx' https://<host>:8089/servicesNS/moogsoft_smart_triage_user/search/search/jobs/export -d search="| savedsearch smart_triage_api_test INC=INCxxxx DeviceType=TestDeviceType" -d output_mode=json -d preview=false   ... View more

We don't like to give blind access to the MC, to temporary Splunk contractors. They need something like the following -  - Search performance and distributed search framework - Indexing performance - Operating system resources usage - Splunk KV store performance - Index and volume usage - Forwarder connections and network performance   Is there a way to access this information from outside the MC? API or any other way? ... View more

We have indexes with retention of a year each, and when looking at _audit, it's pretty obvious that the queries against this index are mostly either daily or weekly. We would like to present to management the "waste" of storage (and maybe potentially making a case for using cheaper storage for older data, ie. older than 3 months). Is anybody aware of any visualization that does it? We tried to combine the information from audit with the retention information that we got via REST, however we don't have a cohesive view that can be appealing to management. Any ideas? ... View more

I need a query that will provide the earliest date for data within an index as well as the indexer it is stored on, specifically looking for which indexes are storing data for one year or more and on which indexers they are being stored on. Any ideas? ... View more

A colleague of mine uses the following dedup version: | strcat entity "-" IP "-" QID "-" Port "-" Tracking_Method "-" Last_Detected Key | dedup Key And I grew up with | dedup entity IP QID Port Tracking_Method Last_Detected One caveat is Tracking_Method doesn't always exist. So which version is better? ... View more

Is there a way to pass a parameter to a report when calling it via -    curl -u user:password -k https://<api_server>:8089/servicesNS/nobody/<app_name>/search/jobs -d "search=savedsearch <savedsearch_name>" -d exec_mode=oneshot -d count=10000   ... View more