I setup half a dozen serverclasses leveraging CMDB sourced .csv and whitelist.from_pathname This works great to managing serverclasses with thousands of clients, though I was surprised these serverclasses whitelisted clients would get stale unless I'd issue a reload deploy-server / reload deploy-server -class Couldn't find any flag on the DS allowing whitelist.from_pathname to update whenever the linked .csv would update. Is there any better way ? ... View more

Running on Splunk 8.08 Even though we already have ITSI setup to recognise dozens of different performance related data sources, IT Essentials Learn shows "Use Case Families were not found." on its homepage and doesn't offer any macro configuration  Both of the only 2 scheduled searches the App offers are erroring out "Telemetry: ITE Procedure Stats" is erroring with "Error in 'eval' command: Type checking failed. 'OR' only takes boolean arguments." on the below code snippet | eval [| rest /servicesNS/nobody/splunk_instrumentation/telemetry | table telemetrySalt | format | rex field=search mode=sed "s/[()]//g"] | inputlookup ite_procedure Likewise for "Telemetry: ITE Procedure Usages" is erroring out on | eval [| rest /servicesNS/nobody/splunk_instrumentation/telemetry | table telemetrySalt | format | rex field=search mode=sed "s/[()]//g"] ... View more

I have a requirement to push a subset of universal and heavy forwarders originating data to a third party, for which I enabled a set of HFs for data forwarding alone. This is working fine, as data arrives uncooked to a target syslog-ng . The troublesome part was being asked to ensure the HF resends the data in case the target undergoes maintenance, or has an outage lasting up to 2 days. Considering Persistent Queues don't work over splunktcp streams, is it even an option for me to push uncooked data to the HFs, enabling a standard TCP input (not splunktcp ) with Persistent Queue enabled, say, to 200GB? Never heard of anyone using this approach. Would this work? ... View more

According to https://docs.splunk.com/Documentation/AddOns/released/MSSQLServer/SQLServerconfiguration Audit events get written into a binary file C:\SQLAudit which then gets read via a DB connect query. This is problematic when we need to monitor hundreds of different MSSQL Server DBs and would imply us setting up several HFs just for this tech stack alone (as well as very infrequent ingestion queries which doesn't seem a great fit for ES) Even though writing audit events via Windows Event Log is less performsnt as it implies using the Windows event API rather than outputting to a file I don't see any other option for us other than doing it and fetching via UF This seems to have been the legacy approach for older TA versions https://answers.splunk.com/comments/170572/view.html ... View more

As in, hypothetical scenario If the CM goes down overnight and it takes forever (eg. 10h) for a standby CM to manually takeover/same CM to be recovered from a VM snapshot but all indexers happily stay up (and able to contact each other), is there any chance a scheduled search not finding some primary buckets ? ... View more

How would we ensure data persistence/queuing when using Ryan Faircloth's (or a similar script) method to batch the syslog messages using a script rather than the default one message per POST of syslog-ng's http() output ? Scenario is if there's an 1h network outage between syslog-ng and the HEC HWFs https://www.rfaircloth.com/2017/02/10/building-perfect-syslog-collection-infrastructure/ ... View more

How to correct this SPL to avoid this error index=win EventCode=528 OR EventCode=4624 LogonType=2 | fields Account_Name [ | inputlookup identities_1 | inputlookup append=true identities_2 | inputlookup append=true identities_3 | rename identity as Account_Name | fields Account_Name watchlist | where watchlist = "true" ] Error in 'fields' command: Invalid argument: 'Account_Name=HELP' ... View more

As the default ES DMA schedule is every 5min, and the ACCELERATE_DM_Splunk_SA_CIM*ACCELERATE jobs TTL is 24h, our dispatch directory is filling up 24h TTL with 5min runs means 288 jobs per DM, which is obviously rewarding us with the famous "The number of search artifacts in the dispatch directory is higher than recommended " Is it expected for these jobs TTL to be 24h? I was somehow expecting it to be 2p Ran btool limits list --debug but there isn't any local conf overwriting the defaults NOTE: results obtained from: | rest /services/search/jobs/ | search label=_ACCELERATE_DM_Splunk_SA_CIM_*ACCELERATE_ ... View more

Need some clarification regarding enabling "Accelerate until maximum time" according to the docs "When selected, runs the acceleration search until the maximum time is reached." Does this mean that even if a DM acceleration takes say 2min, if the "Maximum time" is set to 3600 seconds, it will never consider it as finished until it reaches that value? ... View more

When hitting the endpoint /servicesNS/nobody/SA-ITOA/itoa_interface/deep_dive all I can see are the saved deep dives (ie. "is_named": true) How can we access the ID for the non saved ones? The goal is linking a bespoke dashboard to the app/itsi/deep_dive?savedDeepDiveID=$deep_dive_id$ endpoint generated when we click one of the services mapped out by a service analyser and click on "Open all in Deep Dive" ... View more

The event generation was flawlessly working for weeks but went fully quiet until a sole burst on yesterday at 4pm The configuration file was not touched (generation frequency still the same) so what can cause the indexing to stop ? Enabled the following but can only see the events going into the queue but nothing being indexed debug = true verbose = true Taking a look into eventgen_main (there's nothing in eventgen_error ) 2018-08-14 15:06:44 eventgen INFO MainProcess Start '2' generatorWorkers for sample 'test_sample.txt' 2018-08-14 15:06:44 eventgen INFO MainProcess Worker# 0: Put 50 events in queue for sample 'test_sample.txt' with et '2018-08-14 15:00:44.141400' and lt '2018-08-14 15:06:44.141451' 2018-08-14 15:06:44 eventgen INFO MainProcess Worker# 1: Put 50 events in queue for sample 'test_sample.txt' with et '2018-08-14 15:00:44.141400' and lt '2018-08-14 15:06:44.141451' 2018-08-14 15:06:44 eventgen INFO MainProcess Worker# 0: Put 50 events in queue for sample 'test_sample.txt' with et '2018-08-14 15:00:44.141400' and lt '2018-08-14 15:06:44.141451' 2018-08-14 15:06:44 eventgen INFO MainProcess Worker# 1: Put 50 events in queue for sample 'test_sample.txt' with et '2018-08-14 15:00:44.141400' and lt '2018-08-14 15:06:44.141451' 2018-08-14 15:12:44 eventgen INFO MainProcess Start '2' generatorWorkers for sample 'test_sample.txt' Looking at splunkd, 08-14-2018 15:19:41.588 +0000 INFO LicenseUsage - type=Usage s="/opt/splunk/var/log/splunk/test_service.log" st=test_service_log h="ip-172-31-36-143" o="" idx="default" i="EAE584D7-DBF7-4B6F-819B-36BAD9EEE258" pool="auto_generated_pool_enterprise" b=84 poolsz=53687091201 08-14-2018 15:19:41.588 +0000 INFO LicenseUsage - type=Usage s="/opt/splunk/var/log/splunk/test_service.log" st=test_service_log h="ip-172-31-36-143" o="" idx="default" i="EAE584D7-DBF7-4B6F-819B-36BAD9EEE258" pool="auto_generated_pool_enterprise" b=84 poolsz=53687091201 08-14-2018 15:20:42.620 +0000 INFO LicenseUsage - type=Usage s="/opt/splunk/var/log/splunk/test_service.log" st=test_service_log h="ip-172-31-36-143" o="" idx="default" i="EAE584D7-DBF7-4B6F-819B-36BAD9EEE258" pool="auto_generated_pool_enterprise" b=172203 poolsz=53687091201 Can you give me some pointers on what may be happening ? ... View more