it seems easier to do the opposite : - ask for your splunkd port to be opened on splunk cloud  - then use a REST api to run a remote search from on-prem against splunkcloud search-head, and output the result in a file. see https://docs.splunk.com/Documentation/Splunk/latest/Search/SearchwithSplunkWeb,CLI,orRESTAPI   ... View more

depending on how your sourcetype is defined, the events may be kept in a block, broken per line, broken on new timestamps etc... Also if you are using. "INDEXED_EXTRACTIONS" on the forwarder, the events ma be parsed on the forwarder, not on the indexers (you need to customize the props.conf on the forwarder then) if you want to create a working sourcetype props.conf, I would recommend to pick your sample of data, and try to ingest it on a SH, in the UI, using the editor to improve the parsing. At the end, copy past the props.conf, and deploy it on the indexers (or forwarder if you use special parsing on the forwarder) https://docs.splunk.com/Documentation/Splunk/latest/Data/Setsourcetype ... View more

Could it be a role issue,  check if your user is member or inherit from the role itoa_user (or itoa_analyst, or itoa_admin) ? ... View more

If you have to use those 2 distinct licenses on separate license masters, you can reach splunk support, they can help split the license. 2 new smaller licenses will be issued, and the old one will be revoked. ... View more

You may be are comparing Apples and Oranges. metrics.logs gives a rough estimate of the data ingested per index.  license_usage.log gives the reported volume of data counted on the license. (excluding internal indexes, summary sourcetypes etc...). Also the licensing timestamp is "when the license-master received the usage report from the indexers" (it may be backlog if the Indexers were not able to connect)   A side remark : when you look at logs, always add a * at the end of the source, in case the logs rotated, and some event are reported under the rotated log source. (source=license_usage.log* can find data license_usage.log.1 )   ... View more

Here is the Guide you need : https://docs.splunk.com/Documentation/Splunk/8.0.5/InheritedDeployment/Introduction     Now you can figure much by just looking at the internal logs.   indexers : look at the splunk_server field, it tells you on which indexer the data is stored. index=_internal | stats dc(host) count by splunk_server heavy forwarders / forwarders : look at the metrics logs, and identify who it sending data to who , and look at the fwdType field. index=_internal source=*metrics.log* group=tcpin_connections fwdType=* | stats dc(hostname) values(version) values(os) by fwdType   ... View more

Addendum : Keep in mind that other issues can lead to multiple grouping of the same notables - Rules engine backfill issues -> upgrade to recent versions of ITSI (4.4.4 or more) - "Tsidx reduction" core bug on the indexers, causing old notables to be rediscovered over and over by the realtime searches. -> see workaround ITSI-4606 https://docs.splunk.com/Documentation/ITSI/4.4.0/ReleaseNotes/Knownissues >Workaround: This issue occurs because the indexed realtime search returns events over and over from buckets that use tsidx reduction. Disable tsidx reduction on the itsi_tracked_alerts and itsi_summary indexes and rebuild all old buckets on these indexes. ... View more

If you know the exact origin of the data (by example host, sourcetype, fields), then you can have a search that look for that data over a recent timerange, (stats count) and trigger if there are no results at all.   But if you are actually searching dynamically over several origins  (| stats count by host). Then you need to compare to a list, or older data to notice that once origin is missing. - It could be am hardcoded count, a lookup you maintain, it could be a meta data search, it could be a subsearch with a different timerange.... - or I could be a search that is looking back on a longer timerange, and do a ( | timechart count by host) Or ( | bucket _time span=1h | stats count by _time host), and has some logic to check if the recent intervals are zero). But this may require more longer/expensive searches each time, so it's not good for a frequent alert.  ... View more

"For the two indexes.conf volume settings below - would one take precedence over the other if they use the same path?"   I would recommend not use the same paths for volumes, and not to include one path into another. (and not include indexes paths into other indexes, or mix in the same folders volumes paths and non volume paths..) Otherwise, the splunk measure of the disk used size will be all incorrect, and you may experience data frozen faster. And you probably should see warnings in splunkd.log when you start the indexers, about risks on the paths conflicts. ... View more

If the splunk service is down on the forwarder, then it cannot reach the deployment server, hence cannot receive the app update and restart request. You may have to actually start splunk service directly on the client (with a CLI command or any remote admin management tool)  ... View more

The SAI-ITSI integration has a wizard that asks once, if you want enable the entity migration. If the wizard did not run, or was skipped, there is no way to redo it on a SHCluster. Because the setting is ultimately a modular input to enable, but the "Inputs manager" is not accessible in the UI on a SHCluster. The workaround is to enable the modular input in the inputs.conf directly, from the deployer. On the deployer shcluster apps folder, find the app splunk_app_infrastructure,  create a local folder with an inputs.conf, and add this enabled modular input, [em_entity_migration://job] disabled = 0 Then push it on all the SHC peers from the deployer.  ... View more

ITSI 4.5.* is available for splunkcloud only.  Some of the features change are not for on-premises deployments yet. So currently (2020-06-09) you can study the documentation for version 4.4.*  if you are on-prem. And later see what is the next version for (cloud, on-prem or both) ... View more

Assuming your regex is correct. The way to change the index at index time (with props/transforms) is : on the first splunk instance that will parse the data. It means usually on the indexers. But if you have intermediary heavy forwarder, it will have to be on the first one of the forwarding chain. And if your data is parsed by the monitoring forwarder (look for INDEXED_EXTRACTIONS setting in props), then the parsing is happening on the first forwarder (could be the UF) ... View more

Having a lot of jobs artifacts in the dispatch folder, is directly proportional to the number of search jobs, and the time to live of the search artifact (dispatch.ttl) ad-hoc search jobs artifacts usually stay in the dispatch 10 minutes. scheduled search jobs stay for 2 times the scheduling interval (a job running every 4 hours stay on the dispatch for 8 hours) scheduled search jobs that triggered an alert action ( like tracking or alert email) will stay in dispatch for 24h. search jobs shares will be kept for 7 days. realtime searches stay as long at the search is running Knowing that you can look at your dispatch folder and figure what constitutes the mass of job artifacts. https://docs.splunk.com/Documentation/Splunk/latest/Search/Dispatchdirectoryandsearchartifacts Options to reduce the TTL are : For searches - edit the dispatch.ttl in savedsearch.conf. For a particular search, or in the generic settings. - do the same on a per search basis using the UI > searches&reports > advanced edit for alerts - reduce the ttl in alert_actions.conf - or reduce the number of unnecessary alerts. ... View more

There are no definitive answers, it depends on your search time, the server concurrency load, and the desired time to be notified of alerts or the frequency. Example : you run 2 correlation searches running every 15 minutes, looking back at 15 minutes. If the searchest take 10 seconds to execute, then it will not count on your search concurrency as 2 jobs for 10 seconds every 15 min. If the searches take 20 minutes to run, then it will persistently count at 2 job concurrencies, plus every 15 min, you will have an overlap, and it will count at 4... If you have an alert condition happening t=0, then the correlation search runs as t=10m, then it will take you 10 minutes to be get a notable alert created, and a few seconds to get grouped in an Episode and trigger an action. Changing the frequency will help control the time before you are notified, it may vary based on the type of data you are monitoring, ... View more

Does it work when you run it manually? ITSI runs the KPI search as the role "splunk-system-user", check if that role has the permission to search the indexes A and B. ... View more

They are sourcetype leftover examples of simpler times. They can work on very generic datasets, but if you intend to ingest more specific logs, it's better to create new specific sourcetypes. (or use the ones provided by specific enterprise standard in their apps, and likely compatible with the Common Information Model). The reason is that, if you all many of your events have the same sourcetype, then the day you want to setup a custom filters/field parsing/index time or search time processing, it will apply to a large set of data. While if you apply it to a specific sourcetype, you can control better the scope. ... View more

look at the file mod time, where they created around a splunk restart when the apps were installed? ... View more

if the field uri contains "checkout/orderConfirmation/12598099392" try mysearch | rex field=uri "orderConfirmation\/(?<confirmationNumber)\d+" | table uri confirmationNumber ... View more

We do not see the query referring to a field "total". So maybe is it something coming from a different place ( automatic field extractions, automatic lookups, role search filter ...) ? ... View more