In Splunk ES there is an asset list, "asset_lookup_by_str". This list contains the output from merging asset list input lookups. This merging process is happening "under the hood" in Splunk ES, it's not a normal saved search (even though part of the process seems to be the running of a search with the "entitymerge" command). Typically, over time this merged asset list will collect outdated information, e.g. DNS names no longer in use. Also, with dynamic IPs on assets, this quickly creates strangeness in the list. There is no good out-of-the-box solution for dealing with this, as far as I know. The only solution that is provided by Splunk ES, is to use the "Reset collections" button on the "Asset and Identity management" page, which will delete the whole asset list, and let it be rebuilt with fresh data from your input asset lookups (see attached screenshot). My question is thus, to make sure the asset list in Splunk ES is updated and free of junk, how can you periodically reset the list? One would think that there was a REST endpoint you could use to trigger "reset collections", but I cannot find any. Under are some other possible solutions that I've tested, but can not get to work Deleting the asset list with "outputlookup": | outputlookup "asset_lookup_by_str" Running the command above would indeed delete the list (make it empty). However, the asset list framework in Splunk ES will not "notice" that the list is deleted, and thus not rebuild it. It seems that clicking the "Reset collections" button in the GUI also updates some checkpoint, which triggers a rebuild. I've found no way of manually duplicate this "checkpoint update" behaviour. Deleting the asset list with "outputlookup", and rebuilding it manually with "entitymerge": You might suggest that I could delete the list manually, and rebuild it with the enitymerge command. The search for this is even pre-created and ready for use in the "Asset and Identity management" pages. However, the input asset lookups are hardcoded in the search, meaning that in the future, if you add or remove input asset lookups, the search need to updated as well. This does not seem like a good long term solution. Automatically updating the search based on the input asset lookups seems to require custom scripts, and would be complicated. Deleting the asset list with the "identdelete" command: | identdelete collection="assets_by_str" The "identdelete" is a undocumented command in Splunk, but looking at the internal logs when using the "Reset collections" button, you can find it. It seems from the name of the command that it might be useful, but I've found that it in fact does nothing. See example of the ootb identdelete search that is run when clicking "Reset collections" below (in this case for the asset CIDR list, but it's the same). | `add_entity_source("frothly_aws_assets_2018","frothly_aws_assets_2018")` | `add_entity_source("frothly_assets_2018","frothly_assets_2018")` | table "_source","cim_entity_zone","bunit","category","city","country","dns","ip","is_expected","lat","long","mac","nt_host","owner","pci_domain","priority","requires_av","should_timesync","should_update" | `make_ip_cidr` | inputlookup append=T "asset_lookup_by_cidr" | entitymerge "asset" | outputlookup append=T "asset_lookup_by_cidr" | head 1 | identdelete collection="assets_by_cidr"  Manually clicking the "Reset collections" button: This is not a feasible solution.   ... View more

Hello, We have recently upgraded Splunk Enterprise, on-prem. We now have a button for AI Assistant next to the search bar. As our Splunk solution is not connected to cloud, and is not going to be so, the button is useless. When clicked it says "Contact your admin to activate Splunk AI Assistant", so now we have many users asking us how to activate it for them, which we are not able to do. To avoid confusion, and unnecessary requests, the button should just be removed. Is this possible? ... View more

We have an index with a ton of data. A new use for the data has emerged, so now we want a longer retention time on some of the data in the index. We don't want to simply increase the retention time on the index, because the storage cost is too high. We want to create a new index with a longer retention, pick out the events we need, and copy them to the new index. This is on an indexer cluster. In theory, we could use collect, like this: index=oldindex field=the_events_we_need | collect index=newindex However, because the index is too big, we're having problems running this search. Even though we run it bit-by-bit, still we end up missing events in the new index. Could be due to performance or memory limits, or bucket issues. Is there a better and more reliable way of doing this? ... View more

We have different lookup inputs into the Splunk ES asset list framework. Some values for assets change over time, for example due to DHCP og DNS renaming. When an asset gets a new IP due to e.g. DHCP, the lookup used as input into the asset framework is updated accordingly, but the merged asset lookup "asset_lookup_by_str" will contain both the new and the old IP. So the new IP is appended on the asset, it's not replacing the old IP. Due to "merge magic" that runs under the hood in the asset framework, over time this creates strange assets with many DNS names and many IPs. My question is, how long are asset list field values stored in the Splunk ES asset list framework? Are there any hidden values that keep track of say an IP, and will Splunk eventually remove the IP from the asset in the merged list? Or will the IP stay there forever, and these "multivalue assets" will thus just grow with more and more DNS names and IPs until the mv field limits are reached? And, if I reduce the asset list mv field limits, how does Splunk prioritize what values will be included or not? Does the values already on the merged list have priority, or does any new values have priority? Tried looking for answers in the documentation but could not find answers on my questions there. Hoping someone will share some insights here. Thanks! ... View more

Hi, How do Splunk ES create incidents from notable events? I'm aware that a correlaction search in Splunk ES creates a notable event in the "notable" index, but exactly how does it get from here to the "Incident Review" dashboard in Splunk ES? As far as I know the incidents exists in a KV store collection, and I would then assume that there is some scheduled job that take notable events from the "notable" index, and puts them in the KV store collection. The reason I'm asking is that we are missing incidents in our "Incident Review" dashboard, but the corresponding notable events exists in the notable index. So it looks like the "notable event to incident" job has failed somehow. Is this documented somewhere in more detail? ... View more

Hi, I have an accelerated datamodel. This datamodel have a lookup field based on a KV store lookup, that is, the datamodel takes another calculated numerical field and use the lookup to get a string value for that number. However, after some testing I see that my lookup field the accelerated datamodel does not have the correct value. I cross-checked by using the lookup directly on the raw data, and then it workes fine. I also double checked the expression for the lookup field, making sure it actually uses the correct values. The interesting thing is that the lookup field is indeed calculated, so the  KV store lookup works, and it get a "hit" in the lookup, but the row that is returned from the lookup is wrong. It seems to me that this could be some kind of bug, e.g. related to memory on the search head or something, that causes the lookup field accelerated datamodel to bug out. The KV store lookup is quite big. Anyone else have experienced this, and knows a way around? My lookup field in the datamodel:          { [-]            calculationID: 123456            calculationType: Lookup            comment:            editable: true            lookupInputs: [ [-]              { [-]                inputField: mynumber                lookupField: mynumber              }            ]            lookupName: number_to_string            outputFields: [ [-]              { [-]                comment:                displayName: mystring                editable: true                fieldName: mystring                fieldSearch:                hidden: false                lookupOutputFieldName: mystring                multivalue: false                owner: mydatamodel                required: false                type: string              }            ]            owner: mydatamodel          }  The field "mynumber" is based on an eval expression in the datamodel (which is defined before the lookup field):          { [-]            calculationID: 456789            calculationType: Eval            comment:            editable: true            expression: 'mynumber_raw'            outputFields: [ [-]              { [-]                comment:                displayName: mynumber                editable: true                fieldName: mynumber                fieldSearch:                hidden: false                multivalue: false                owner: mydatamodel                required: false                type: string              }            ]            owner: mydatamodel          }   ... View more

We're looking over our environment for potential safety flaws. One question that came up is whether an admin-user is available by default on Splunk Universal Forwarders (UF). I'm not thinking about the user the UF runs as on the OS, but an admin user on the application layer. Earlier Splunk Enterprise had a default admin password "changeme". Did this also apply for UFs? How can we make sure that there is no admin users on our UFs, or that if there is, that they have proper passwords?  ... View more