Hi, Difficult question... Whe have some problems with search performance. Looking at the job inspector i noticed within the slow jobs the command.search.kv is taking a lot of time.  What is this? And where is this part of the search-command executed (indexer or search-head)? I notice especialy wineventlogs are taking a lot of this kv time. I created a blank SH, no apps at all, timed some searches with some different indexes and installed some different apps. I noticed when this command.search.kv takes more time. Sometimes this is correct in relation to the app/event match if looking at the props.conf.  Turning the right app  off makes this command.search.kv decrease a lot to almost zero. But with winevents.. no go.. it stays high.  Also even without the fieldextracts etc installed on this blank SH, most fields are extracted. If those field were extracted at index time.. i can imagine there wil be no command.search.kv time wasted (wild guess). does the indexer extract these fields at search time (strange strange) and wil this be the command.search.kv?? So is it possible this command.search.kv also run's on the indexers? And so.. does this lookup / field extraction cost most off the time?   Thanks in advance greets Jari     ... View more

L.s.,   At our company we have multiple heavy forwarders. Normaly they talk to the central license manager, but for migrtation reason whe have to get them talking to themself. So a forwarder license is in order i think. looks like an easy process. I did below ./splunk edit licenser-groups Forwarder -is_active 1 this will set in /opt/splunk/etc/system/local/server.conf  the settings below: [license] active_group = Forwarder and [lmpool:auto_generated_pool_forwarder] description = auto_generated_pool_forwarder quota = MAX slaves = * stack_id = forwarder Have to set by myself master_uri = self restart the server and it looks like a go. When i do this on every heavy it will give the error in _internal Duplicate license hash: [FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFD], also present on peer . And Duplicated license situation not fixed in time (72-hour grace period). Disabling peer.. Why?? Is it realy going to disable the forwarders when it uses the forwarder.license? What to do about it, where did i go wrong?   Thanks in advance   greetz  Jari ... View more

Hi, We have two indexes wich are stuck in fixeup task.  Our environment exist off  some indexing peers  wich are atached to smartstore.   This mornig there is a warning no sf and rf is met. Two indexes are in this degraded state. Checking the bucket status there are two buckets from two different indexes whish doesn't get fixed. Those buckets are mentioned in the search factor fix, replication factor fix and generation. The last has the notice "No possible primaries". Searching on the indexer which is mentioned in the bucket info it says: DatabaseDirectoryManager [838121 TcpChannelThread] - unable to check if cache_id="bid|aaaaaa~183~839799B0-6EAF-436C-B12A-2CDC010C1319|" is stable with CacheManager as it is not present in CacheManager and ERROR ClusterSlaveBucketHandler [838121 TcpChannelThread] - Failed to trigger replication (err='Cannot replicate remote storage enabled warm bucket, bid=aaaaaa~183~839799B0-6EAF-436C-B12A-2CDC010C1319 until it's uploaded' what can be wrong, and what to do about it?   Thanks in advance Splunk enterprise v9.0.5,  on premisse smartstore. ... View more

Hi I am writing the implementation document for Splunk on Nutanix.  Thinking about backup for disaster recovery, data on Nutanix object store (so Smartstore), and the fact that the Nutanix object store is WORM. I like the deployment to be protected against ransomware attacks. Putting all the data in a backup is not possible because the size is too big and changes too fast. I see that the Nutanix object store is WORM, so data could not be altered by ransomware.. so far so good. But the data on the indexers in the cache itself? The hot data is not protected with WORM, I think that's  the way it works ( must be some downside somewhere) But the warm data in the cache? Suppose there is a ransonware attack.. it would like to change the object store but fails. But it will (i think) change the "warm"  data in the cache on the indexers. There is a difference by now between two of the same files (in cache and object store). Ans possible even on the different indexers... What will splunkd do?  Is it the way I tell it above? Pls. give me your opinion 🙂 greetz jari ... View more

Hi, I was looking for the retention period for my index main and _internal and noticed the data is outide our default frozenTimePeriodInSecs. Standard this is 555 days. I went searching for the buckets which are outside the normal time with the searches from this answer: https://community.splunk.com/t5/Getting-Data-In/bucket-retention-and-frozenTimePeriodInSecs/m-p/116365 and to my surprise.. the buckets are al hot.. and way old.  What bothers me are those buckets are all in folder called hot_quar_v1_xxxx and the normal folders starting with previous xxxxx from the quar folder. What to do with this data? I want to reduce the frozenTimePeriodInSecs from 555 days to say 30 days.. but it doesn't seems to work this way if there are buckets as old as 100961834 secs in those hot_quar folders. Is there a way to clean those folders and set the frozenTimePeriodInSecs?   thanx in advance Jari ... View more

Hi, some questions... Last weekend we've got an error on the indexers. It is a multisite indexers with 6<>6 indexers (each site 6 indexers).  Some indexers went down and the data storage went sky high. Stil not sure why. But when we started the indexers which where down, the data storage went partly back on nomal except one indexer. I noticed a lot off excess buckets...  very very much. I started removing these buckets,  but it stopped on one point and never went further with cleaning.  Could this be because of the data part of this one indexer is full (it is at this moment in automatic detention state). I don't see the activity on the cluster master, so it seems it is finished.. but i can't start a new action to clean, is says "Previously scheduled Remove Excess Buckets is running".  I tried a rolling restart (in maint mode), but it doesn't allow because of the "remove excess buckets is running".. How can i stop this "Previously scheduled Remove Excess Buckets is running"  ? thanks in advance.. ... View more

L.s.,   At our company we deploy the Windows UF on all the vdi machine's. For security reason we make use of the security policy where powershell is prohibited to run. On a vdi machine with the command Get-ExecutionPolicy -List | Format-Table -AutoSize i get below: Scope ExecutionPolicy ----- --------------- MachinePolicy Undefined UserPolicy Undefined Process Undefined CurrentUser Undefined LocalMachine Restricted The last part is where the problem is.. it restricts the local system account from executing the scripts in the universal forwarder bin folder.  It gives an error for the splunk-powershell.ps1 script and then consumes a lot of cpu.   What can we do to use the scripts and use the secutiry policy? The policy is a demand in our company, so we can't dump that one.   Thanx in advance.. ... View more

Hi, we have got a inputs.conf with : [monitor:///home/.../.bash_history] disabled = 0 crcSalt = <SOURCE> whitelist = \.bash_history$ Just to monitor the .bash_history file.  But when i look at "./splunk list monitor"  it list every file in the /home/... folders.  Besides that.. the splunkd process just uses much cpu. (no wonder with so many files in the "list monitor" i think). Why is the splunkd on the universal forwarder monitoring every file in the /home/... folders while all he has to do is check .bash_history? What am i doing wrong with this input?   thanks in advance Jari p.s. Splunk version 8.1.3 ... View more

L.s., I want to get the latency from the input from a forwarder to an index. So whe use the app Meta_woot. It creates an inputlookup file meta-woot. In this file are the latest in-time and host names and index names. So far so good. Next is to use this file for calculating  if a host is late or recent or delayed. Those searches are in the app and works fine. But i want a little extension, i want a table with the indexes as leading, and then calculate (by index) the percentage recent/late host, and sum it a one outcome (per index) So far the theory, now my tries. I used below serach. | inputlookup meta_woot where index=* | eval convert_late=(1440*60) | eval convert_delayed=(60*60) | eval last_time=(now()-recentTime) | eval last_time_indexed=case(last_time < convert_delayed, "Recent", last_time > convert_late, "Late", last_time > convert_delayed, "Delayed") | eval compliant_host=if(last_time_indexed="Recent", "1","0") | stats count(compliant_host) as chost by index, compliant_host This gives me a result where the outcome has split into indexname vs compliant_host and chost index compliant_host chost main 0 11 main 1 123 msad 1 6 nmon 1 5 openshift 1 1 temp_log 1 1 wineventlog 1 2   Now the question, how do i calculate the percentage for index main ( (123+11)/11) so i get an percentage value. How do i calculate with values after a stats command?? Pls help Thanx in advance greetz Jari ... View more