I've found something in de search.log from the indexer peer itself. In the non ES search i found: INFO SearchOperator:kv [2700279 StreamSearch] - Trying to disable extractor name="wel-message" INFO SearchOperator:kv [2700279 StreamSearch] - Disabling extractor name="wel-message" - it does not extract any of the required fields   And on the ES search i found only: INFO SearchOperator:kv [2495326 StreamSearch] - Trying to disable extractor name="wel-message" Not disabling this extraction (and multiple other extractions. So it seems that in fast mode not all extractions are disabled on the ES SHC. Question is why.. ... View more

Yep... same search. I am going the search properties way now, Dirst thing i noticed in the results:   ES SHC: eventFieldCount 7 non ES: eventFieldCount 6 greets Jari     ... View more

Hi @PickleRick    The "strange" thing is.. both searches are in fast mode... even the one which gives the Message field. I can't find where this field is extracted during the search, and it isn't a indexed field. (if it was, the fast mode should have showed it on both SHC).   greetz Jari ... View more

Hi @gcusello    I can understand the same machine. but about time the search itself has the time in it ( earliest=1761217200 latest=1761220800 ) so that must be no problem. Both searches are run from Search.. so i think same app And indeed the ES has much more extra overhead, and maybe there is something for the "Messages' field. But the question is in fact what can cause the "Messages" field to apear in the interesting fields. after that i will look further for the slow searches. greets Jari ... View more

Hi @livehybrid  Thanx for the quick response. This fields.conf is new for me, but /opt/splunk/bin/splunk btool fields list gave no Messages field in the list with fields.    So " Field Discovery is somewhat limited" , but to which degree ?   greets   Jari ... View more

A search with specific  start/end date (for the test always the same) took 223.179 secs.. from which 117.82 secs.  It always seems the time with this command.search.kv is almost half the time of search duration. We have a clustered site. 12 indexers and smartstore. First bvlame was smartstore.. but the buckets are in cache so no smartstore anymore. The question is why are the fields extracted (from wineventlogs) when there are no fieldextractions in the apps (or are winevents extracted buy default because the server is "vanilla"). It seems to cost a lot off time to extract those. ... View more

I just found it in a few files inside the : ./apps/splunk_monitoring_console/lookups/hwf-list.csv ./apps/splunk_monitoring_console/lookups/dmc_forwarder_assets.csv ./apps/splunk_monitoring_console/lookups/dmc_forwarder_assets.csv.c didn't removed them yet. The fact is we are going to rebuild the dmc/lm in a matter of weeks and wil see if these errors wil appear again. But i think they won't appear again. Until now it doesn't seem to matter, it all works great. grts jari ... View more

I have a heavy's without master_uri and Manager_uri. They are luckely working okay besides the error. In etc/licenses is only download-trial folder. No forwarder.license ... View more

Thanks for the response Paul.. I removed the master_uri. I can understand why, it is now manager_uri see below: /opt/splunk/etc/system/local/server.conf [license] /opt/splunk/etc/system/local/server.conf active_group = Forwarder /opt/splunk/etc/system/default/server.conf connection_timeout = 30 /opt/splunk/etc/system/default/server.conf manager_uri = self /opt/splunk/etc/system/default/server.conf receive_timeout = 30 /opt/splunk/etc/system/default/server.conf report_interval = 1m /opt/splunk/etc/system/default/server.conf send_timeout = 30 /opt/splunk/etc/system/default/server.conf squash_threshold = 2000 /opt/splunk/etc/system/default/server.conf strict_pool_quota = true I did something else, and that is remove the heavy's from the distributed search peers. Why they where there i don't know. It resolved one thing, the warning about disabling the peer...   The only thing remaining is the duplicate license hash (ffffff...) in the _internal index. I can understand the hash itself. Every forwaredr with this license has this hash. What i don't understand is why this warnimng. And it is only the warning for the heavy's  which were in the distributed serach peers. Not the one's which were not in that list. It seems something remained  someweher and keeps looking to the license on these heavy's and keeps reporting it is the same license...   Any idea? ... View more

I also did a dbinspect on this index and searched for the bicketId. It gives below: bucketId aaaaaa~183~839799B0-6EAF-436C-B12A-2CDC010C1319 eventCount 1660559027 eventCount 0 guId B5D4AECD-273A-4CB5-88B4-F6C5C75C3564 hostCount 0 id 183 index aaaaaa modTime 01/30/2024:15:24:57 path /opt/splunk/data/cold/aaaaaa/db_1660559027_1659954230_183_839799B0-6EAF-436C-B12A-2CDC010C1319 rawSize 0 sizeOnDiskMB 3.078125 sourceCount 0 sourceTypeCount 0 splunk_server server1.bez.nl startEpoch 1659954230 state cold tsidxState full I don't understand the fact that it says it is in cold. This index (as all on these servers) are migrated to Smartstore. so this path is wrong. Am i missing something? And the eventcount 0? rawsize 0? but also a startEpoch and endEpoch without events? ... View more

Thanks, this is what i was hoping for. So the manual is in this case only for safety i suppose. Already tested it on a development environment and then questioned myself why taking the peers down. Will continue testing.   greetz Jari ... View more

Thanks for the answer. So it doesn't matter if the target is a s2s 9997/tcp target and a 8088 Hec target at the same time?  grts Jari   ... View more

Okay, thats another point for consideration. So no WORM for the S3 storage? pitty but understandable. off course the delete command i never thought about , just because they told to use the retention from the s3 itself. ... View more

It solved itself after restart two ther indexers and remove excess buckets per index (what strangly was allowed). After a while it seems there was enough space for removing the rest off the excess buckets.   ... View more

Hi SinghK Thanx for the idea, i will look at that., could be the way to go.   But why is the UF calling the splunk-powershell.ps1 script in de bin foder, what is its purpose.. ... View more