I see another feed on the SH server at /opt/apps/splunk/etc/apps/SA-ThreatIntelligence/local/data/threat_intel/emerging_threats_compromised_ip_blocklist.csv Is there a way to see via the UI?     ... View more

We also wonder whether the Windows event logs have information about the SQL Server audit information. A good conversation about the Oracle audit trails at - https://community.splunk.com/t5/Splunk-Search/How-to-index-Oracle-audit-trails-stored-in-aud-files/m-p/170020    ... View more

About - Another consideration is how well balanced the data is among your indexers. An uneven distribution means one indexer is working harder than the others and is slowing down the search So, it seems that for this particular index, the primary buckets are available on only four out of of eight indexers and Support suggested to re balance the data. To be clear - the queries go only against the primary buckets?   ... View more

About the time range. I see the following in _audit -     Audit:[timestamp=07-10-2020 11:35:16.459, user= <user>, action=search, info=completed, search_id=' <user>__ <user>_YmJoX3ByaXZhdGViYW5raW5nX3Vp__search18_1594389297.3145638', total_run_time=5997.91, event_count=12146884, result_count=7, available_count=0, scan_count=2942980670, drop_count=0, exec_time=1594389297, api_et=N/A, api_lt=N/A, search_et=N/A, search_lt=N/A, is_realtime=0, savedsearch_name="search18", search_startup_time="1292", searched_buckets=630, eliminated_buckets=0, considered_events=2942980670, total_slices=3512873, decompressed_slices=3615466, duration.command.search.index=1146121, invocations.command.search.index.bucketcache.hit=0, duration.command.search.index.bucketcache.hit=0, invocations.command.search.index.bucketcache.miss=0, duration.command.search.index.bucketcache.miss=0, invocations.command.search.index.bucketcache.error=0, duration.command.search.rawdata=8063766, invocations.command.search.rawdata.bucketcache.hit=0, duration.command.search.rawdata.bucketcache.hit=0, invocations.command.search.rawdata.bucketcache.miss=0, duration.command.search.rawdata.bucketcache.miss=0, invocations.command.search.rawdata.bucketcache.error=0, sourcetype_count__<index name>=12146884, roles=<roles>', search='search index=<index name> Error* | eval ERROR_TYPE = if(searchmatch("SQLException"), "SQL Exceptions", (if(searchmatch("JwtTokenUtil"), "JWTToken Error", (if(searchmatch("JwtAuthenticationTokenFilter"), "Authentication error", (if(searchmatch("java.lang.OutOfMemoryError: Java heap space"), "Out of memory error", (if(searchmatch("sendEmail"), "Send Mail error", (if(searchmatch(" message was not delivered"), "IMAP error", (if(searchmatch("sever not reponding"), "Server error", (if(searchmatch("Timed Out"),"Timed Out Error", "OTHER"))))))))))))))) | stats count by ERROR_TYPE'][n/a]     Which field specifies the duration? The two queries that ended up consuming lots of cpu used the chart and timechart commands. Are these two prone to use lots of cpu in certain cases? ... View more

Search > Activity > Instance shows the top 20 memory searches and nothing stands out. Under Resource Usage: Instance, the graph shows us that **search** took 10 GBs of memory just before the crash and the **splunkd** process took over 20 GBs of memory. What can it be? ... View more

In the MC under Resource Usage: Instance, I see that the SH maxed out the memory. Where can I see the search(es) that caused it? ... View more

The problem was that the developer inserted the NULL character using SED. Before - | rex mode=sed s/'//g | rex "session.radius.last.attr.class is (?<radius>\w+)" | rex mode=sed field=radius "s/([0-9A-Fa-f]{2})/%\1/g" | eval radius=urldecode(substr(radius,3)) After - | rex "session.radius.last.attr.class is (?<radius>\w+)" | rex mode=sed field=radius "s/([0-9A-Fa-f]{2})/%\1/g" | eval radius=urldecode(substr(radius,3)) Sed works like s/replace-me/with-me/g = that would replace the string "replace-me" with "with-me". So the top (BEFORE) in the first SED replaced the single quote with nothing (NULL) Character. causing the issue – I didn't realize \w+ would match NULL bytes. But it certainly did. ... View more

We decided to use the enable_memory_tracker feature as in How can I generate a search which uses lots of memory? ... View more

Right @codebuilder, but it's a generic message. It seems that the index=_internal sourcetype=splunkd component=SearchProcessMemoryTracker event_message="*Forcefully terminated*" brings data about these cases. ... View more

The query (index=* OR index=_*) | table * did it and it produced the message in the UI, saying - -- The search processs with sid=1590066363.14344 was forcefully terminated because its physical memory usage (6456.715000 MB) has exceeded the 'searchprocessmemoryusagethreshold' (4000.000000 MB) setting in limits.conf. Where do we enable the MC admin message for this case, when it happens? ... View more