Karma to both answers above.  Don't let ES run on the Cluster Master and It sounds like you have beefy servers, but ES can bring the beefiest of servers to its knees if you are not careful.  The Splunk ES Content Pack has close to 6000 (I could be underestimating the number it could be higher) correlation / finding searches.  If someone goes in and turns on all of those searches and you have your own stuff running, Splunk will absolutely choke and die.   As a best practice I track each and every search that is running on my ES instances and map the time windows that those searches run.  Any new searches activated are set to run in the time windows that have the least amount of searches running.  Just remember that the general rule of thumb is that each search that runs, occupies one cpu core and one gig of ram while running.   Additionally, if you have the resources, you can open the pandora's box of multithreading and / or allowing more concurrent searches - but do this ONLY as a last resort and you should validate by running top, or checking the Management console or whatever tool you use to validate that you have spare cpu and ram to allow more concurrent searches.   ... View more

While you probably could upload a file as a part of your app (I'm not sure if it would pass vetting though), Dashboard Studio has no native method of playing audio files and last time I checked it still didn't support extending by means of custom JS code. ... View more

Wait a second. You can't use props/transforms after the events have been indexed. You can do that during indexing after initial ingestion by input. ... View more

Yup. You're right. I keep forgetting about that option. For me it's clearer to do those two operations separately. I wonder though whether there is a performance difference. ... View more

Thanks, Im wondering if its a permissions issue. The details on what the process is running as and the ownership of the files in /opt/splunkforwarder should help rule it in/out either way! Let me know if you can get hold of this information. Thanks ... View more

No, I am not using a Splunk add-on I am using the Splunk forwarder to send the logs ... View more

what will be the log file name? As I mentioned i did not do any configuration for logging. helper.log_info was already present in the default python script. ... View more

Hi @dinesh001kumar  To remove any uncertainty, it *is* possible to have custom css/js within apps in Splunk Cloud, however I dont think you can upload the files via the Splunk Cloud UI, instead you will need to package the files within your custom app in <app_name>/appserver/static directory.  Once packaged and uploaded to Splunk Cloud this should work the same as you may have previously used on-premise with Splunk Enterprise.  For more information check out https://docs.splunk.com/Documentation/SplunkCloud/latest/AdvancedDev/UseCSS As @bowesmana stated, if you are on Victoria experience Splunk Cloud stack then your app should sail through appsinpect without having to be manually inspected.  🌟 Did this answer help you? If so, please consider: Adding karma to show it was useful Marking it as the solution if it resolved your issue Commenting if you need any clarification Your feedback encourages the volunteers in this community to continue contributing ... View more

Hi @Emre  Try the following: | eval message=json_extract(_raw,"message") | spath input=message   🌟 Did this answer help you? If so, please consider: Adding karma to show it was useful Marking it as the solution if it resolved your issue Commenting if you need any clarification Your feedback encourages the volunteers in this community to continue contributing ... View more

Hello, this query seems to be working but the clients field is a multivalue field for some sourcetype ,so it results are spread out ,can you modify it ? ... View more

@sgarcia  Blank h (host) values in license usage logs are due to Splunk's "squashing" process. This occurs to optimize log storage and performance when there are too many unique values to track individually If possible, analyze your actual event data Eg: index=wineventlog | stats sum(len(_raw)) as bytes by host | eval GB=round(bytes/1024/1024/1024, 2) | sort -GB Also check Licensing usage reports and to split by host for last 60 days if available or try from metrics.log to identify which ones have high thruput. index="_internal" source="*metrics.log" group="per_host_thruput" Regards, Prewin Splunk Enthusiast | Always happy to help! If this answer helped you, please consider marking it as the solution or giving a Karma. Thanks! ... View more

You are likely correct in that the UF does not read the log file twice, as it reads it initially the first time for a batch ingestion and then never again when monitoring. The log file does not update its modification time or size as it's never closed by the application. I believe the CRC would change when the oldest events are overwritten, as that occurs at the top of the file. But as pointed out by you and others, this is not desirable behaviour.  So, assuming none of the checks for change in the log file works. Do you have any ideas on how I can make the UF open and read the file, or what mechanisms that prevents it? As stated in the initial post I have already tried a few things, are there any more tricks?   ... View more

Have you created that table with stats ... values? You should try list instead of values and then those lines will keep their orders and amounts also match. After that you could use above mvzip trick to split those to correct rows. ... View more

Hi @sawwinnaung , good for you, see next time! Ciao and happy splunking Giuseppe P.S.: Karma Points are appreciated by all the contributors 😉 ... View more

@Trevorator  I dont think there is any Splunk setting to enable automatic retry or continuation for .tsidx file operations. The only way to ensure all data is accelerated and .tsidx files are preserved is to maintain a healthy infrastructure and address any resource limitations. Regards, Prewin Splunk Enthusiast | Always happy to help! If this answer helped you, please consider marking it as the solution or giving a kudos/Karma. Thanks! ... View more

As @bowesmana points out, join is not the correct approach.  @livehybrid gives a logic that implements your requirements.  But the implementation inherits some convoluted logic in your original attempt. (The use of tstats requires that field System_Name is the same as host or is otherwise extracted at index time.  But your original SPL seems to imply the opposite.  I will not make such assumption below.) Your original expression | eval MISSING = if(isnull(last_seen_ago_in_seconds) OR last_seen_ago_in_seconds>7200,"MISSING","GOOD") really just says a System contained in system_info.csv is MISSING if no matching System_Name appears in sourcetype log, or a matching System_name appears in sourcetype log but is more than 2 hours ago. Is this correct?  There should be no need to even evaluate last_seen_ago_in_seconds if you simply filter search with earliest=-2h which is more efficient, too.  Additionally, the field MISSING is unnecessary in the table because it will always have value "MISSING" according to your logic. Here is a much simplified logic: index=servers sourcetype=logs earliest=-2h | stats latest(_time) as Time by System_Name sourcetype | append [inputlookup system_info.csv | fields System Location Responsible ``` ^^^ only necessary if there are more than these three fields ``` | rename System as System_Name] | stats values(sourcetype) as _last2hours values(Location) as Location values(Responsible) as Responsible by System_Name | where isnull(_last2hours)   ... View more

in splunkd.log , of HF2 I could see these  06-13-2025 12:34:19.086 +0800 INFO Metrics - group=per_host_thruput, ingest_pipe=2, series="lmpsplablr001", kbps=4.247, eps=24.613, kb=131.668, ev=763, avg_age=2.279, max_age=3 06-13-2025 12:34:19.086 +0800 INFO Metrics - group=per_index_thruput, ingest_pipe=2, series="_internal", kbps=2.206, eps=13.032, kb=68.396, ev=404, avg_age=2.233, max_age=3 06-13-2025 12:34:19.086 +0800 INFO Metrics - group=per_index_thruput, ingest_pipe=2, series="_metrics", kbps=2.041, eps=11.581, kb=63.272, ev=359, avg_age=2.331, max_age=3 06-13-2025 12:34:19.086 +0800 INFO Metrics - group=per_source_thruput, ingest_pipe=2, series="/mnt/splunk/splunk/var/log/splunk/audit.log", kbps=0.000, eps=0.032, kb=0.000, ev=1, avg_age=0.000, max_age=0 06-13-2025 12:34:19.086 +0800 INFO Metrics - group=per_source_thruput, ingest_pipe=2, series="/mnt/splunk/splunk/var/log/splunk/metrics.log", kbps=4.082, eps=23.355, kb=126.545, ev=724, avg_age=2.312, max_age=3 06-13-2025 12:34:19.086 +0800 INFO Metrics - group=per_source_thruput, ingest_pipe=2, series="/mnt/splunk/splunk/var/log/splunk/splunkd_access.log", kbps=0.165, eps=1.226, kb=5.123, ev=38, avg_age=1.711, max_age=3 06-13-2025 12:34:19.086 +0800 INFO Metrics - group=per_sourcetype_thruput, ingest_pipe=2, series="splunk_audit", kbps=0.000, eps=0.032, kb=0.000, ev=1, avg_age=0.000, max_age=0 06-13-2025 12:34:19.086 +0800 INFO Metrics - group=per_sourcetype_thruput, ingest_pipe=2, series="splunk_metrics_log", kbps=2.041, eps=11.677, kb=63.272, ev=362, avg_age=2.312, max_age=3 06-13-2025 12:34:19.086 +0800 INFO Metrics - group=per_sourcetype_thruput, ingest_pipe=2, series="splunkd", kbps=2.041, eps=11.677, kb=63.272, ev=362, avg_age=2.312, max_age=3 06-13-2025 12:34:19.086 +0800 INFO Metrics - group=per_sourcetype_thruput, ingest_pipe=2, series="splunkd_access", kbps=0.165, eps=1.226, kb=5.123, ev=38, avg_age=1.711, max_age=3 06-13-2025 12:34:19.086 +0800 INFO Metrics - group=queue, ingest_pipe=2, name=tcpout_my_syslog_group, max_size=512000, current_size=0, largest_size=0, smallest_size=0 06-13-2025 12:34:19.086 +0800 INFO Metrics - group=queue, ingest_pipe=2, name=tcpout_primary_indexers, max_size=512000, current_size=0, largest_size=219966, smallest_size=0 06-13-2025 12:34:19.086 +0800 INFO Metrics - group=queue, ingest_pipe=2, name=aggqueue, max_size_kb=2560000, current_size_kb=0, current_size=0, largest_size=260, smallest_size=0 06-13-2025 12:34:19.086 +0800 INFO Metrics - group=queue, ingest_pipe=2, name=indexqueue, max_size_kb=2560000, current_size_kb=0, current_size=0, largest_size=8, smallest_size=0 06-13-2025 12:34:19.086 +0800 INFO Metrics - group=queue, ingest_pipe=2, name=nullqueue, max_size_kb=2560000, current_size_kb=0, current_size=0, largest_size=1, smallest_size=0 06-13-2025 12:34:19.086 +0800 INFO Metrics - group=queue, ingest_pipe=2, name=parsingqueue, max_size_kb=2560000, current_size_kb=0, current_size=0, largest_size=2, smallest_size=0 06-13-2025 12:34:19.086 +0800 INFO Metrics - group=queue, ingest_pipe=2, name=syslog_system, max_size_kb=97, current_size_kb=0, current_size=0, largest_size=1, smallest_size=0 06-13-2025 12:34:19.086 +0800 INFO Metrics - group=queue, ingest_pipe=2, name=syslog_system2, max_size_kb=97, current_size_kb=0, current_size=0, largest_size=1, smallest_size=0 06-13-2025 12:34:19.086 +0800 INFO Metrics - group=queue, ingest_pipe=2, name=typingqueue, max_size_kb=2560000, current_size_kb=0, current_size=0, largest_size=257, smallest_size=0 06-13-2025 12:34:19.086 +0800 INFO Metrics - group=syslog_connections, ingest_pipe=2, syslog_system2:x.x.x.x:514:x.x.x.x:514, sourcePort=8089, destIp=x.x.x.x, destPort=514, _tcp_Bps=0.00, _tcp_KBps=0.00, _tcp_avg_thruput=3.71, _tcp_Kprocessed=2266, _tcp_eps=0.00 06-13-2025 12:34:19.086 +0800 INFO Metrics - group=syslog_connections, ingest_pipe=2, syslog_system:y.y.y.y:514:y.y.y.y:514, sourcePort=8089, destIp=y.y.y.y, destPort=514, _tcp_Bps=0.00, _tcp_KBps=0.00, _tcp_avg_thruput=0.35, _tcp_Kprocessed=213, _tcp_eps=0.00 06-13-2025 12:34:19.086 +0800 INFO Metrics - group=tcpout_connections, ingest_pipe=2, name=primary_indexers:z.z.z.z0:9997:0:0, sourcePort=8089, destIp=z.z.z.z0, destPort=9997, _tcp_Bps=1545.33, _tcp_KBps=1.51, _tcp_avg_thruput=1.51, _tcp_Kprocessed=45, _tcp_eps=0.50, kb=44.94 06-13-2025 12:34:19.086 +0800 INFO Metrics - group=thruput, ingest_pipe=2, name=index_thruput, instantaneous_kbps=0.000, instantaneous_eps=0.000, average_kbps=0.000, total_k_processed=0.000, kb=0.000, ev=0 06-13-2025 12:34:19.086 +0800 INFO Metrics - group=thruput, ingest_pipe=2, name=thruput, instantaneous_kbps=4.505, instantaneous_eps=26.000, average_kbps=10.456, total_k_processed=6705.000, kb=139.648, ev=806, load_average=1.500   any abnormalities in this entries ? I suspect that issue is with HF 2 only .. when HF 2 stopped , everyother things works fine ..if HF2 service initiated, it start utilize 1GB to 50GB of memory only  out of 130GB , then the HF1 start use memory and log ingestion getting stopped .. especially from syslog server ( large log volume input) -> this index getting affected first  Hence increasing memory in HF 2 not helpful here ... View more

Thanks, I really do appreciate! ... View more

Thank you for the kind reply. We have an elasticsearch setup without authentication and without Certificates so I tried to comment stanza as you suggest use_ssl = 0 # opt_ca_certs_path = but no success 😞   Than i decided to go into the python scripts and try to comment out the cert(strip) line   /opt/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/input_module_elasticsearch_json.py #opt_ca_certs_path = opt_ca_certs_path.strip()     Now I get another SSL  Error in log : 2025-06-11 12:00:03,503 ERROR pid=2813813 tid=MainThread file=base_modinput.py:log_error:309 | Get error when collecting events. Traceback (most recent call last): File "/opt/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/aob_py3/modinput_wrapper/base_modinput.py", line 128, in stream_events self.collect_events(ew) File "/opt/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/elasticsearch_json.py", line 96, in collect_events input_module.collect_events(self, ew) File "/opt/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/input_module_elasticsearch_json.py", line 153, in collect_events results = search_index(opt_elasticsearch_instance_url, opt_port, opt_user, opt_secret, opt_elasticsearch_indice, opt_date_field_name, opt_time_preset, size, from_number, opt_ca_certs_path) File "/opt/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/input_module_elasticsearch_json.py", line 102, in search_index response = client.search(**search_params, scroll="1m") File "/opt/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/elasticsearch/_sync/client/utils.py", line 414, in wrapped return api(*args, **kwargs) File "/opt/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/elasticsearch/_sync/client/__init__.py", line 3859, in search return self.perform_request( # type: ignore[return-value] File "/opt/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/elasticsearch/_sync/client/_base.py", line 285, in perform_request meta, resp_body = self.transport.perform_request( File "/opt/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/elastic_transport/_transport.py", line 329, in perform_request meta, raw_data = node.perform_request( File "/opt/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/elastic_transport/_node/_http_urllib3.py", line 199, in perform_request raise err from None elastic_transport.TlsError: TLS error caused by: TlsError(TLS error caused by: SSLError([SSL: UNKNOWN_PROTOCOL] unknown protocol (_ssl.c:1161))) If anyone managed to onboard elasticsearch data without authentication or certificate validation please adivse  ... View more

  also in future we would be Decommissioning the indexer after i have send the data to sh and then i will be sending data directly to sh ... View more

@caschmid  \d+ matches only digits, not any word. If "This is my" is always constant, you can try below rex field=_raw "This is my (?<string>\w+)" | stats count by string Regards, Prewin Splunk Enthusiast | Always happy to help! If this answer helped you, please consider marking it as the solution or giving a kudos/Karma. Thanks! ... View more

First thing to debug inputs is usually, after verifying the config checking the output of splunk list monitor and splunk list inputstatus   ... View more

Hi @tgulgund  For the benefit of others who find this answer, this is slightly incorrect. It *is* possible for a single setting for all your datasources, If you want to refresh all datasources on a dashboard studio dashboard then update the defaults as per my previous message. @livehybrid wrote: Hi @tgulgund @Prewin27  You can set a default refresh time which will apply automatically to all data sources (unless a specific datasource is overwritten, edit the source of your dashboard and find the "defaults" section, under defaults->dataSources->ds.search->options create a new "refresh" key with a value containing your intended refresh interval, such as this: { "title": "testing", "description": "", "inputs": {}, "defaults": { "dataSources": { "ds.search": { "options": { "queryParameters": { "earliest": "-24h@h", "latest": "now" }, "refresh": "60s" } } } }, "visualizations": { ... ... } } 🌟 Did this answer help you? If so, please consider: Adding karma to show it was useful Marking it as the solution if it resolved your issue Commenting if you need any clarification Your feedback encourages the volunteers in this community to continue contributing   ... View more

This is one huge search. Check each of the "component searches" on their own and see how they fare. Since some of them are raw event searches over a half a year worth of data, possibly through a significant subset of that data, I expect them to be slow just because you have to plow through all those events (and one of those subsearches has a very ugly field=* condition which makes Splunk have to parse every single event!). If you need that literal functionality from those searches, I see no other way than using some acceleration techniques - the searches themselves don't seem to be very much "optimizable". You might try to change some of them to tstats with PREFIX/TERM but only if your data fits the prerequisites. ... View more