After install of a new Enterprise 9.0 instance, there's a lot of new logging appearing in _internal. Notably, this log line is being generated every 15 seconds and there's no clear indication in documentation how to disable it.     2022-06-23 09:25:05,957 INFO [assist::supervisor_modular_input.py] [context] [build_supervisor_secrets] [4932] Secret load failed, key=tenant_id, error=[HTTP 404] https://127.0.0.1:8090/servicesNS/nobody/splunk_assist/storage/passwords/tenant_id?output_mode=json     source = D:\Splunk\var\log\splunk\splunk_assist_supervisor_modular_input.log sourcetype = splunk_assist_uiassets_modular_input.log* This is a substantial increase in overall volume of logs with "error" in them, not to mention the rest of the logging related to these new "assist supervisor" processes.  splunkd.log is flooded with messages from instance_id_modular_input.py executing.   The Splunk Assist documentation (https://docs.splunk.com/Documentation/Splunk/9.0.0/DMC/AssistIntro) has no information on how to adjust the log level or disable specific components. This is on an instance *without* a Splunk Assist activation code installed, meaning this is generating at this volume out-of-box.   It's incredibly frustrating that searching this log file name "splunk_assist_uiassets_modular_input.log" returns 0 results in all of Splunk Docs. How is this useful if there's no information on what to do with it, and why am I paying more for Cloud Compute to ingest all this additional volume without any instruction for how to configure it? Any assistance in finding relevant documentation would be appreciated. Edit: There's a new .conf file for this - assist.conf - that is completely undocumented. Nothing in the configuration file reference doc page. https://docs.splunk.com/Documentation/Splunk/9.0.0/Admin/assistconf The inputs generating all this extra logging are located in $SPLUNK_HOME/etc/apps/splunk_assist Until more information becomes available, I've disabled them: [supervisor_modular_input://default] disabled = 1 [instance_id_modular_input://default] disabled = 1 [uiassets_modular_input://default] disabled = 1 [selfupdate_modular_input://default] disabled = 1  

Hi all, My team needs to clear an alert with a totally different department before we consider it "published" for the purposes of audit etc. I need a SIMPLE way to mark an alert as "in review"  that has the ability to make the distinction between "published" and "in review" clear on dashboards.  Requirements: 1. something simple that a non-tech team won't mess up 2. readable by dashboards Thanks in advance!

Hello I'm having trouble with metric data not being sent from our HF to our Enterprise deployment. I'll add a diagram later to better explain myself. For now, our deployment looks a bit like this: Monitored File -----> UF -----> HF ------> Indexer (WORKS!) We decided to send our collectd metric data through the UF but for some reason, the metric data got lost while the monitored file data reached our indexer: Monitored File ------┐                                         ├------> UF --------> HF ------> Indexer (Only file works) Metric Data ---------┘ As part of the debugging test I ran to solve this I realized that sending the metric data straight to the indexer while having the file data pass through the HF works! while assuring me that the problem is not with the UF or the metric data, I still need the HF to act as a proxy in our network. Since we have the Forwarder License on the HF we can't run searches on it and `splunkd.log` or `metrics.log` are not showing any errors. Can anyone point me to some setting on the HF that I might miss? I've been trying to solve this for a couple of days but can't seem to make any progress.   Edit: Let me know what .conf files or other configurations you need me to share.

hi Can I make the below mstats into one SPL? Below there are 2. I am pulling data from the same metric. I have a "where" in one.       | mstats sum("mx.http.server.requests") as total WHERE "index"="murex_metrics" mx.env="feature/monitoring/nfr/new-log-metrics-19" | appendcols [ mstats sum("mx.http.server.requests") as declined WHERE "index"="murex_metrics" mx.env="feature/monitoring/nfr/new-log-metrics-19" status.code>=400 | appendpipe [ stats count | eval declined=0 | where count=0 | table declined ]] | eval percent_declined=100 * declined / total | table percent_declined     I looked at this question and I don't think I can use "stats style eval aggr functions"    | stats count(eval(ip=="37.25.139.34"))   https://community.splunk.com/t5/Splunk-Search/How-to-use-eval-with-mstats/td-p/563751 So perhaps I need to run 2 mstats, I was hoping I could just run 1. However i think it is not possible for what i am trying to do.   Rob

Hello, Is it possible to relate the issues displayed in Splunk UI (attached  below) to OS data or Splunk logs:   In other words, Given the OS metrics(RAM,CPU,SWAP,....) of the servers hosting Splunk and Splunk logs, can we relate some trends in this data to below issues: The percentage of high priority searches lagged (33%) over the last 24 hours is very high and exceeded the yellow thresholds (10%) on this Splunk instance. Total Searches that were part of this percentage=18. Total lagged Searches=6 The percentage of small buckets created over the last hour is high and exceeded the red thresholds  for index=...,  and possibly more indexes, on this indexer. At the time this alert fired, total buckets created=13, small buckets=11 The percentage of non high priority searches delayed (54%) over the last 24 hours is very high and exceeded the red thresholds (20%) on this Splunk instance. Total Searches that were part of this percentage=574144. Total delayed Searches=314696 Search peer down Disk space/file system under this mount point ... is exceeding the limits 80%   Would really appreciate your response.

Hi Team, I have two panels . For my 1st panel the query is: <title>DataGraphNodes Exceptions Details</title> <table> <search> <query>index=abc ns=sidh-datagraph3-c2 OR sidh-datagraph3 nodeException node="*" |rex field=_raw "message=(?P&lt;datetime&gt;\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}\.\d+)\s"|stats count by ns app_name node nodeMethodName nodeException datetime |rename node as "Node"| rename nodeMethodName as "NodeMethod"|rename nodeException as "Node-Exception" | rename datetime as "Log_Time"|fields - count</query> <earliest>$field1.earliest$</earliest> <latest>$field1.latest$</latest> </search> <drilldown> <set token="show_panel">true</set> <set token="selected_value">$click.value$</set> </drilldown> </table>   And I am getting result like this: ns                                        app-name                   Node        Node method                Exception   Log-Time sidh-datagraph3  data-graph-acct-b          https       getDetailsBySENo             Invalid Id                2022-06-21 sidh2                          data-acct-b                      https          invalid                                    InvalidId                2022-06-22   Foe 2nd panel I want when I click on 1st panel row the details should come based on the row I will select on 1st panel. My 2nd panel query is: <panel depends="$show_panel$"> <table> <title> Events</title> <search> <query>index=abc ns=sidh-datagraph3-c2 OR sidh-datagraph3 nodeException $node$ $selected_value$ </query> <earliest>$field1.earliest$</earliest> <latest>$field1.latest$</latest> </search> <option name="count">100</option> </table> </panel>   But its not coming proper in 2nd panel all the results are coming. I want only the row that I select in 1st panel that exception will come. Can someone guide me.

Using the Splunk Python SDK, is there any way to know the size of the CSV file that will be generated after streaming and writing all the results? I've managed to achieve this with the approach below, but it makes me to download and iterate the CSV lines two times. splunk_job = service.jobs.create(query, **kwargs) # waits job to be done splunk_job_result_args = { "output_mode": "csv" } splunk_job_results = splunk_job.results(**splunk_job_result_args) results_length = 0 for bytes_csv_line in splunk_job_results: results_length += len(bytes_csv_line) splunk_job_results.close() # checks if the results_length exceeds the limit # if not, executes the following: splunk_job_results = splunk_job.results(**splunk_job_result_args) for bytes_csv_line in splunk_job_results: # writes the bytes in a file splunk_job_results.close()  

I tried to install Splunk using below commands in powershell, which installs without any issue and the service runs in services.msc msiexec.exe /i <location of .msi file>  AGREETOLICENSE=Yes SPLUNKUSERNAME=admin SPLUNKPASSWORD=PASSWORD1 LAUNCHSPLUNK=1 /qb But using the same command, when I run this in AWS, it still pics Windows AMI, but not the Splunk AMI.  Can anyone advise?    

Hello, My alert produces a table like this:   Time |ID | FILE_NAME |STATUS _time1 |3 |file1.csv |SUCCESS _time2 |5 |file2.csv |DATA_ERROR     I want to send an Inline table that only contains STATUS=DATA_ERROR. But in the body of the email, I still want to use the token $result.Time$ and $result.FILE_NAME$ from the STATUS=SUCCESS. Email body example: 1. File name success detail: File name: file1.csv Effective time: _time1 2. Data error detail: ID |FILE_NAME|STATUS 5  |file2.scv        |DATA_ERROR So it's basically, hide the STATUS=SUCCESS row- but still use its values in the token email. Thank you in advance

Find large CSV lookups above 400 mb (500 mb limit) : | rest splunk_server=* /servicesNS/-/-/data/transforms/lookups getsize=true f=size f=title f=type f=filename f=eai*|fields splunk_server filename title type size eai:appName |where isnotnull(size)|eval KB = round(size / 1024, 2)|fields - size | sort - KB | search KB>400000   Use this to reduce CSV lookup (example) : | inputlookup file.csv | eval time_epoch = strftime(_time,"%s") | where time_epoch>relative_time(now(),"-100d@d") | outputlookup file.csv append=false  

I have the below query, I need the scatter point visualization for this. time on the x axis and the build duration  on the y axis for different job url as labels How to achieve this. index="maas-01" sourcetype="jenkins_run:pipeline/describe" source=* "content.stages{}.stage_name"="build:execute" |rename content.stages{}.stage_duration_sec as duration content.stages{}.stage_name as name content.build_id as id | eval trimed_source = trim (source, "jenkins_run:/job/") | eval job_url = substr(trimed_source, 1, len(trimed_source )-2) |search job_url IN ($_job_url$) | table id _time name duration job_url | eval res=mvzip(name, duration) | eval name=mvindex(name, mvfind(res, "^build:execute.+")), duration=mvindex(duration, mvfind(res, "^build:execute.+")) | eval time=strptime(strftime(_time, "%Y-%m-%d %H:%M:%S.%N"),"%Y-%m-%d %H:%M:%S.%N") |eval bEx_Duration_minutes=round(duration/60, 2) | fields job_url time bEx_Duration_minutes I just need the time in human readable format , not any epoch number.  Any possibility of using scatter plot for above query with default _time? or is there any other way we can do this. Below is the visualisation which is getting generated. Need the output like below only but with readable date and time or Date only.    

Hi  We face a challenge We have created one alert in which we are monitoring one of the windows service (cloud gate way service) So basically if this service is not running or stopped splunk will trigger an alert for that.   Wanted to check if any possibility is there that if Splunk trigger such type of alert then to resolve the same Splunk will go to that server , login the server and will restart the service   We have identified one solution for this  By excute the alert action using the script  MAY I know where we can set the script (host=CSG196) can we deploy the script in host Can anyone suggest to resolve this issue