I am creating a python script using Splunk SDK for Python to automate one of my tasks - update all savedsearches to run as 'fast' mode (instead of 'verbose' and 'smart') This is where I'm at: query = '| rest /servicesNS/-/-/saved/searches | where disabled=0 | search "display.page.search.mode"!="fast" "eai:acl.app"!="splunk_archiver" "eai:acl.app"!="splunk_monitoring_console" "eai:acl.app"!="splunk_instrumentation" "eai:acl.owner"!="nobody" | table title cron_schedule eai:acl.owner actions eai:acl.app action.email.message.alert action.email.to action.email.cc search updated description display.page.search.mode' service = client.connect( host='<my_splunk_search_head_url>', port=443, scheme='https', verify=False, autologin=True, token=token, ) job = service.search(query=query, output_mode='json') total_elapsed_time = 0 while not job.is_done(): sleep(.2) total_elapsed_time += 0.2 print(f"Job is {job.is_done()} after {total_elapsed_time} secs") list_rule_not_running_fast_mode = [] rr = results.JSONResultsReader(job.results(output_mode='json')) for result in rr: if isinstance(result, results.Message): print(f'{result.type}: {result.message}') elif isinstance(result, dict): print(result['title']) list_rule_not_running_fast_mode.append(result['title']) all_saved_searchs = service.saved_searches The query could return all the savedsearches not running in 'fast' mode, however, the service.saved_searches list could not find all the returned savedsearch names. So in short, the problem is, service.saved_searches does not contain all of my savedsearches. What am I missing here? ... View more

Hi, As the title suggests, I want to "revive" a Splunk lab in our company, which was running on version 8.1.2. My target is version 9.2.10 The lab consists of a full Splunk deployment - it has Cluster Master, multisite Indexer Cluster, Search Head Cluster, SH Deployer, Deployment Server and a few UFs. I have read through some docs about upgrading Splunk, and it appeared to me that I have to through 2 upgrade steps: From 8.1.2 to 9.0 (I am planning to upgrade it to 9.0.4 first), then from 9.0.4 to 9.2.10. Currently Splunk is running in the /home/splunk folder. What I concern most is: How to retain the data of the Indexers? I saw a vid showing the upgrade process, in which they tar the whole $SPLUNK_HOME folder for backup. But that process is quite challenging if you have TBs of data in the $SPLUNK_HOME/var/lib folder, right? Is there any other way to retain data, after upgrading? Also, is there any other thing I should take note of? Any suggestions, recommendations, is welcome ... View more

Hi, I am configure the apps on the UF from a Deploy Server, and get this weird behavior: What I am trying to do is assigning the sourcetype "nginx:access" for every file that have this path "/var/log/nginx/access.log*" [monitor:///var/log] whitelist = (\.log|log$|messages|secure|auth|mesg$|cron$|acpid$|\.out) blacklist = (lastlog|anaconda\.syslog) disabled = 0 [monitor:///var/log/nginx] whitelist = (^access\.log) sourcetype = nginx:access disabled = 0 However, upon deploying the apps, I see that there are still numerous weird sourcetypes assigning to the source "/var/log/nginx/access.log". For example, when running | tstats values(sourcetype) where index=*_linux source=/var/log/nginx/access.log by source I got: access-100 access-13 access-204 access-29 access-66 access-7 access-too_small access.log-16 access.log-4 access.log-85 access_combined access_combined_wcookie nginx:access  How do I troubleshoot/fix this? ... View more

I am looking for the best way in terms of performance when adding filtering of certain events for security rules. Normally for a security rule, it starts off with quite a large scope, for example: index=windows source=XmlWinEventLog:Security process_name=ipconfig.exe  Then often in your environment, you would have to filter benign processes, behaviors. Currently, this is how I am writing filters index=windows source=XmlWinEventLog:Security EventCode=4688 process_name=ipconfig.exe | search NOT process_command_line="ipconfig /all" | search NOT process_parent_path=*benign.exe host=BENIGN_HOSTS This gives the best readability, but I am looking for best performance. Then what is the best way to write filters?  ... View more

I want to get the `search` of the requests that made to the search head using API (not UI) like Splunk Python SDK. My idea is to parse the _internal log to get the search_id, then join it with _audit log to get the search. So here is my SPL: index=_internal sourcetype=splunkd_access source=*splunkd_access.log method=POST useragent IN (axios*, curl*, python-requests*, splunk-sdk-python*, node-fetch*) NOT user IN (splunk-system-user, "-") | rex field=uri_path ".*/search/jobs/(?<search_id>[^/]+)" | eval search_id = "'" . search_id . "'" | where isnotnull(search_id) AND !like(search_id, "'export'") | join search_id [ search index=_audit action=search info=granted | fields search_id search ] | table _time host clientip user useragent search_id search However, this query `search` column returned nothing, though the search_id column has the correct value as `'<search_id>'`. If I take out the `'<search_id>'` and make a query like: index=_audit action=search info=granted search_id="'<search_id>'" | table _time search I could get the corresponding search. Somehow my `join` command is not working.  ... View more