Example query of calculating step_time by adding step_offset_ms to the _time field source="graphics.log" host="splunk_lab" sourcetype="display_log" | rex "(?<step_offset_ms>\d+(\.)?\d+)\s(?<step_description>.+)" | eval epoch_time=_time | eval step_time=epoch_time+step_offset_ms | convert ctime(step_time) | table _time step_offset_ms step_time step_description | sort _time step_time ... View more

I'm also looking into this as I have the use case of migrating apps that were created by internal users using the Splunk UI to Splunk Cloud. If we just package up the app as is and try to upload to Splunk Cloud, it gets rejected due to some config files being under local. Apps with a lot of objects can be very time consuming to merge manually. One of my colleges Mittal came up with the creative solution of just taring up the app, copying it over to our test deployer, and deploying the app to the test search cluster. During that process, the deployer does the merge of local and default to default for you. Then your merged version is under SPLUNK_HOME/etc/apps on the search cluster member. ... View more

You can can use netcat (nc) command to test sending messages to either TCP or UDP 514 or other ports on the Linux command line. Example commands (replace localhost with your ip or fqdn) if not testing directly on the same server which hosts the syslog service. udp test to localhost echo -n "<14>mytesthost This is a syslog ***UDP 514*** Test" | nc -u -w5 -v localhost 514 tcp test to localhost echo -n "<14>mytesthost This is a syslog ***TCP 514*** Test" | nc -w5 -v localhost 514 I also like this free testing tool for Windows (Essential NetTools) https://www.tamos.com/download/main/ ... View more

You could install a Universal Forwarder on the same host as the indexer and have it take over the monitoring/forwarding of Splunk indexer logs to all destinations. I don't have your exact use case, however, I do run Universal Forwarders on my indexers and search heads to monitor things like CPU and OS metrics, without having to worry about restarting the indexers. That way I can restart the universal forwarders on my indexers at any time without impacting the indexer if I want to make changes to what I'm collecting or collection intervals, etc. ... View more

Not a direct fix for your issue, however, possibly a workaround. Sounds like you are having a collision between the version of VMware and NetApp technical addons on the same heavy forwarder? Assuming you are running on Linux you can stack multiple instances of Splunk and just offset the port numbers. Most of my heavy forwarder servers have 2-5 Splunk heavy forwarder instances hosting different tecnical addons to limit blast radius due to upgrades and different version requirements, etc. I have DB connect hosted by itself and the Splunk addon for AWS hosted by itself. I tend to group smaller, less important and/or similar apps together such as Web URL or API checking addons. Anything more important, needing more resources, or that has special requirements, I host in its own instance. I run my heavy forwarders in a directory structure like the following with offset ports. /opt/spl/hf01/splunk port 8000/8089 (netapp here) /opt/spl/hf02/splunk port 8001/8090 (vmware here) /opt/spl/hf03/splunk port 8002/8091 (one more addons here) /opt/spl/hf04/splunk port 8003/8092 (one ore more addons here) ... ... View more

Check out the Splunk App for Infrastructure https://splunkbase.splunk.com/app/3975/#/details And Splidgets - SearchFu! for Splunk (a large repository of common Splunk queries) https://splunkbase.splunk.com/app/3648/ ... View more

You might also tinker with the transaction command using startswith and endswith https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Transaction ... View more

you can optionally add a field extraction for your Job id like so | rex field=source "/(?<job_id>job_\d+)/" ... View more

This will look for events where first event time is under 10 minutes. You can adjust the threshold age_seconds below up/down to your liking. Adjust the index and sourcetype constraints to match your actual index and sourcetype. | tstats min(_time) as first_time where index="*" sourcetype="*" source="*job_*" by sourcetype source | fields first_time index sourcetype source | eval current_time=now() | eval age_seconds=current_time-first_time | search age_seconds<=600 | eval age=tostring(age_seconds,"duration") | eval age = replace(age,"(\d+)\:(\d+)\:(\d+)","\1h \2min \3s") | convert ctime(first_time) ctime(current_time) | table current_time first_time age age_seconds sourcetype source ... View more

Whoohoo!!! ... View more

A sales rep or Splunk support should be able to give you a license unlock. You can also try ./splunk clean all assuming you don't mind deleting all data and resetting to defaults. ... View more

I believe you have to request every 6 months. One is for organizations and one is for individuals. https://www.splunk.com/en_us/resources/personalized-dev-test-licenses.html https://dev.splunk.com/enterprise/dev_license ... View more

I tested on a free Slack instance and had the same issue of it pinning to one channel. This seems like an an issue on the Slack side. I couldn't figure out how to grant more channels to my generic Slack app I created for the purpose of having an incoming webhook. I will post if I find the answer. ... View more

It's been a few years since we configured. I'll attempt to track down. ... View more

Even though you created the webhook with one channel, I believe it should be usable to post to any channel. A far as I know we only have one webhook defined, however, we are able to post to any valid slack channel. In each alert you will still specify #channel ... View more

This should identify locations of Splunk Enterprise or Splunk Universal Forwarders. I've only tested on Linux. # using locate command (very fast if available) locate --regex "splunk(forwarder)?/var/log/splunk/splunkd.log$" | awk -F "/var" '{print $1}' # using find command find / -type f -name "splunkd.log" 2>/dev/null | awk -F "/var" '{print $1}' ... View more

This should identify locations of Splunk Enterprise or Splunk Universal Forwarders. I've only tested on Linux. # using locate command (very fast if available) locate --regex "splunk(forwarder)?/var/log/splunk/splunkd.log$" | awk -F "/var" '{print $1}' # using find command find / -type f -name "splunkd.log" 2>/dev/null | awk -F "/var" '{print $1}' ... View more

Ok @jeffrey_berry I see you are using the the optional stanzaPrefix which I had not known existed. For me it works up to monitor:// then returns nothing if I add monitor:/// I don't see any good examples in the docs so not sure if it's designed to match the entire stanza. I've always used grep to filter my results. Are you trying to return something specific from the config or just reporting a bug or possible enhancement? ./splunk btool Usage: btool [options] CONF_FILE {list|layer|add|delete} [stanzaPrefix] Usage: btool [options] CONF_FILE {list|layer|add|delete} [stanzaPrefix] btool [options] {check|validate-strptime|validate-regex} btool [options]Options: --debug --debug-logfile=FILENAME --debug-print=[user|app|stanza|sourcefile] --user=SPLUNK_USERNAME --app=SPLUNK_APP --dir=ETC_DIR --searchpool=SEARCHPOOL_DIR --slave-apps=SLAVE_APPS --peername=SEARCH_PEER_NAME --expand-stanzas=[true|false] ... View more

In the setup page for the Splunk slack addon you can set a default webhook url. After setting the default you should no longer have to specify a webhook for each alert, only the channel. You should only need one web hook per slack instance/company not per channel. The configuration URL will be something like this. https://yoursplunkhosthere:8000/en-US/manager/slack_alerts/apps/local/slack_alerts/setup?action=edit ... View more

https://docs.splunk.com/Documentation/SplunkCloud/latest/Service/SplunkCloudservice#Splunk_Cloud_service_limits_and_constraints I believe this implies that Splunk Cloud is running on a 32 core host(max_searches_per_cpu) + 6(base_max_searches) searches coming to a total of 38 concurrent searches per search head. You may also be eligible for a search cluster in Splunk Cloud at volumes of 1TB/day and higher (Check with your sales rep). Once on a search cluster which requires a minimum of 3 nodes, you should be able to run 3 x (38) or a total of 114 concurrent searches. By default Splunk cloud max_searches_perc will only let the scheduler use up to half of the 38 searches (19). You may be able to adjust that percentage upward if have fewer adhoc search users and wish to give the scheduler the ability to run more concurrent searches or lower the value to leave more adhoc searches for users. max_searches_perc = <integer> * The maximum number of searches the scheduler can run, as a percentage of the maximum number of concurrent searches, see [search] max_searches_per_cpu for how to set the system wide maximum number of searches. * Default: 50 ... View more

Search 1 index=_audit sourcetype=audittrail | fillnull value=NULL | table _time host user action info Search 2 index=_audit sourcetype=audittrail | fillnull value=NULL | table _time host user action info _raw | transaction user action | table _time duration host user action info _raw Search 3 index=_internal sourcetype=splunk_web_access uri_path="/en-US/app/*" * user!="-" | rex field=uri_path "/app/(?<splunk_app>[^/]+)/(?<splunk_view>[^/]+)" | eval splunk_page=splunk_app+"-"+splunk_view | top 100 user splunk_page ... View more

Doing a cursory review of this app, it looks like it uses macros to define the indexes Splunk will search for the Jenkins data. You may need to adjust these values to match the indexes where you've sent the data to. You can adjust the values under settings --> advanced search --> search macros. Look at the leftmost column Name for the following entries. [jenkins_statistics_index] definition = index="jenkins_statistics" [jenkins_console_index] definition = index="jenkins_console" [jenkins_index] definition = index="jenkins" ... View more

I'm not able to recreate your issue. Splunk is monitoring $SPLUNK_HOME/var/log/splunk out of the box. You can also use the the --debug switch to show the full path to the conf file. i.e. ./splunk btool --debug inputs list /opt/spl/splunk/bin/splunk btool --debug inputs list | grep "var/log" /opt/spl/splunk/etc/apps/introspection_generator_addon/default/inputs.conf [monitor:///opt/spl/splunk/var/log/introspection] /opt/spl/splunk/etc/system/default/inputs.conf [monitor:///opt/spl/splunk/var/log/splunk] /opt/spl/splunk/etc/system/default/inputs.conf [monitor:///opt/spl/splunk/var/log/splunk/license_usage_summary.log] /opt/spl/splunk/etc/system/default/inputs.conf [monitor:///opt/spl/splunk/var/log/splunk/splunk_instrumentation_cloud.log*] /opt/spl/splunk/etc/system/default/inputs.conf [monitor:///opt/spl/splunk/var/log/watchdog/watchdog.log*] _ /opt/spl/splunk/bin/splunk btool --debug inputs list | grep "\[" /opt/spl/splunk/etc/system/default/inputs.conf [SSL] /opt/spl/splunk/etc/system/default/inputs.conf [batch:///opt/spl/splunk/var/run/splunk/search_telemetry/*search_telemetry.json] /opt/spl/splunk/etc/system/default/inputs.conf [batch:///opt/spl/splunk/var/spool/splunk] /opt/spl/splunk/etc/system/default/inputs.conf [batch:///opt/spl/splunk/var/spool/splunk/...stash_new] /opt/spl/splunk/etc/system/default/inputs.conf [blacklist:/opt/spl/splunk/etc/auth] /opt/spl/splunk/etc/system/default/inputs.conf [blacklist:/opt/spl/splunk/etc/passwd] /opt/spl/splunk/etc/system/default/inputs.conf [fschange:/opt/spl/splunk/etc] /opt/spl/splunk/etc/apps/splunk_httpinput/default/inputs.conf [http] /opt/spl/splunk/etc/system/default/inputs.conf [monitor:///opt/spl/splunk/etc/splunk.version] /opt/spl/splunk/etc/apps/introspection_generator_addon/default/inputs.conf [monitor:///opt/spl/splunk/var/log/introspection] /opt/spl/splunk/etc/system/default/inputs.conf [monitor:///opt/spl/splunk/var/log/splunk] /opt/spl/splunk/etc/system/default/inputs.conf [monitor:///opt/spl/splunk/var/log/splunk/license_usage_summary.log] /opt/spl/splunk/etc/system/default/inputs.conf [monitor:///opt/spl/splunk/var/log/splunk/splunk_instrumentation_cloud.log*] /opt/spl/splunk/etc/system/default/inputs.conf [monitor:///opt/spl/splunk/var/log/watchdog/watchdog.log*] /opt/spl/splunk/etc/system/default/inputs.conf [script] /opt/spl/splunk/etc/apps/introspection_generator_addon/default/inputs.conf [script:///opt/spl/splunk/etc/apps/introspection_generator_addon/bin/collector.path] /opt/spl/splunk/etc/apps/splunk_instrumentation/default/inputs.conf [script:///opt/spl/splunk/etc/apps/splunk_instrumentation/bin/instrumentation.py] /opt/spl/splunk/etc/apps/splunk_instrumentation/default/inputs.conf [script:///opt/spl/splunk/etc/apps/splunk_instrumentation/bin/on_splunk_start.py] /opt/spl/splunk/etc/apps/splunk_instrumentation/default/inputs.conf [script:///opt/spl/splunk/etc/apps/splunk_instrumentation/bin/schedule_delete.py] /opt/spl/splunk/etc/apps/splunk_monitoring_console/default/inputs.conf [script:///opt/spl/splunk/etc/apps/splunk_monitoring_console/bin/dmc_config.py] /opt/spl/splunk/etc/system/default/inputs.conf [splunktcp] /opt/spl/splunk/etc/system/default/inputs.conf [tcp] /opt/spl/splunk/etc/system/default/inputs.conf [udp] ... View more

Query which will give the number of seconds and human readable time_spent on a splunk page. You may need to tweak the transaction parameters such as maxspan and maxpause etc. to your liking. https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Transaction index=_internal sourcetype=splunk_web_access uri_path="/en-US/app/*" * user!="-" | rex field=uri_path "/app/(?<splunk_app>[^/]+)/(?<splunk_view>[^/]+)" | eval splunk_page=splunk_app+"-"+splunk_view | transaction maxpause=60s user splunk_page | stats sum(duration) as duration by user splunk_page | eval time_spent=tostring(duration, "duration") | search duration > 0 ... View more

Hi @itsmevic I've vetted on a generic Splunk lab server and re-uploaded the files linked above a few minutes back. This time I did a direct copy vs copying and pasting in the browser. Hopefully they render for you now. If not, let me know I can can just paste some of the searches directly instead of the dashboards. Rob ... View more