I just spent a while dealing with this, so wanted to share what I did to resolve the issue in case it will be useful for anyone else. Scenario: I updated the DB Connect app, after which Java 8 was no longer supported.  After this when accessing the DB Connect app I got the following errors: Cannot communicate with task server, please check your settings. DBX Server is not available, please make sure it is started and listening on 9996 port or consult documentation for details. I installed JDK 17.0.7, then went into DB Connect - Configuration - Settings where I updated the value of JRE Installation Path (JAVA_HOME). When saving it, I got the error Failed to restart task server.  Restarting Splunk did not help. After a couple of hours of troubleshooting, and it not really behaving any differently, I wondered what if it is trying to use the old JRE still?  So I renamed the old JRE directory from /usr/lib/jre1.8.0_151 to /usr/lib/jre1.8.0_151_z to guarantee it could not find it and then splunkd.log then had a helpful entry after restarting: ERROR ModularInputs [1677 MainThread] - <stderr> Introspecting scheme=server: /opt/splunk/etc/apps/splunk_app_db_connect/linux_x86_64/bin/server.sh: line 33: /usr/lib/jre1.8.0_151/bin/java: No such file or directory host = splunk-ifw-p02.austin.utexas.edu source = /opt/splunk/var/log/splunk/splunkd.log sourcetype = splunkd ... so it was still trying to use the older JRE (in fact, that was the original JRE I installed for DB Connect, I had since then already installed a newer JRE a couple of years ago that I *thought* it was using) To find out where this old JRE was being specified, I ran grep -rnw '/opt/splunk/etc/apps/splunk_app_db_connect' -e 'jre1.8.0_151' Which returned one file: /opt/splunk/etc/apps/splunk_app_db_connect/linux_x86_64/bin/customized.java.path   After updating the content of customized.java.path and restarting Splunk, DB Connect was working again. Searching for files containing jdk-17.0.7 returned 7 files in total..... it looks like the Settings page in DB Connect is updating all of them except for the one I had to edit. I suppose adding $JAVA_HOME would have worked as well, I was trying to get it to work without doing that since it had worked fine in the past without it set: this way there is one less place to have to go (outside of Splunk itself) to update it in the future when the JDK has to be updated again, and also not knowing if any (non-Splunk) updates or other processes in the future may change it in /etc/profile ... View more

I am also having issues with the host field appearing when i run a search for this data. I commented out the field alias in props and for some reason the host field still does not exist in the search. When running a tstats on the index by host we see values for host. But not when we just simply search the data ... View more

Check out the FIELDALIAS commands in ./Splunk_TA_microsoft-iis/default/props.conf. Depending on what you're logging out of IIS, they can jack up your field extractions. Usually it disappears your host field, but I wouldn't put it past them to mess up the punct field too. See also: https://answers.splunk.com/answers/693737/splunk-720-field-aliases-incorrect-behavior.html ... View more

Tje very first thing that I would do in your situation is move the DS, if that is changing. The problem with that is that most people who do not know better use the CLI and set deploy-server to setup Deployment Clients instead of dropping a DeploymentClient app in $SPLUNK_HOME/etc/apps/ . Make sure that when you update your DCs that you fix this mistake and use an app. ... View more

Yes!! it is possible. Make changes to two .js files as below: \Splunk\etc\apps\timeline_app\appserver\static\visualizations\timeline\src\timline.js \Splunk\etc\apps\timeline_app\appserver\static\visualizations\timeline\visualization.js save files and Logout and re-login splunk to reflect the changes. In this case I used '%a %H : %M' as desired time format. Any format can be used including "DD/MM/YY HH:MM" ... View more

So, I would stand up a second indexer, connect it to my first indexer, and have my health data go to the second indexer and all my IT data go to the first indexer? But then I'm centrally managing everything? And either indexer could be used as a search head for all the data? And, when it comes time to separate my search heads and indexers my new search heads would be searching across both indexers? As you can see, we haven't scaled beyond a single Splunk indexer / search head yet. But, am I getting the general principle right? ... View more

What worked for me was using the Rebuild forwarder assets... button in Monitoring Console > Settings > Forwarder Monitoring Setup. See: https://docs.splunk.com/Documentation/Splunk/7.1.1/DMC/Configureforwardermonitoring ... View more

Well, this sure worked when nothing else I tried did. Until someone comes up with a better answer, this is how it's done! 😉 ... View more

This answer is not valid, dur2sec does not support milliseconds. Proof: index=* | head 1 | eval CallDuration="00:00:38.60" | convert dur2sec(CallDuration) AS duration -> results in no duration field. ... View more

What I did was to reassemble the two halves of the GC event using transactions. Here's an example: sourcetype=PIA_stdout ("GC" OR "secs") | transaction endswith=secs keeporphans=true | search date_hour>7 date_hour<18 | rex field=source "/webserv/(?<environment>[^/]+)/servers/" | rex field=_raw "(?<starting_occupancy>[0-9]+)K->(?<ending_occupancy>[0-9]+)K\((?<heap_size>[0-9]+)K\), (?<pause_time>[0-9.]+) secs\]" | stats max(pause_time) AS "Max Pause" BY environment In this case, I didn't care about "after hours" GCs, but I couldn't filter on date_hour in the first search because the second half of the events (the part with the "secs]" at the end) didn't get date_hour fields. I had to assemble the two halves into a transaction before I could do a date_hour filter. Also, when the GC event took less that apx 3 seconds, it didn't get split. So, I need keeporphans=true to keep the non-jacked-up GC events. ... View more

Check the IPS to see if the Splunk IP is in the allowed hosts config on the IPS. Hope this helps. ... View more

For the one that works, my guess is that you just got lucky as it had a 50/50 chance of picking the one you wanted. It's probably too late for this, but it may have behooved you to have your private and public IPs on separate subnets. ... View more

Actually, the default behavior appears to be to look for a date in the filename if it can't find a date in the event. ... View more

Here's another thing, isn't my sourcetype override happening at index time and my field extractions happening at search time? : yeah thats what i think,,,btw why the KV_MODE=none setting ?? ... View more

Hey did you ever get this figured out? Don't want to reinvent the wheel and I would like to get field extraction working for these logs as well. ... View more

I know, but I like to filter the data before splunk indexes it. So I'm running rsyslog on the splunk server which records the data to files which splunk then reads. ... View more

Sorry, it was a typo on my part. The subsearch (the part in the square brackets) needs to start with the search command. I fixed the query above. I'm pretty sure Araitz's approach is more efficient. I was just trying to illustrate a different approach. Unfortunately, I've been indexing Windows DHCP events since before Araitz wrote his app, so my field names are guesses based on Araitz's comment. Hopefully I got them right. 🙂 ... View more

What is the resource cost of doing this on the indexers? After all, in a Windows heavy environment, some of the events with these descriptions probably make up the bulk of logged security events. Is there a way of doing it on an intermediary step? Ie. putting an intermediary forwarder between the Windows forwarders and the indexers? ... View more

Anyone aware if there is a open source version of syslog agent for AS400.. Seems developing a bespoke syslog tool (as mentioned by southeringtonp ) or using the licensed ones is only option? ... View more

Splunk needs to fix this bug. date_hour should use the offset time specified in props.conf . Why have different behaviors for time across different operators (timechart offsets via props.conf TZ=GMT setting, but date_hour does not)? At least add us a local_date_hour . @BenjaminWyatt - this works for CST offset from GMT: eval date_hour = date_hour-6 | eval date_hour = if(date_hour<0, 24 + date_hour, date_hour) to account for negative date hours. ... View more