What I did was to reassemble the two halves of the GC event using transactions. Here's an example: sourcetype=PIA_stdout ("GC" OR "secs") | transaction endswith=secs keeporphans=true | search date_hour>7 date_hour<18 | rex field=source "/webserv/(?<environment>[^/]+)/servers/" | rex field=_raw "(?<starting_occupancy>[0-9]+)K->(?<ending_occupancy>[0-9]+)K\((?<heap_size>[0-9]+)K\), (?<pause_time>[0-9.]+) secs\]" | stats max(pause_time) AS "Max Pause" BY environment In this case, I didn't care about "after hours" GCs, but I couldn't filter on datehour in the first search because the second half of the events (the part with the "secs]" at the end) didn't get datehour fields. I had to assemble the two halves into a transaction before I could do a date_hour filter. Also, when the GC event took less that apx 3 seconds, it didn't get split. So, I need keeporphans=true to keep the non-jacked-up GC events. ... View more

Check out the FIELDALIAS commands in ./SplunkTAmicrosoft-iis/default/props.conf. Depending on what you're logging out of IIS, they can jack up your field extractions. Usually it disappears your host field, but I wouldn't put it past them to mess up the punct field too. See also: https://answers.splunk.com/answers/693737/splunk-720-field-aliases-incorrect-behavior.html ... View more

Once you've figured out which FIELDALIAS commands are the problem, you can actually override them in a ./SplunkTAmicrosoft-iis/local/props.conf file by setting them equal to nothing, like: FIELDALIAS-s_computername = That's a better practice than making changes to the stuff in ./default. ... View more

Your inputs.conf settings are fine. You'll want to take a look at the ./SplunkTAmicrosoft-iis/default/props.conf file on your search head. There's a bunch of over-ambitious FIELDALIAS commands there, and depending on exactly what fields you're logging from IIS, those FIELDALIAS commands can jack-up your field extractions. I recommend commenting out this one: FIELDALIAS-s_computername = s_computername as host And either do a debug/refresh on your search head, or just restart the search head if that won't bum people out. ... View more

What worked for me was using the Rebuild forwarder assets... button in Monitoring Console > Settings > Forwarder Monitoring Setup. See: https://docs.splunk.com/Documentation/Splunk/7.1.1/DMC/Configureforwardermonitoring ... View more

Yeah. Same. We're logging VDI machines that are pretty ephemeral, so my production indexer is complaining about 4758 "missing" forwarders. Some of those are legit, but it's pretty painful to try to figure out which ones. ... View more

Here's what happened to me. I had installed the DB Connect app before installing the Oracle JDK version 1.8. After installing the right JDK and correcting your Java path (and getting that error), you need to restart Splunk one more time. Seems that Splunk will only do the initial Task Server startup on Splunk startup. After restarting Splunk, I had a running Task Server and everything was okay. ... View more

Sending escaped quotes to the searchmatch function of eval worked for me: startswith=eval(searchmatch"\"Sophos Anti-Virus service entered the stopped\"")) ... View more

If each sourcetype's field names are unique, you could do: index=A (sourcetype=st1 OR sourcetype=st2 OR sourcetype=st3 OR sourcetype=st4) | eval alertName=case(sourcetype==st1,"Alert1",sourcetype==st2,"Alert2",sourcetype==st3,"Alert3",sourcetype==st4,"Alert4") | eval column1=coalesce(field1,field3,field5,field7) | eval column2=coalesce(field2,field4,field6,field8) | dedup sourcetype, column1 | table alertName, column1, column2 ... View more

So, I would stand up a second indexer, connect it to my first indexer, and have my health data go to the second indexer and all my IT data go to the first indexer? But then I'm centrally managing everything? And either indexer could be used as a search head for all the data? And, when it comes time to separate my search heads and indexers my new search heads would be searching across both indexers? As you can see, we haven't scaled beyond a single Splunk indexer / search head yet. But, am I getting the general principle right? ... View more

Thanks. I'll try to install an indexer on the same machine as a configured forwarder and see how it goes. We wanted to separate them because we're certain we'll never need to correlate their data with ours and we didn't want their work to impact the licensing or performance of our Splunk deployment. ... View more