The way I handle this is to have a script run as a cron job from a maintenance server that queries the process scheduler tables in the database and writes out what I want to see about process scheduler jobs as key=value pairs into a text file and have a Splunk UF monitor the text file. Ugly but it works. ... View more

What I did was to reassemble the two halves of the GC event using transactions. Here's an example: sourcetype=PIA_stdout ("GC" OR "secs") | transaction endswith=secs keeporphans=true | search date_hour>7 date_hour<18 | rex field=source "/webserv/(?<environment>[^/]+)/servers/" | rex field=_raw "(?<starting_occupancy>[0-9]+)K->(?<ending_occupancy>[0-9]+)K\((?<heap_size>[0-9]+)K\), (?<pause_time>[0-9.]+) secs\]" | stats max(pause_time) AS "Max Pause" BY environment In this case, I didn't care about "after hours" GCs, but I couldn't filter on date_hour in the first search because the second half of the events (the part with the "secs]" at the end) didn't get date_hour fields. I had to assemble the two halves into a transaction before I could do a date_hour filter. Also, when the GC event took less that apx 3 seconds, it didn't get split. So, I need keeporphans=true to keep the non-jacked-up GC events. ... View more

Check out the FIELDALIAS commands in ./Splunk_TA_microsoft-iis/default/props.conf. Depending on what you're logging out of IIS, they can jack up your field extractions. Usually it disappears your host field, but I wouldn't put it past them to mess up the punct field too. See also: https://answers.splunk.com/answers/693737/splunk-720-field-aliases-incorrect-behavior.html ... View more

Once you've figured out which FIELDALIAS commands are the problem, you can actually override them in a ./Splunk_TA_microsoft-iis/local/props.conf file by setting them equal to nothing, like: FIELDALIAS-s_computername = That's a better practice than making changes to the stuff in ./default. ... View more

Your inputs.conf settings are fine. You'll want to take a look at the ./Splunk_TA_microsoft-iis/default/props.conf file on your search head. There's a bunch of over-ambitious FIELDALIAS commands there, and depending on exactly what fields you're logging from IIS, those FIELDALIAS commands can jack-up your field extractions. I recommend commenting out this one: FIELDALIAS-s_computername = s_computername as host And either do a debug/refresh on your search head, or just restart the search head if that won't bum people out. ... View more

What worked for me was using the Rebuild forwarder assets... button in Monitoring Console > Settings > Forwarder Monitoring Setup. See: https://docs.splunk.com/Documentation/Splunk/7.1.1/DMC/Configureforwardermonitoring ... View more

Yeah. Same. We're logging VDI machines that are pretty ephemeral, so my production indexer is complaining about 4758 "missing" forwarders. Some of those are legit, but it's pretty painful to try to figure out which ones. ... View more

Thanks. Lucky for me, I just have a few local accounts used for ops wall mount displays, and their passwords are documented. This would be a bummer for someone with a lot of local accounts. If Splunk ever gets around to writing a guide for how to expand out of a single instance into multiple servers, splunk.secret would be a super important thing to mention early on. The other thing is that, when you copy your $SPLUNK_HOME/etc/apps all your inputs come along for the ride, and you want to give some early thought as to where your inputs are running when the dust settles. ... View more

Thanks! That got me my LDAP config. I did have to manually re-enter the password for my LDAP binding account, but otherwise that's progress!Oh, and the fun part was trying to get the password saved faster than the search head could get the account locked out. 🙂 Now, how do I replicate local accounts? Thanks again. ... View more

I'm doing the same thing, basically. Moving from a single Splunk server to a separate search head and indexer. I copied $SPLUNK_HOME/etc/apps and $SPLUNK_HOME/etc/users to my new search head. What's missing is my authentication config. What do I need to copy to get that? ... View more

Here's what happened to me. I had installed the DB Connect app before installing the Oracle JDK version 1.8. After installing the right JDK and correcting your Java path (and getting that error), you need to restart Splunk one more time. Seems that Splunk will only do the initial Task Server startup on Splunk startup. After restarting Splunk, I had a running Task Server and everything was okay. ... View more

Are you doing this for security reasons or for performance reasons? ... View more

Sending escaped quotes to the searchmatch function of eval worked for me: startswith=eval(searchmatch"\"Sophos Anti-Virus service entered the stopped\"")) ... View more

If each sourcetype's field names are unique, you could do: index=A (sourcetype=st1 OR sourcetype=st2 OR sourcetype=st3 OR sourcetype=st4) | eval alertName=case(sourcetype==st1,"Alert1",sourcetype==st2,"Alert2",sourcetype==st3,"Alert3",sourcetype==st4,"Alert4") | eval column1=coalesce(field1,field3,field5,field7) | eval column2=coalesce(field2,field4,field6,field8) | dedup sourcetype, column1 | table alertName, column1, column2 ... View more

I'd like YYYY-MM-DD HH:MM as an option, please. ... View more

So, I would stand up a second indexer, connect it to my first indexer, and have my health data go to the second indexer and all my IT data go to the first indexer? But then I'm centrally managing everything? And either indexer could be used as a search head for all the data? And, when it comes time to separate my search heads and indexers my new search heads would be searching across both indexers? As you can see, we haven't scaled beyond a single Splunk indexer / search head yet. But, am I getting the general principle right? ... View more

Thanks. I'll try to install an indexer on the same machine as a configured forwarder and see how it goes. We wanted to separate them because we're certain we'll never need to correlate their data with ours and we didn't want their work to impact the licensing or performance of our Splunk deployment. ... View more

There's now a convert function for this: ... | convert dur2sec(CallDuration) AS duration ... View more

Well, this sure worked when nothing else I tried did. Until someone comes up with a better answer, this is how it's done! 😉 ... View more

In version 2.1.2 of the "Splunk App for Microsoft Exchange", the TA-Exchange-2010-MailboxStore has a scripted input that collects the mailbox audit logs into a sourcetype called MSExchange:2010:MailboxAudit ... View more

You're looking for something like this? source="mySourceType" [search source="mySourceType" myIPAddressField=192.168.53.105 | fields host] ... View more

For the one that works, my guess is that you just got lucky as it had a 50/50 chance of picking the one you wanted. It's probably too late for this, but it may have behooved you to have your private and public IPs on separate subnets. ... View more