My historical answer was I built a second RPM that deployed enough configuration bits to the Splunk forwarder, required the Splunk forwarder RPM, and ensured that the forwarder was started. In other words, it wrote a Splunk app into the Forwarder's /opt/splunkforwarder/etc/apps that set up talking to the deployment server to get the rest of the configurations, ensured boot-start was set, and that it started/restarted the forwarder after deployment.  It wasn't hard to create as a custom rpm/dpkg.   That way, I used the same deployment mechanism as all other packages and could ensure all the site-local bits were there, and just "install this package, which forces installation of the Splunk UF" was all they had to do. ... View more

When it comes to pulling in requirements for a dashboard, the number one requirement I have is "what is the core point of the overall dashboard?"  Your questions are good, but may create a "metrics at the cost of use" mindset where people are so worried about building metrics and pre-answering questions that they don't just sit down and build something useful, quickly.  Also, your approach may work against interrelated dashboards that drill down from one to the other, a good usage model I've used very effectively, and you also see on the monitoring console (MC).   Dashboards aren't something to be greedy or stingy about.  Let people build them.   That said, my core rule of thumb is something you may not have noticed:  Good dashboards minimize scrolling.  That means that a well written dashboard, users don't have to scroll (much).  Use pop-up panels (with close links for the pop-up), use drill downs to other dashboards or just build highly targeted dashboards.  It helps, a lot. ... View more

The question of which inputs to enable or not is always a "the inputs that provide the logs you care about".  You don't have to worry about the HF vs UF question in this area. That said, make sure you aren't routing through a HF just for the sake of it.  HFs should only be typically used for one of a few reasons: When you cannot route (actual routing, not just firewall) from the UF to the IDX (either on time schedule or permanently) When you need modular inputs Almost every other scenario, it's best to not go through a HF.  Your question makes me suspect you may be routing through a HF. When a HF (or IDX) receives logs from another source, it will "cook" or parse the logs then send them according to the outputs.conf.  There are no log types being discussed here that put you in danger of a logarithmic volume growth or logging loop.    ... View more

First thing to check is of course btool splunk btool inputs list --debug for the respective inputs.  If it shows renderXml set to false, for those inputs, then some other setting is overriding the setting you are trying to apply.  Adding the --debug flag lets you know which file provides the winning setting, so you can figure out where it is coming from. ... View more

The above referenced procedure was accurate to add Exchange Online.  Note, by default, it queries one week back and only grabs one hour at a time every query interval, meaning it takes a while to catch up to current mail messages. ... View more

Working with the Exchange administrator, it isn't clear even with the documentation how to assign the Exchange Admin role for Oauth since there isn't a clear username to assign that role.  Any one figure this one out?   ... View more

git + POSIX sh + ssh forced commands Each app is its own repo in git.  If I'm lucky, I'm using gitlab which allows me to have folder hierarchy to identify apps by function (e.g. apps for the CM, apps for the SHD, apps for the DS to push to the UFs).  Then, I built a shell script that is attached to a forced command in ~splunk/.ssh/authorized_keys so when you authenticate to that account, you pass the name of a repo.  It validates that is a valid repo, git pulls the main branch of that repo, then uses a token attached to a local Splunk user that only has the deployment permissions to deploy the update (e.g. `splunk reload deploy-server` or the SHD deployment command sequence). Repos intended to be updated by non-admins, I lock the repo so it can't get administrative config files (e.g. authentication.conf) by push rules. ... View more

With a moderate number of workstations like that, the downside of a forwarder is you know when they stop reporting in. The upside is you know when they stop reporting in. VDI are typically considered more transitory, less permanent, so a forwarder does make less sense. You may need to consider a different solution for each technology type. Some have had luck with a syslog daemon on Windows to forward log events. There really is not a single best answer in my view. Each technology has advantages. Forwarders are easier to monitor, harder to deal with systems that are expected to come up and down regularly, but also deal surprisingly well with network disconnects. Event forwarders are simple, but as you've seen, not well known for reliability. Some use syslog or other protocols to introduce more answers. ... View more

The approach I'd at least consider would be to construct an alert that determines if the lookup table needs modifying. Run that alert periodically, keep that alert query simple. To make this alert work, you need some kind of time column or similar method to know "here's how recently updated this lookup table is". If you can use the lookup table to return a timestamp, you could even have that construct an earliest= field specification for your main search for data, so that the search starts at the time of the lookup table update timestamp. Then, you'd need a custom alert action that executes a non-scheduled saved search to perform the lookup table update. Since a custom alert action could be a script, it could be used to trigger the saved search that does the update. ... View more

I typically recommend the UF on Windows servers. It makes monitoring for problems much easier, such as systems that have stopped sending any data. As for workstations, that may be a bit stickier, but if your number of workstations is small, a UF is hardly outrageous compared to many other agents I've seen on workstations. The major question to ask yourself is what happens when that workstation (which includes laptops presumably) goes home and then is brought online? ... View more

The app for parsing PAN logs does not share the field extractions. You can use the search category of the app to construct your arbitrary app. Sounds like you are not necessarily searching from within the Palo Alto Networks app. ... View more

Look at http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Bucket There are examples of almost exactly what you want. ... View more

That's the point. You're capturing the sourcetypes into a field. A transform to define a new field with the reduced portion allows you to clump them according to the pattern you identified into a new field. rex field=sourcetype "^[^:]+:(?<newsrctyp>.*)" would strip out the portion up to and including the first colon for you and place into the field newsrctyp, which is now grouped. ... View more

Just for sanity's sake, see if that token works for you in the alert body. $result.host$ means insert from the results, the value of the field named host. There are some oddities for some cases in that token, but source and host should just work. ... View more

Run a rex to create a new field based on the sourcetype to capture the portion of the sourcetype you care about. Use that new transformed sourcetype field instead. ... View more

If you make sure every access list has the log keyword, you can then analyze the 106100 log events generated and do charts based on the hexid in the log event. Those hexids are unique to the rule. ... View more

Starting as root and then handing off to a less privileged user is actually not a desired practice, but a kludge around the lack of traditional security role models in older *nix. It creates a security hole that actually needs correcting as there remains code that runs as root when you don't want that. Solaris, you can add priv_netaddr privilege to the Solaris account to allow opening a low numbered port with no other root privileges. Linux, you can try the Linux capability by a similar name to see if that works (I have not tested that). If you want to port forward, just create the iptables NAT rules to do the port forwarding, outside Splunk. I've done that with many applications in a production operation without issue. There are a few issues about the NAT table size if the number of connections is measured in the thousands, but that is unlikely with the Splunk search web interface. ... View more

Put the token $result.host$ in the subject section of the alert configuration. See http://docs.splunk.com/Documentation/Splunk/6.2.3/Alert/Setupalertactions for more details. ... View more

Due to how the UF operates, Linux capabilities are not an option. The access system call will incorrectly claim that the splunkd cannot open the file, when an actual attempt to open it for reading would succeed, so splunkd never tries. Some of us have found this problem and spent time confirming the issue. I've used setfacl, though that is a bit fragile at times. If you only have a very fixed number of files, you could write a configuration management policy to ensure file acls and default file acls are set for the appropriate files and logs. If you do use file ACLs, make sure and set the default file ACLs on the directory, and set log rotation parameters appropriately (default ACLs protect you from most log rotation issues, unless you create new directories). ... View more