Ended up solving this myself. My solution is to tell people not to use the GUI for this as it does a very poor job on returning the actual error for you to begin tracking the issue down. What's best is to just use the config files, then test in the search bar yourself but ENABLE DEBUGGING in the search bar. The errors that are returned are far more useful. In the end after making my changes I'd then go to my HF and run
| ldaptestconnection domain=domain.net debug=true
The results this returned allowed me to very quickly diagnose a DNS issue and fix within minutes. It would be nice if the devs would make the test-connection actually return the error results and also run in debug since that is likely what you want to see when testing. shrug
... View more
While early filtering is a good rule of thumb, in this instance remember the "where" command is categorized as a Distributable Streaming search process, so this would also be done at the index level and more importantly can be done BEFORE the final output, so it does not necessarily generate more traffic as Splunk will send it down as well knowing this fact about the "where" command.
But, like I said, and learned from a great teacher I had, that is generally a good rule of thumb to follow 😉
Also, the above about Distributable Streaming goes for: eval, fields, rex, where, etc.
For the curious, here's a great read to understand how searching works wrt different commands:
... View more