Splunk Search

Saved Search vs. Open Search Server Error in Query

gearmstrong
Path Finder

Good day Splunkers,

Today doing an audit of my Alerts, I opened one in "Open Search" and immediately got "Server Error" upon trying to run it.  Checked Scheduler Logs and Alert is Successfully running and issuing email as action.  It is a very lengthy beast for "PowerShell Command Execution" checking tons of IOC conditions. 

I 'Ctrl-X'd' my way through script eventually arriving at the conclusion that it appears to be breaking on one particular search term.  Now this worked previously (when originally authored in version 7.x) and currently working via Scheduler so I have questions around why running in Scheduler works vs. in Open Search.... do they run differently?  Is this 7.x 8.x (Python 2 vs. 3 syntax) difference maybe? 

We are running Enterprise 'All-in-one' Version:8.0.0 Build:1357bef0a7f6.  Below is first part of our query which is enough for you to test with... gist of it is the '-' in "Set-ExecutionPolicy" (worked before) now causes "Server Error".

For testing simply add a wildcard in place of dash - change *Set-ExecutionPolicy* to *Set*ExecutionPolicy*.  Other dashes in query do not seem to affect this.  This is very weird behavior and has hurt my brain a bit this morning.  Please advise....

index=security OR index=wineventlog source="WinEventLog:Microsoft-Windows-PowerShell/Operational"
EventCode=4103 OR EventCode=4104 OR EventCode=4688 OR EventCode=24577 Message=*Set-ExecutionPolicy*

 

Best regards,

Greg

 

 

0 Karma
1 Solution

gearmstrong
Path Finder

Answer is query was being ran from 'un-trusted' browser which is routed through Application Gateway which has very very specific firewall rules such as to block "get-psdrive" or "set-executionpolicy" but allow "get-psadforestinfo" for example. 

Anyway... I eventually discovered that if I ran the query on localhost browser it worked.  This also explains why job created (and scheduled) in version 7.x still worked but I couldn't edit/save via 'Open in Search' option.

So, my fault for not catching this before now, but I did want to share with you in case you experience similar issues maybe you can investigate whether you have any App Gateways in play!

Best regards,

Greg

View solution in original post

0 Karma

gearmstrong
Path Finder

Answer is query was being ran from 'un-trusted' browser which is routed through Application Gateway which has very very specific firewall rules such as to block "get-psdrive" or "set-executionpolicy" but allow "get-psadforestinfo" for example. 

Anyway... I eventually discovered that if I ran the query on localhost browser it worked.  This also explains why job created (and scheduled) in version 7.x still worked but I couldn't edit/save via 'Open in Search' option.

So, my fault for not catching this before now, but I did want to share with you in case you experience similar issues maybe you can investigate whether you have any App Gateways in play!

Best regards,

Greg

0 Karma

gearmstrong
Path Finder

Ok... more strange behavior from a second Alert which says is working in Scheduler Logs.  This one is our "Alert Suspicious/Administrative Processes" Alert.  I used Open Search and got Server Error.  Noticed an obvious typo with "sysprep.exe ORsysteminfo.exe ".  "OR" needs a space but when fixed I still see Server Error. So this means what is scheduled vs. what we open are different copies...  yes?

Same procedure followed to identify culprit.  "OR powershell.exe".  When removed the SPL compiles and executes properly!  I confirmed this is correct exe name and no other issues with SPL.  

I can provide code but at this point...

 

0 Karma

gearmstrong
Path Finder

Fixed my second issue by using "OR Powershell*" vs. "OR Powershell.exe".   (not as clean as I would like but it works again)

Can these two issues be related to some sort of 'Injection Protection' or similar?  Just odd that these SPL queries used to work just fine as Saved Alerts before.

Thanks,

Greg

0 Karma
Get Updates on the Splunk Community!

Dashboards: Hiding charts while search is being executed and other uses for tokens

There are a couple of features of SimpleXML / Classic dashboards that can be used to enhance the user ...

Splunk Observability Cloud's AI Assistant in Action Series: Explaining Metrics and ...

This is the fourth post in the Splunk Observability Cloud’s AI Assistant in Action series that digs into how ...

Brains, Bytes, and Boston: Learn from the Best at .conf25

When you think of Boston, you might picture colonial charm, world-class universities, or even the crack of a ...