topic Re: Splunk CLI returns "Action forbidden" in Security

Splunk CLI returns "Action forbidden"

busyninja — Tue, 06 Jul 2021 10:41:03 GMT

I can query my Splunk instance using CLI with the following command:

/opt/splunk/bin/./splunk search 'index=* host=* mitre_technique!=- | stats count BY mitre_technique | fields - count' -auth user:password -app 'custom_app' -preview true

It returns results:

mitre_technique
---------------------------------------------------
T1003 - /etc/passwd and /etc/shadow
T1007 - System Service Discovery
T1011 - Exfiltration over Bluetooth
T1016 - Internet Connection Discovery
T1018 - Remote System Discovery
T1025 - Data from Removable Media
T1033 - System Owner/User Discovery
...

However, if I run it from within a python script:

print(subprocess.Popen(["/"+postpath+"splunk/bin/./splunk", "search", "'index=*", "host=*", "mitre_technique!=-", "|", "stats", "count", "BY", "mitre_technique", "|", "fields", "-", "count'", "-auth", splunkuser.strip()+":"+splunkpswd.strip(), "-app", "'custom_app'", "-preview", "true"]).communicate())

It returns:

"Action forbidden."

(None, None)

Does anyone know why this is? How can I get results returned from running the command in my python script?

Thank you in advance

Re: Splunk CLI returns "Action forbidden"

kamlesh_vaghela — Tue, 06 Jul 2021 11:35:10 GMT

@busyninja

Can you please try this?

import subprocess cmd = [ "/opt/splunk/bin/splunk", "search", "index=_internal earliest=-15m | stats count BY sourcetype | fields - count", "-auth", "admin:admin123", "-app", "search", "-preview", "true"] # cmd = ["/opt/splunk/bin/splunk", "status", "-auth", "admin:admin123"] # cmd = ["/"+postpath+"splunk/bin/splunk", "search", "index=* host=* mitre_technique!=- | stats count BY mitre_technique | fields - count", "-auth", splunkuser.strip()+":"+splunkpswd.strip(), "-app", "custom_app", "-preview", "true"] proc = subprocess.Popen(cmd, stdout=subprocess.PIPE) while True: record = proc.stdout.readline() if not record: break print(record) proc.wait()

You can try this also.

import os stream = os.popen("/opt/splunk/bin/splunk search 'index=_internal earliest=-15m | stats count BY sourcetype | fields - count' -auth admin:admin123 -app 'search' -preview true") # stream = os.popen( # "/opt/splunk/bin/splunk status -auth admin:admin123") output = stream.read() print(output)

Thanks
KV
▄︻̷̿┻̿═━一

If any of my reply helps you to solve the problem Or gain knowledge, an upvote would be appreciated.

Re: Splunk CLI returns "Action forbidden"

busyninja — Tue, 06 Jul 2021 15:15:36 GMT

@kamlesh_vaghela

Your second solution worked! Thank you

Although I now get results, they are only a subset of total results compared to the actual events returned in from running the command outside of the script. There are 64 events in total but only 23 shown from executing the CLI command from within the script, for the same search. Do you know why this is?

Re: Splunk CLI returns "Action forbidden"

kamlesh_vaghela — Wed, 07 Jul 2021 05:38:28 GMT

@busyninja

It should work from script as well. can you please share your sample search& script?

Re: Splunk CLI returns "Action forbidden"

busyninja — Wed, 07 Jul 2021 08:46:23 GMT

My script is rather big but here is the command I run:

mitre_techniques = os.popen('/'+postpath+'splunk/bin/./splunk search "index=* host=* mitre_technique!=- | stats count BY mitre_technique | fields - count" -auth '+splunkuser.strip()+':'+splunkpswd.strip()+' -app "custom_app" -maxout 0 -max_time 0 -header false').read() print(mitre_techniques)

Which returns 23 results

T1007 - System Service Discovery
T1018 - Remote System Discovery
T1053 - Scheduled Task
T1059 - PowerShell
T1059 - Windows Command Shell
T1070 - Network Share Connection Removal
T1106 - Native API
T1112 - Modify Registry
T1203 - Exploitation for Client Execution
T1204 - User Execution
T1216 - PubPrn
T1218 - Control Panel
T1218 - Msiexec
T1218 - Rundll32
T1489 - Service Stop
T1518 - Security Software Discovery
T1543 - Windows Service
T1546 - Accessibility Features
T1546 - Screensaver
T1548 - Bypass User Account Control
T1564 - Run Virtual Instance
T1569 - Service Execution
T1574 - Path Interception by Search Order Hijacking

However, I run the exact same command at the command line:

sudo /opt/splunk/bin/./splunk search "index=* host=* mitre_technique!=- | stats count BY mitre_technique | fields - count" -auth admin:splunksplunk -app "custom_app" -maxout 0 -max_time 0 -header false

I get 64 results:

T1003 - /etc/passwd and /etc/shadow
T1007 - System Service Discovery
T1011 - Exfiltration over Bluetooth
T1016 - Internet Connection Discovery
T1018 - Remote System Discovery
T1025 - Data from Removable Media
T1033 - System Owner/User Discovery
T1052 - Exfiltration over Physical Medium
T1053 - Scheduled Task
T1056 - Keylogging
T1057 - Process Discovery
T1059 - PowerShell
T1059 - Windows Command Shell
T1069 - Domain Groups
T1069 - Local Groups
T1070 - Clear Linux or Mac System Logs
T1070 - File Deletion
T1070 - Network Share Connection Removal
T1083 - File and Directory Discovery
T1087 - Local Account
T1091 - Replication through Removable Media
T1105 - Ingress Tool Transfer
T1106 - Native API
T1112 - Modify Registry
T1113 - Screen Capture
T1200 - Hardware Additions
T1203 - Exploitation for Client Execution
T1204 - User Execution
T1216 - PubPrn
T1218 - Control Panel
T1218 - Msiexec
T1218 - Rundll32
T1485 - Data Destruction
T1489 - Service Stop
T1497 - Time Based Evasion
T1518 - Security Software Discovery
T1543 - Windows Service
T1546 - Accessibility Features
T1546 - Screensaver
T1546 - Unix Shell Configuration
T1547 - Dylib Hijacking
T1547 - Dynamic Linker Hijacking
T1547 - Kernel Modules and Extensions
T1547 - Message Modification
T1547 - Service Modification
T1547 - XDG Autostart Entries
T1548 - Bypass User Account Control
T1548 - Sudo and Sudo Caching
T1552 - Credentials in Files
T1552 - Private Keys
T1555 - Credentials from Password Stores
T1555 - Keychain
T1556 - Pluggable Authentication Modules
T1560 - Archive Collected Data
T1560 - Archive via Utility
T1562 - Disable of Modify Tools
T1564 - Hidden Users
T1564 - Run Virtual Instance
T1569 - Launchctl
T1569 - Service Execution
T1570 - Lateral Tool Transfer
T1573 - Asymmetric Cryptography
T1573 - Symmetric Cryptography
T1574 - Path Interception by Search Order Hijacking

I did notice something odd when running the command at the command line (not using the script). If I wrap the search query in single quotes (') I get 23 results but if I use double quotes (") I get 64 results...

But annoyingly, I get 23 results regardless of whether I use single or double quotes in the script.

Re: Splunk CLI returns "Action forbidden"

kamlesh_vaghela — Wed, 07 Jul 2021 10:49:35 GMT

Strange.. Your scripts looks good. with single quotes and double quotes the result set are same.

Try this.

CLIs

Scripts

import os output = os.popen( '/opt/splunk/bin/splunk search "| makeresults count=200 | eval a=1| accum a" -auth admin:admin123 -app "search" -maxout 0 -max_time 0 -header false -preview true').read() print(output)

import os output = os.popen( '/opt/splunk/bin/splunk search \'| makeresults count=200 | eval a=1| accum a\' -auth admin:admin123 -app "search" -maxout 0 -max_time 0 -header false -preview true').read() print(output)

Re: Splunk CLI returns "Action forbidden"

busyninja — Wed, 07 Jul 2021 12:22:31 GMT

Strange - all of those commands work on the command line and in the script.

Re: Splunk CLI returns "Action forbidden"

busyninja — Wed, 07 Jul 2021 12:47:02 GMT

So I have changed my search slightly to carry out some more testing:

/opt/splunk/bin/splunk search "index=* host=* mitre_technique!=-" -auth admin:password -app 'custom' -preview true -maxout 0 -header false

I have the following results:

From Script:

07/14/09 01:39:29,N/A,C:/Windows/System32/reg.exe,N/A,True,c:/windows/system32/reg.exe
07/14/09 01:14:31,N/A,C:/Windows/SysWOW64/rundll32.exe,N/A,True,c:/windows/syswow64/rundll32.exe
11/20/10 12:17:00,N/A,C:/Windows/system32/cmd.exe,N/A,False,c:/windows/system32/cmd.exe
11/20/10 12:17:00,N/A,C:/Windows/SysWOW64/cmd.exe,N/A,True,c:/windows/syswow64/cmd.exe
06/06/11 17:55:30,N/A,C:/Program Files (x86)/Common Files/Adobe/Acrobat/ActiveX/AcroIEHelperShim.dll,N/A,False,c:/program files (x86)/common files/adobe/acrobat/activex/acroiehelpershim.dll
11/20/10 13:24:22,N/A,C:/Windows/System32/desk.cpl,N/A,False,c:/windows/system32/desk.cpl
07/14/09 01:39:35,N/A,C:/Windows/system32/sc.exe,N/A,True,c:/windows/system32/sc.exe
06/10/09 20:39:47,N/A,C:/Windows/Microsoft.NET/Framework64/v2.0.50727/csc.exe,N/A,True,c:/windows/microsoft.net/framework64/v2.0.50727/csc.exe
03/27/10 14:59:04,N/A,C:/Program Files/Microsoft Office/Office14/WINWORD.EXE,N/A,True,c:/program files/microsoft office/office14/winword.exe
09/09/11 15:34:58,N/A,C:/Program Files (x86)/Internet Explorer/IELowutil.exe,N/A,True,c:/program files (x86)/internet explorer/ielowutil.exe
07/14/09 01:38:53,N/A,C:/Windows/system32/scrnsave.scr,N/A,True,c:/windows/system32/scrnsave.scr
06/06/11 17:55:30,N/A,C:/Program Files (x86)/Adobe/Reader 10.0/Reader/AcroRd32.exe,N/A,False,c:/program files (x86)/adobe/reader 10.0/reader/acrord32.exe
07/14/09 01:39:20,N/A,C:/WINDOWS/system32/WindowsPowerShell/v1.0/powershell.exe,N/A,False,c:/windows/system32/windowspowershell/v1.0/powershell.exe
11/20/10 13:24:58,N/A,C:/Windows/system32/msiexec.exe,N/A,True,c:/windows/system32/msiexec.exe
11/20/10 12:17:22,N/A,C:/Windows/system32/msiexec.exe,N/A,False,c:/windows/system32/msiexec.exe
11/20/10 12:17:22,N/A,C:/Windows/SysWOW64/msiexec.exe,N/A,True,c:/windows/syswow64/msiexec.exe
11/20/10 13:25:13,N/A,C:/Windows/System32/sdclt.exe,N/A,True,c:/windows/system32/sdclt.exe
11/20/10 13:25:12,N/A,C:/Windows/system32/schtasks.exe,N/A,True,c:/windows/system32/schtasks.exe
05/19/11 20:05:00,N/A,C:/Program Files (x86)/McAfee/Common Framework/McScript_InUse.exe,N/A,True,c:/program files (x86)/mcafee/common framework/mcscript_inuse.exe
06/20/10 02:06:46,N/A,C:/Program Files (x86)/Adobe/Reader 9.0/Reader/AcroRd32.exe,N/A,False,c:/program files (x86)/adobe/reader 9.0/reader/acrord32.exe
07/14/09 01:39:16,N/A,C:/Windows/system32/magnify.exe,N/A,False,c:/windows/system32/magnify.exe
09/09/11 15:34:56,N/A,C:/Program Files/Internet Explorer/iexplore.exe,N/A,True,c:/program files/internet explorer/iexplore.exe
09/09/11 15:35:02,N/A,C:/Program Files (x86)/Internet Explorer/iexplore.exe,N/A,True,c:/program files (x86)/internet explorer/iexplore.exe
07/14/09 01:39:31,N/A,C:/Windows/System32/rundll32.exe,N/A,True,c:/windows/system32/rundll32.exe
11/20/10 13:25:00,N/A,C:/Windows/system32/mstsc.exe,N/A,False,c:/windows/system32/mstsc.exe
07/14/09 01:39:06,N/A,C:/Windows/system32/displayswitch.exe,N/A,False,c:/windows/system32/displayswitch.exe
11/20/10 13:24:33,N/A,C:/Windows/system32/cmd.exe,N/A,True,c:/windows/system32/cmd.exe
07/14/09 01:39:37,N/A,C:/Windows/system32/services.exe,N/A,True,c:/windows/system32/services.exe
11/08/09 03:49:06,N/A,C:/Program Files/VMware/VMware Tools/VMwareService.exe,N/A,True,c:/program files/vmware/vmware tools/vmwareservice.exe
11/08/09 03:49:46,N/A,C:/Program Files/VMware/VMware Tools/VMUpgradeHelper.exe,N/A,True,c:/program files/vmware/vmware tools/vmupgradehelper.exe
11/08/09 03:49:02,N/A,C:/Program Files/VMware/VMware Tools/VMwareTray.exe,N/A,True,c:/program files/vmware/vmware tools/vmwaretray.exe
11/08/09 03:49:28,N/A,C:/Program Files/VMware/VMware Tools/VMwareUser.exe,N/A,True,c:/program files/vmware/vmware tools/vmwareuser.exe
07/14/09 01:14:42,N/A,C:/Windows/system32/tasklist.exe,N/A,False,c:/windows/system32/tasklist.exe
07/14/09 01:14:42,N/A,C:/Windows/SysWOW64/tasklist.exe,N/A,True,c:/windows/syswow64/tasklist.exe
07/14/09 01:38:52,N/A,C:/Windows/System32/wscui.cpl,N/A,False,c:/windows/system32/wscui.cpl
11/20/10 13:24:22,N/A,C:/Windows/System32/appwiz.cpl,N/A,False,c:/windows/system32/appwiz.cpl

But from command line (not using script):

...

So, in this instance, for some reason the script is not outputting JSON...

Re: Splunk CLI returns "Action forbidden"

busyninja — Thu, 08 Jul 2021 06:14:42 GMT

I can't get my head around why the results in a script would be different to the standard CLI when the commands are identical...

Re: Splunk CLI returns "Action forbidden"

kamlesh_vaghela — Thu, 08 Jul 2021 07:15:48 GMT

@busyninja

Can you please try to store OP in file and see it has also same problem or not?

/opt/splunk/bin/splunk search '| makeresults count=200 | eval a=1| accum a' -auth admin:admin123 -app 'search' -preview true -maxout 0 -header false > data1.txt

Script

Re: Splunk CLI returns "Action forbidden"

busyninja — Thu, 08 Jul 2021 10:06:19 GMT

I have managed to solve the issue. The problem was that I was not waiting long enough for the data to be indexed between starting the server and running the commands in the script - they was nothing in-between. Of course, when I then ran the same command at the CLI directly, enough time had passed for the remaining data to populate the search results. I have also decided to utilise the REST API instead of using the Splunk CLI.

This is the excerpt of my script now:

subprocess.Popen(["/"+postpath+"splunk/bin/./splunk", "start"], stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate() time.sleep(30) apiout = subprocess.Popen(["curl", "-u", "admin:password", "-k", "https://localhost:8089/services/search/jobs", "-d", "search=search index=* host=* mitre_technique!=- | stats count BY mitre_technique | fields - count"], stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate() jobid = re.findall(r"<sid>(?P<sid>[^<]+)</sid>", str(apiout[0])) time.sleep(30) searchout = subprocess.Popen(["curl", "-u", "admin:password", "-k", "https://localhost:8089/services/search/jobs/"+jobid[0]+"/results/", "--get"], stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate() print(searchout)

This combination of using the REST API and a time.sleep(#) has solved the issue but thank you so much for your help and contribution.

Re: Splunk CLI returns "Action forbidden"

kamlesh_vaghela — Thu, 08 Jul 2021 13:14:18 GMT

Cool. Glad to help you @busyninja . 🙂 Please accept the answer to help community and If any of my reply helps you to solve the problem Or gain knowledge, an upvote would be appreciated.