I have managed to solve the issue. The problem was that I was not waiting long enough for the data to be indexed between starting the server and running the commands in the script - they was nothing in-between. Of course, when I then ran the same command at the CLI directly, enough time had passed for the remaining data to populate the search results. I have also decided to utilise the REST API instead of using the Splunk CLI. This is the excerpt of my script now: subprocess.Popen(["/"+postpath+"splunk/bin/./splunk", "start"], stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate() time.sleep(30) apiout = subprocess.Popen(["curl", "-u", "admin:password", "-k", "https://localhost:8089/services/search/jobs", "-d", "search=search index=* host=* mitre_technique!=- | stats count BY mitre_technique | fields - count"], stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate() jobid = re.findall(r"<sid>(?P<sid>[^<]+)</sid>", str(apiout[0])) time.sleep(30) searchout = subprocess.Popen(["curl", "-u", "admin:password", "-k", "https://localhost:8089/services/search/jobs/"+jobid[0]+"/results/", "--get"], stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate() print(searchout)   This combination of using the REST API and a time.sleep(#) has solved the issue but thank you so much for your help and contribution. ... View more

I can't get my head around why the results in a script would be different to the standard CLI when the commands are identical... ... View more

So I have changed my search slightly to carry out some more testing: /opt/splunk/bin/splunk search "index=* host=* mitre_technique!=-" -auth admin:password -app 'custom' -preview true -maxout 0 -header false I have the following results: From Script: 07/14/09 01:39:29,N/A,C:/Windows/System32/reg.exe,N/A,True,c:/windows/system32/reg.exe 07/14/09 01:14:31,N/A,C:/Windows/SysWOW64/rundll32.exe,N/A,True,c:/windows/syswow64/rundll32.exe 11/20/10 12:17:00,N/A,C:/Windows/system32/cmd.exe,N/A,False,c:/windows/system32/cmd.exe 11/20/10 12:17:00,N/A,C:/Windows/SysWOW64/cmd.exe,N/A,True,c:/windows/syswow64/cmd.exe 06/06/11 17:55:30,N/A,C:/Program Files (x86)/Common Files/Adobe/Acrobat/ActiveX/AcroIEHelperShim.dll,N/A,False,c:/program files (x86)/common files/adobe/acrobat/activex/acroiehelpershim.dll 11/20/10 13:24:22,N/A,C:/Windows/System32/desk.cpl,N/A,False,c:/windows/system32/desk.cpl 07/14/09 01:39:35,N/A,C:/Windows/system32/sc.exe,N/A,True,c:/windows/system32/sc.exe 06/10/09 20:39:47,N/A,C:/Windows/Microsoft.NET/Framework64/v2.0.50727/csc.exe,N/A,True,c:/windows/microsoft.net/framework64/v2.0.50727/csc.exe 03/27/10 14:59:04,N/A,C:/Program Files/Microsoft Office/Office14/WINWORD.EXE,N/A,True,c:/program files/microsoft office/office14/winword.exe 09/09/11 15:34:58,N/A,C:/Program Files (x86)/Internet Explorer/IELowutil.exe,N/A,True,c:/program files (x86)/internet explorer/ielowutil.exe 07/14/09 01:38:53,N/A,C:/Windows/system32/scrnsave.scr,N/A,True,c:/windows/system32/scrnsave.scr 06/06/11 17:55:30,N/A,C:/Program Files (x86)/Adobe/Reader 10.0/Reader/AcroRd32.exe,N/A,False,c:/program files (x86)/adobe/reader 10.0/reader/acrord32.exe 07/14/09 01:39:20,N/A,C:/WINDOWS/system32/WindowsPowerShell/v1.0/powershell.exe,N/A,False,c:/windows/system32/windowspowershell/v1.0/powershell.exe 11/20/10 13:24:58,N/A,C:/Windows/system32/msiexec.exe,N/A,True,c:/windows/system32/msiexec.exe 11/20/10 12:17:22,N/A,C:/Windows/system32/msiexec.exe,N/A,False,c:/windows/system32/msiexec.exe 11/20/10 12:17:22,N/A,C:/Windows/SysWOW64/msiexec.exe,N/A,True,c:/windows/syswow64/msiexec.exe 11/20/10 13:25:13,N/A,C:/Windows/System32/sdclt.exe,N/A,True,c:/windows/system32/sdclt.exe 11/20/10 13:25:12,N/A,C:/Windows/system32/schtasks.exe,N/A,True,c:/windows/system32/schtasks.exe 05/19/11 20:05:00,N/A,C:/Program Files (x86)/McAfee/Common Framework/McScript_InUse.exe,N/A,True,c:/program files (x86)/mcafee/common framework/mcscript_inuse.exe 06/20/10 02:06:46,N/A,C:/Program Files (x86)/Adobe/Reader 9.0/Reader/AcroRd32.exe,N/A,False,c:/program files (x86)/adobe/reader 9.0/reader/acrord32.exe 07/14/09 01:39:16,N/A,C:/Windows/system32/magnify.exe,N/A,False,c:/windows/system32/magnify.exe 09/09/11 15:34:56,N/A,C:/Program Files/Internet Explorer/iexplore.exe,N/A,True,c:/program files/internet explorer/iexplore.exe 09/09/11 15:35:02,N/A,C:/Program Files (x86)/Internet Explorer/iexplore.exe,N/A,True,c:/program files (x86)/internet explorer/iexplore.exe 07/14/09 01:39:31,N/A,C:/Windows/System32/rundll32.exe,N/A,True,c:/windows/system32/rundll32.exe 11/20/10 13:25:00,N/A,C:/Windows/system32/mstsc.exe,N/A,False,c:/windows/system32/mstsc.exe 07/14/09 01:39:06,N/A,C:/Windows/system32/displayswitch.exe,N/A,False,c:/windows/system32/displayswitch.exe 11/20/10 13:24:33,N/A,C:/Windows/system32/cmd.exe,N/A,True,c:/windows/system32/cmd.exe 07/14/09 01:39:37,N/A,C:/Windows/system32/services.exe,N/A,True,c:/windows/system32/services.exe 11/08/09 03:49:06,N/A,C:/Program Files/VMware/VMware Tools/VMwareService.exe,N/A,True,c:/program files/vmware/vmware tools/vmwareservice.exe 11/08/09 03:49:46,N/A,C:/Program Files/VMware/VMware Tools/VMUpgradeHelper.exe,N/A,True,c:/program files/vmware/vmware tools/vmupgradehelper.exe 11/08/09 03:49:02,N/A,C:/Program Files/VMware/VMware Tools/VMwareTray.exe,N/A,True,c:/program files/vmware/vmware tools/vmwaretray.exe 11/08/09 03:49:28,N/A,C:/Program Files/VMware/VMware Tools/VMwareUser.exe,N/A,True,c:/program files/vmware/vmware tools/vmwareuser.exe 07/14/09 01:14:42,N/A,C:/Windows/system32/tasklist.exe,N/A,False,c:/windows/system32/tasklist.exe 07/14/09 01:14:42,N/A,C:/Windows/SysWOW64/tasklist.exe,N/A,True,c:/windows/syswow64/tasklist.exe 07/14/09 01:38:52,N/A,C:/Windows/System32/wscui.cpl,N/A,False,c:/windows/system32/wscui.cpl 11/20/10 13:24:22,N/A,C:/Windows/System32/appwiz.cpl,N/A,False,c:/windows/system32/appwiz.cpl   But from command line (not using script): 07/14/09 01:39:29,N/A,C:/Windows/System32/reg.exe,N/A,True,c:/windows/system32/reg.exe 07/14/09 01:14:31,N/A,C:/Windows/SysWOW64/rundll32.exe,N/A,True,c:/windows/syswow64/rundll32.exe 11/20/10 12:17:00,N/A,C:/Windows/system32/cmd.exe,N/A,False,c:/windows/system32/cmd.exe 11/20/10 12:17:00,N/A,C:/Windows/SysWOW64/cmd.exe,N/A,True,c:/windows/syswow64/cmd.exe 06/06/11 17:55:30,N/A,C:/Program Files (x86)/Common Files/Adobe/Acrobat/ActiveX/AcroIEHelperShim.dll,N/A,False,c:/program files (x86)/common files/adobe/acrobat/activex/acroiehelpershim.dll 11/20/10 13:24:22,N/A,C:/Windows/System32/desk.cpl,N/A,False,c:/windows/system32/desk.cpl 07/14/09 01:39:35,N/A,C:/Windows/system32/sc.exe,N/A,True,c:/windows/system32/sc.exe 06/10/09 20:39:47,N/A,C:/Windows/Microsoft.NET/Framework64/v2.0.50727/csc.exe,N/A,True,c:/windows/microsoft.net/framework64/v2.0.50727/csc.exe 03/27/10 14:59:04,N/A,C:/Program Files/Microsoft Office/Office14/WINWORD.EXE,N/A,True,c:/program files/microsoft office/office14/winword.exe 09/09/11 15:34:58,N/A,C:/Program Files (x86)/Internet Explorer/IELowutil.exe,N/A,True,c:/program files (x86)/internet explorer/ielowutil.exe 07/14/09 01:38:53,N/A,C:/Windows/system32/scrnsave.scr,N/A,True,c:/windows/system32/scrnsave.scr 06/06/11 17:55:30,N/A,C:/Program Files (x86)/Adobe/Reader 10.0/Reader/AcroRd32.exe,N/A,False,c:/program files (x86)/adobe/reader 10.0/reader/acrord32.exe 07/14/09 01:39:20,N/A,C:/WINDOWS/system32/WindowsPowerShell/v1.0/powershell.exe,N/A,False,c:/windows/system32/windowspowershell/v1.0/powershell.exe 11/20/10 13:24:58,N/A,C:/Windows/system32/msiexec.exe,N/A,True,c:/windows/system32/msiexec.exe 11/20/10 12:17:22,N/A,C:/Windows/system32/msiexec.exe,N/A,False,c:/windows/system32/msiexec.exe 11/20/10 12:17:22,N/A,C:/Windows/SysWOW64/msiexec.exe,N/A,True,c:/windows/syswow64/msiexec.exe 11/20/10 13:25:13,N/A,C:/Windows/System32/sdclt.exe,N/A,True,c:/windows/system32/sdclt.exe 11/20/10 13:25:12,N/A,C:/Windows/system32/schtasks.exe,N/A,True,c:/windows/system32/schtasks.exe 05/19/11 20:05:00,N/A,C:/Program Files (x86)/McAfee/Common Framework/McScript_InUse.exe,N/A,True,c:/program files (x86)/mcafee/common framework/mcscript_inuse.exe 06/20/10 02:06:46,N/A,C:/Program Files (x86)/Adobe/Reader 9.0/Reader/AcroRd32.exe,N/A,False,c:/program files (x86)/adobe/reader 9.0/reader/acrord32.exe 07/14/09 01:39:16,N/A,C:/Windows/system32/magnify.exe,N/A,False,c:/windows/system32/magnify.exe 09/09/11 15:34:56,N/A,C:/Program Files/Internet Explorer/iexplore.exe,N/A,True,c:/program files/internet explorer/iexplore.exe 09/09/11 15:35:02,N/A,C:/Program Files (x86)/Internet Explorer/iexplore.exe,N/A,True,c:/program files (x86)/internet explorer/iexplore.exe 07/14/09 01:39:31,N/A,C:/Windows/System32/rundll32.exe,N/A,True,c:/windows/system32/rundll32.exe 11/20/10 13:25:00,N/A,C:/Windows/system32/mstsc.exe,N/A,False,c:/windows/system32/mstsc.exe 07/14/09 01:39:06,N/A,C:/Windows/system32/displayswitch.exe,N/A,False,c:/windows/system32/displayswitch.exe 11/20/10 13:24:33,N/A,C:/Windows/system32/cmd.exe,N/A,True,c:/windows/system32/cmd.exe 07/14/09 01:39:37,N/A,C:/Windows/system32/services.exe,N/A,True,c:/windows/system32/services.exe 11/08/09 03:49:06,N/A,C:/Program Files/VMware/VMware Tools/VMwareService.exe,N/A,True,c:/program files/vmware/vmware tools/vmwareservice.exe 11/08/09 03:49:46,N/A,C:/Program Files/VMware/VMware Tools/VMUpgradeHelper.exe,N/A,True,c:/program files/vmware/vmware tools/vmupgradehelper.exe 11/08/09 03:49:02,N/A,C:/Program Files/VMware/VMware Tools/VMwareTray.exe,N/A,True,c:/program files/vmware/vmware tools/vmwaretray.exe 11/08/09 03:49:28,N/A,C:/Program Files/VMware/VMware Tools/VMwareUser.exe,N/A,True,c:/program files/vmware/vmware tools/vmwareuser.exe 07/14/09 01:14:42,N/A,C:/Windows/system32/tasklist.exe,N/A,False,c:/windows/system32/tasklist.exe 07/14/09 01:14:42,N/A,C:/Windows/SysWOW64/tasklist.exe,N/A,True,c:/windows/syswow64/tasklist.exe 07/14/09 01:38:52,N/A,C:/Windows/System32/wscui.cpl,N/A,False,c:/windows/system32/wscui.cpl 11/20/10 13:24:22,N/A,C:/Windows/System32/appwiz.cpl,N/A,False,c:/windows/system32/appwiz.cpl {"Description": "sandboxed app permission store", "BusName": "org.freedesktop.impl.portal.PermissionStore", "ExecStart": "/usr/libexec/xdg-permission-store", "Type": "dbus"} {"Description": "Portal service (GTK+/GNOME implementation)", "Type": "dbus", "BusName": "org.freedesktop.impl.portal.desktop.gtk", "ExecStart": "/usr/libexec/xdg-desktop-portal-gtk"} {"Description": "Bluetooth OBEX service", "Type": "dbus", "BusName": "org.bluez.obex", "ExecStart": "/usr/lib/bluetooth/obexd", "Alias": "dbus-org.bluez.obex.service"} {"Description": "GNOME Date & Time handling", "OnFailure": "gnome-session-failed.target", "CollectMode": "inactive-or-failed", "RefuseManualStart": "true", "RefuseManualStop": "true", "PartOf": "gsd-datetime.target", "After": "gnome-session-initialized.target", "Type": "dbus", "ExecStart": "/usr/libexec/gsd-datetime", "Restart": "on-failure", "BusName": "org.gnome.SettingsDaemon.Datetime", "TimeoutStopSec": "5"} {"Description": "GNOME Maintenance of expirable data", "OnFailure": "gnome-session-failed.target", "CollectMode": "inactive-or-failed", "RefuseManualStart": "true", "RefuseManualStop": "true", "PartOf": "gsd-housekeeping.target", "After": "gnome-session-initialized.target", "Type": "dbus", "ExecStart": "/usr/libexec/gsd-housekeeping", "Restart": "on-failure", "BusName": "org.gnome.SettingsDaemon.Housekeeping", "TimeoutStopSec": "5"} ... ... ...   So, in this instance, for some reason the script is not outputting JSON... ... View more

Strange - all of those commands work on the command line and in the script. ... View more

My script is rather big but here is the command I run: mitre_techniques = os.popen('/'+postpath+'splunk/bin/./splunk search "index=* host=* mitre_technique!=- | stats count BY mitre_technique | fields - count" -auth '+splunkuser.strip()+':'+splunkpswd.strip()+' -app "custom_app" -maxout 0 -max_time 0 -header false').read() print(mitre_techniques) Which returns 23 results T1007 - System Service Discovery T1018 - Remote System Discovery T1053 - Scheduled Task T1059 - PowerShell T1059 - Windows Command Shell T1070 - Network Share Connection Removal T1106 - Native API T1112 - Modify Registry T1203 - Exploitation for Client Execution T1204 - User Execution T1216 - PubPrn T1218 - Control Panel T1218 - Msiexec T1218 - Rundll32 T1489 - Service Stop T1518 - Security Software Discovery T1543 - Windows Service T1546 - Accessibility Features T1546 - Screensaver T1548 - Bypass User Account Control T1564 - Run Virtual Instance T1569 - Service Execution T1574 - Path Interception by Search Order Hijacking   However, I run the exact same command at the command line: sudo /opt/splunk/bin/./splunk search "index=* host=* mitre_technique!=- | stats count BY mitre_technique | fields - count" -auth admin:splunksplunk -app "custom_app" -maxout 0 -max_time 0 -header false I get 64 results: T1003 - /etc/passwd and /etc/shadow T1007 - System Service Discovery T1011 - Exfiltration over Bluetooth T1016 - Internet Connection Discovery T1018 - Remote System Discovery T1025 - Data from Removable Media T1033 - System Owner/User Discovery T1052 - Exfiltration over Physical Medium T1053 - Scheduled Task T1056 - Keylogging T1057 - Process Discovery T1059 - PowerShell T1059 - Windows Command Shell T1069 - Domain Groups T1069 - Local Groups T1070 - Clear Linux or Mac System Logs T1070 - File Deletion T1070 - Network Share Connection Removal T1083 - File and Directory Discovery T1087 - Local Account T1091 - Replication through Removable Media T1105 - Ingress Tool Transfer T1106 - Native API T1112 - Modify Registry T1113 - Screen Capture T1200 - Hardware Additions T1203 - Exploitation for Client Execution T1204 - User Execution T1216 - PubPrn T1218 - Control Panel T1218 - Msiexec T1218 - Rundll32 T1485 - Data Destruction T1489 - Service Stop T1497 - Time Based Evasion T1518 - Security Software Discovery T1543 - Windows Service T1546 - Accessibility Features T1546 - Screensaver T1546 - Unix Shell Configuration T1547 - Dylib Hijacking T1547 - Dynamic Linker Hijacking T1547 - Kernel Modules and Extensions T1547 - Message Modification T1547 - Service Modification T1547 - XDG Autostart Entries T1548 - Bypass User Account Control T1548 - Sudo and Sudo Caching T1552 - Credentials in Files T1552 - Private Keys T1555 - Credentials from Password Stores T1555 - Keychain T1556 - Pluggable Authentication Modules T1560 - Archive Collected Data T1560 - Archive via Utility T1562 - Disable of Modify Tools T1564 - Hidden Users T1564 - Run Virtual Instance T1569 - Launchctl T1569 - Service Execution T1570 - Lateral Tool Transfer T1573 - Asymmetric Cryptography T1573 - Symmetric Cryptography T1574 - Path Interception by Search Order Hijacking   I did notice something odd when running the command at the command line (not using the script). If I wrap the search query in single quotes (') I get 23 results but if I use double quotes (") I get 64 results... But annoyingly, I get 23 results regardless of whether I use single or double quotes in the script. ... View more

@kamlesh_vaghela  Your second solution worked! Thank you Although I now get results, they are only a subset of total results compared to the actual events returned in from running the command outside of the script. There are 64 events in total but only 23 shown from executing the CLI command from within the script, for the same search. Do you know why this is? ... View more