[Error] Pass token values from a dashboard as parameters to python script

[mcayrol]

Hello, 

I created a dashbord with a text input, the token is then passed to a panel that executes this command:

<query>| makeresults | eval technique_id="$technique_id$" | where isnotnull(technique_id) | mitrepurplelab $technique_id$</query>

the purpose of this command is to trigger a custom command with this config:

[mitrepurplelab]
filename = mitrepurplelab.py
enableheader = true
outputheader = true
requires_srinfo = true
chunked = true
streaming = true

the mitrepurplelab.py script is then triggered, here is its code:

import sys
import requests
import logging

logging.basicConfig(filename='mitrepurplelab.log', level=logging.DEBUG, format='%(asctime)s - %(levelname)s - %(message)s')
def main():
logging.debug(f "Arguments received: {sys.argv}")
if len(sys.argv) != 2:
logging.error("Incorrect usage: python script.py <technique_id>")
print("Usage: python script.py <technique_id>")
return

technique_id = sys.argv[1]
url = "http://192.168.142.146:5000/api/mitre_attack_execution"

# Make sure your JWT token is complete and correctly formatted
token = "token
headers = {
"Authorization": f "Bearer {token}"
}
params = {
"technique_id": technique_id
}

response = requests.post(url, headers=headers, params=params)

if response.status_code == 200:
print("Request successful!")
print("Server response:")
print(response.json())

else:
logging.error(f "Error: {response.status_code}, Response body: {response.text}")
print(f "Error: {response.status_code}, Response body: {response.text}")

if __name__ == "__main__":
main()

the script works well when run by hand, for example :

python3 bin/mitrepurplelab.py T1059.003

but when I execute it via the dashboard I get this error:

in the panel search.log I get this:

02-09-2024 10:37:46.075 INFO  dispatchRunner [1626 MainThread] - Search process mode: preforked (reused process by new user) (build 1fff88043d5f).
02-09-2024 10:37:46.075 INFO  dispatchRunner [1626 MainThread] - registering build time modules, count=1
02-09-2024 10:37:46.075 INFO  dispatchRunner [1626 MainThread] - registering search time components of build time module name=vix
02-09-2024 10:37:46.076 INFO  BundlesSetup [1626 MainThread] - Setup stats for /opt/splunk/etc: wallclock_elapsed_msec=7, cpu_time_used=0.00727909, shared_services_generation=2, shared_services_population=1
02-09-2024 10:37:46.080 INFO  UserManagerPro [1626 MainThread] - Load authentication: forcing roles="admin, power, user"
02-09-2024 10:37:46.080 INFO  UserManager [10446 RunDispatch] - Setting user context: splunk-system-user
02-09-2024 10:37:46.080 INFO  UserManager [10446 RunDispatch] - Done setting user context: NULL -> splunk-system-user
02-09-2024 10:37:46.080 INFO  UserManager [10446 RunDispatch] - Unwound user context: splunk-system-user -> NULL
02-09-2024 10:37:46.080 INFO  UserManager [10446 RunDispatch] - Setting user context: admin
02-09-2024 10:37:46.080 INFO  UserManager [10446 RunDispatch] - Done setting user context: NULL -> admin
02-09-2024 10:37:46.080 INFO  dispatchRunner [10446 RunDispatch] - search context: user="admin", app="Ta-Purplelab", bs-pathname="/opt/splunk/etc"
02-09-2024 10:37:46.080 INFO  SearchParser [10446 RunDispatch] - PARSING: | makeresults | eval technique_id="T1059.003" | where isnotnull(technique_id) | mitrepurplelab T1059.003
02-09-2024 10:37:46.081 INFO  dispatchRunner [10446 RunDispatch] - Search running in non-clustered mode
02-09-2024 10:37:46.081 INFO  dispatchRunner [10446 RunDispatch] - SearchHeadInitSearchMs=0
02-09-2024 10:37:46.081 INFO  dispatchRunner [10446 RunDispatch] - Executing the Search orchestrator and iterator model (dfs=false).
02-09-2024 10:37:46.081 INFO  SearchOrchestrator [10446 RunDispatch] - SearchOrchestrator is constructed.  sid=admin__admin_VGEtUHVycGxlbGFi__search1_1707475066.37, eval_only=0
02-09-2024 10:37:46.081 INFO  SearchOrchestrator [10446 RunDispatch] -  Initialized the SRI
02-09-2024 10:37:46.081 INFO  SearchFeatureFlags [10446 RunDispatch] - Initializing feature flags from config. feature_seed=2135385444
02-09-2024 10:37:46.081 INFO  SearchFeatureFlags [10446 RunDispatch] - Setting feature_flag=parallelreduce:enablePreview:true
02-09-2024 10:37:46.081 INFO  SearchFeatureFlags [10446 RunDispatch] - Setting feature_flag=search:search_retry:false
02-09-2024 10:37:46.081 INFO  SearchFeatureFlags [10446 RunDispatch] - Setting feature_flag=search:search_retry_realtime:false
02-09-2024 10:37:46.081 INFO  SearchFeatureFlags [10446 RunDispatch] - Setting feature_flag=parallelreduce:autoAppliedPercentage:false
02-09-2024 10:37:46.081 INFO  SearchFeatureFlags [10446 RunDispatch] - Setting feature_flag=subsearch:enableConcurrentPipelineProcessing:false
02-09-2024 10:37:46.081 INFO  SearchFeatureFlags [10446 RunDispatch] - Setting feature_flag=subsearch:concurrent_pipeline_adhoc:false
02-09-2024 10:37:46.081 INFO  SearchFeatureFlags [10446 RunDispatch] - Setting feature_flag=append:support_multiple_data_sources:false
02-09-2024 10:37:46.081 INFO  SearchFeatureFlags [10446 RunDispatch] - Setting feature_flag=join:support_multiple_data_sources:false
02-09-2024 10:37:46.081 INFO  SearchFeatureFlags [10446 RunDispatch] - Setting feature_flag=search_optimization::set_required_fields:stats:false
02-09-2024 10:37:46.081 INFO  SearchFeatureFlags [10446 RunDispatch] - Setting feature_flag=searchresults:srs2:false
02-09-2024 10:37:46.081 INFO  SearchFeatureFlags [10446 RunDispatch] - Setting feature_flag=search:read_final_results_from_timeliner:true
02-09-2024 10:37:46.081 INFO  SearchFeatureFlags [10446 RunDispatch] - Setting feature_flag=search:fetch_remote_search_telemetry:true
02-09-2024 10:37:46.081 INFO  SearchFeatureFlags [10446 RunDispatch] - Setting feature_flag=testing:boolean_flag:false
02-09-2024 10:37:46.081 INFO  SearchFeatureFlags [10446 RunDispatch] - Setting feature_flag=testing:percent_flag:true
02-09-2024 10:37:46.081 INFO  SearchFeatureFlags [10446 RunDispatch] - Setting feature_flag=testing:legacy_flag:true
02-09-2024 10:37:46.081 INFO  SearchOrchestrator [10446 RunDispatch] - Search feature_flags={"v":1,"enabledFeatures":["parallelreduce:enablePreview","search:read_final_results_from_timeliner","search:fetch_remote_search_telemetry","testing:percent_flag","testing:legacy_flag"],"disabledFeatures":["search:search_retry","search:search_retry_realtime","parallelreduce:autoAppliedPercentage","subsearch:enableConcurrentPipelineProcessing","subsearch:concurrent_pipeline_adhoc","append:support_multiple_data_sources","join:support_multiple_data_sources","search_optimization::set_required_fields:stats","searchresults:srs2","testing:boolean_flag"]}
02-09-2024 10:37:46.081 INFO  ISplunkDispatch [10446 RunDispatch] - Not running in splunkd. Bundle replication not triggered.
02-09-2024 10:37:46.081 INFO  SearchOrchestrator [10449 searchOrchestrator] - Initialzing the run time settings for the orchestrator.
02-09-2024 10:37:46.081 INFO  UserManager [10449 searchOrchestrator] - Setting user context: admin
02-09-2024 10:37:46.081 INFO  UserManager [10449 searchOrchestrator] - Done setting user context: NULL -> admin
02-09-2024 10:37:46.081 INFO  AdaptiveSearchEngineSelector [10449 searchOrchestrator] - Search execution_plan=classic
02-09-2024 10:37:46.082 INFO  SearchOrchestrator [10449 searchOrchestrator] - Creating the search DAG.
02-09-2024 10:37:46.082 INFO  SearchParser [10449 searchOrchestrator] - PARSING: | makeresults | eval technique_id="T1059.003" | where isnotnull(technique_id) | mitrepurplelab T1059.003
02-09-2024 10:37:46.082 INFO  DispatchStorageManagerInfo [10449 searchOrchestrator] - Successfully created new dispatch directory for search job. sid=dc5edf3eebc8ccb6_tmp dispatch_dir=/opt/splunk/var/run/splunk/dispatch/dc5edf3eebc8ccb6_tmp
02-09-2024 10:37:46.082 INFO  SearchParser [10449 searchOrchestrator] - PARSING: premakeresults 
02-09-2024 10:37:46.082 INFO  DispatchThread [10449 searchOrchestrator] - BatchMode: allowBatchMode: 1, conf(1): 1, timeline/Status buckets(0):0, realtime(0):0, report pipe empty(0):0, reqTimeOrder(0):0, summarize(0):0, statefulStreaming(0):0
02-09-2024 10:37:46.082 INFO  DispatchThread [10449 searchOrchestrator] - required fields list to add to remote search = *
02-09-2024 10:37:46.082 INFO  DispatchCommandProcessor [10449 searchOrchestrator] - summaryHash=f2df6493ea859e37 summaryId=A6ADAC30-27EC-4F28-BEB9-3BD2C7EC3E53_Ta-Purplelab_admin_f2df6493ea859e37 remoteSearch=premakeresults 
02-09-2024 10:37:46.082 INFO  DispatchCommandProcessor [10449 searchOrchestrator] - summaryHash=NSf2df6493ea859e37 summaryId=A6ADAC30-27EC-4F28-BEB9-3BD2C7EC3E53_Ta-Purplelab_admin_NSf2df6493ea859e37 remoteSearch=premakeresults 
02-09-2024 10:37:46.082 INFO  DispatchThread [10449 searchOrchestrator] - Getting summary ID for summaryHash=NSf2df6493ea859e37
02-09-2024 10:37:46.084 INFO  DispatchThread [10449 searchOrchestrator] - Did not find a usable summary_id, setting info._summary_mode=none, not modifying input summary_id=A6ADAC30-27EC-4F28-BEB9-3BD2C7EC3E53_Ta-Purplelab_admin_NSf2df6493ea859e37
02-09-2024 10:37:46.085 INFO  SearchParser [10449 searchOrchestrator] - PARSING: | makeresults | eval technique_id="T1059.003" | where isnotnull(technique_id) | mitrepurplelab T1059.003
02-09-2024 10:37:46.085 INFO  ChunkedExternProcessor [10449 searchOrchestrator] - Running process: /opt/splunk/bin/python3.7 /opt/splunk/etc/apps/Ta-Purplelab/bin/mitrepurplelab.py
02-09-2024 10:37:46.155 ERROR ChunkedExternProcessor [10449 searchOrchestrator] - Failed attempting to parse transport header: Usage: python script.py <technique_id>
02-09-2024 10:37:46.161 ERROR ChunkedExternProcessor [10449 searchOrchestrator] - Error in 'mitrepurplelab' command: External search command exited unexpectedly.
02-09-2024 10:37:46.161 INFO  ScopedTimer [10449 searchOrchestrator] - search.optimize 0.076785640
02-09-2024 10:37:46.161 WARN  SearchPhaseGenerator [10449 searchOrchestrator] - AST processing error, exception=31SearchProcessorMessageException, error=Error in 'mitrepurplelab' command: External search command exited unexpectedly.. Fall back to 2 phase.
02-09-2024 10:37:46.161 INFO  SearchPhaseGenerator [10449 searchOrchestrator] -  Executing two phase fallback for the search=| makeresults | eval technique_id="T1059.003" | where isnotnull(technique_id) | mitrepurplelab T1059.003
02-09-2024 10:37:46.161 INFO  SearchParser [10449 searchOrchestrator] - PARSING: | makeresults | eval technique_id="T1059.003" | where isnotnull(technique_id) | mitrepurplelab T1059.003
02-09-2024 10:37:46.162 INFO  ChunkedExternProcessor [10449 searchOrchestrator] - Running process: /opt/splunk/bin/python3.7 /opt/splunk/etc/apps/Ta-Purplelab/bin/mitrepurplelab.py
02-09-2024 10:37:46.232 ERROR ChunkedExternProcessor [10449 searchOrchestrator] - Failed attempting to parse transport header: Usage: python script.py <technique_id>
02-09-2024 10:37:46.239 ERROR ChunkedExternProcessor [10449 searchOrchestrator] - Error in 'mitrepurplelab' command: External search command exited unexpectedly.
02-09-2024 10:37:46.239 ERROR SearchPhaseGenerator [10449 searchOrchestrator] - Fallback to two phase failed with SearchProcessorException: Error in 'mitrepurplelab' command: External search command exited unexpectedly.
02-09-2024 10:37:46.239 WARN  SearchPhaseGenerator [10449 searchOrchestrator] - Failed to create search phases: exception=31SearchProcessorMessageException, error=Error in 'mitrepurplelab' command: External search command exited unexpectedly.
02-09-2024 10:37:46.240 INFO  SearchStatusEnforcer [10449 searchOrchestrator] - sid=admin__admin_VGEtUHVycGxlbGFi__search1_1707475066.37, newState=BAD_INPUT_CANCEL, message=Error in 'mitrepurplelab' command: External search command exited unexpectedly.
02-09-2024 10:37:46.240 ERROR SearchStatusEnforcer [10449 searchOrchestrator] - SearchMessage orig_component=ChunkedExternProcessor sid=admin__admin_VGEtUHVycGxlbGFi__search1_1707475066.37 message_key=CHUNKED:UNEXPECTED_EXIT message=Error in 'mitrepurplelab' command: External search command exited unexpectedly.
02-09-2024 10:37:46.240 INFO  SearchStatusEnforcer [10449 searchOrchestrator] - State changed to BAD_INPUT_CANCEL: Error in 'mitrepurplelab' command: External search command exited unexpectedly.
02-09-2024 10:37:46.240 INFO  SearchStatusEnforcer [10449 searchOrchestrator] - Enforcing disk quota = 10485760000
02-09-2024 10:37:46.242 INFO  DispatchManager [10449 searchOrchestrator] - DispatchManager::dispatchHasFinished(id='admin__admin_VGEtUHVycGxlbGFi__search1_1707475066.37', username='admin')
02-09-2024 10:37:46.242 INFO  UserManager [10449 searchOrchestrator] - Unwound user context: admin -> NULL
02-09-2024 10:37:46.242 INFO  SearchOrchestrator [10446 RunDispatch] - SearchOrchestrator is destructed.  sid=admin__admin_VGEtUHVycGxlbGFi__search1_1707475066.37, eval_only=0
02-09-2024 10:37:46.242 INFO  SearchStatusEnforcer [10446 RunDispatch] - SearchStatusEnforcer is already terminated
02-09-2024 10:37:46.242 INFO  UserManager [10446 RunDispatch] - Unwound user context: admin -> NULL
02-09-2024 10:37:46.242 INFO  LookupDataProvider [10446 RunDispatch] - Clearing out lookup shared provider map
02-09-2024 10:37:46.242 INFO  dispatchRunner [1626 MainThread] - RunDispatch is done: sid=admin__admin_VGEtUHVycGxlbGFi__search1_1707475066.37, exit=0

the error seems to come from the fact that the argument went wrong: 

02-09-2024 10:37:46.162 INFO  ChunkedExternProcessor [10449 searchOrchestrator] - Running process: /opt/splunk/bin/python3.7 /opt/splunk/etc/apps/Ta-Purplelab/bin/mitrepurplelab.py
02-09-2024 10:37:46.232 ERROR ChunkedExternProcessor [10449 searchOrchestrator] - Failed attempting to parse transport header: Usage: python script.py <technique_id>
02-09-2024 10:37:46.239 ERROR ChunkedExternProcessor [10449 searchOrchestrator] - Error in 'mitrepurplelab' command: External search command exited unexpectedly.

I don't understand why, because you can see that the argument is well transmitted to the custom command.

and I can't retrieve the information about what is transmitted as an argument to the python script by the custom command

If you have any ideas, it would be a great help!

[ITWhisperer]

Does the argument need to be in quotes or passed as a field (so the SPL parser doesn't look for a field called T1059.003 and not find it so passes null?

<query>| makeresults | eval technique_id="$technique_id$" | where isnotnull(technique_id) | mitrepurplelab "$technique_id$"</query>

<query>| makeresults | eval technique_id="$technique_id$" | where isnotnull(technique_id) | mitrepurplelab technique_id</query>

[mcayrol]

It's not that because in the search log we saw that the $technique_id$ is well pass (T1059.003)

02-09-2024 10:37:46.161 INFO  SearchParser [10449 searchOrchestrator] - PARSING: | makeresults | eval technique_id="T1059.003" | where isnotnull(technique_id) | mitrepurplelab T1059.003

 And even when i'm doing this command, I have the same issue : 

| mitrepurplelab T1059.003

I think the issue is with the commands.conf

When i put command.arg.1 = $technique_id$ on the commands.conf the script try to run with $technique_id$ as an argument but literraly $technique_id$ not 1059.003 so It doesn't work 

[ITWhisperer]

What does your mitrepurplelab.log show as being passed in argv?

logging.basicConfig(filename='mitrepurplelab.log', level=logging.DEBUG, format='%(asctime)s - %(levelname)s - %(message)s')
def main():
logging.debug(f "Arguments received: {sys.argv}")

[mcayrol]

ok so when i'm adding this to commands.conf : 

command.arg.1 = T1059.003

The script work well but with the argument is fixed, we don't want that

Yeah i set mitrepurplelab.log to have more information and it is interesting because when I  do : 

| mitrepurplelab T1059.003

I have : 

2024-02-09 13:29:43,221 - DEBUG - Arguments reçus: ['/opt/splunk/etc/apps/Ta-Purplelab/bin/mitrepurplelab.py']
2024-02-09 13:29:43,221 - ERROR - Usage incorrect: python script.py <technique_id>

Like the T1059.003 was not pass 

And when I launch the script by the dashboard I have the same output. 

But when I remove chunked = true  and add 

enableheader = true
outputheader = true
requires_srinfo = true
supports_getinfo = true
supports_multivalues = true
supports_rawargs = true
python.version = python3

To commands.conf

I have this ouput :

2024-02-09 13:43:38,870 - DEBUG - Arguments reçus: ['/opt/splunk/etc/apps/Ta-Purplelab/bin/mitrepurplelab.py', '__GETINFO__', 'technique_id']
2024-02-09 13:43:38,870 - ERROR - Usage incorrect: python script.py <technique_id>

We're getting close... 

[ITWhisperer]

What did you get in the mitrepurplelab.log when you tried

<query>| makeresults | eval technique_id="$technique_id$" | where isnotnull(technique_id) | mitrepurplelab "$technique_id$"</query>

and

<query>| makeresults | eval technique_id="$technique_id$" | where isnotnull(technique_id) | mitrepurplelab technique_id</query>

[mcayrol]

I test 

<query>| makeresults | eval technique_id="$technique_id$" | where isnotnull(technique_id) | mitrepurplelab "$technique_id$"</query>

and I got : 

2024-02-09 14:24:52,100 - DEBUG - Arguments reçus: ['/opt/splunk/etc/apps/Ta-Purplelab/bin/mitrepurplelab.py', '__GETINFO__', '"T1059.003"']
2024-02-09 14:24:52,100 - ERROR - Usage incorrect: python script.py <technique_id>

This time the Tehnique is well retreive but the syntax is not correct for the script I guess 

[ITWhisperer]

Try removing the extra stuff you put in and set chunked to true again

[mcayrol]

When I do that I have this again

2024-02-09 14:39:40,578 - DEBUG - Arguments reçus: ['/opt/splunk/etc/apps/Ta-Purplelab/bin/mitrepurplelab.py']
2024-02-09 14:39:40,578 - ERROR - Usage incorrect: python script.py <technique_id>

[mitrepurplelab]

chunked = true
python.version = python3
filename = mitrepurplelab.py

[ITWhisperer]

Try adding 

supports_rawargs = true

Other than that, do you have any documentation for the mitrepurplelab custom command that would indicate what values should be there? 

[mcayrol]

I've found a workaround in the meantime

Since I know what I'm getting, I clean up the arguments before loading them into my python script