topic Re: [Error] Pass token values from a dashboard as parameters to python script in Dashboards & Visualizations

[Error] Pass token values from a dashboard as parameters to python script

mcayrol — Fri, 09 Feb 2024 11:02:21 GMT

Hello,

I created a dashbord with a text input, the token is then passed to a panel that executes this command:

<query>| makeresults | eval technique_id="$technique_id$" | where isnotnull(technique_id) | mitrepurplelab $technique_id$</query>

the purpose of this command is to trigger a custom command with this config:

[mitrepurplelab] filename = mitrepurplelab.py enableheader = true outputheader = true requires_srinfo = true chunked = true streaming = true

the mitrepurplelab.py script is then triggered, here is its code:

import sys import requests import logging logging.basicConfig(filename='mitrepurplelab.log', level=logging.DEBUG, format='%(asctime)s - %(levelname)s - %(message)s') def main(): logging.debug(f "Arguments received: {sys.argv}") if len(sys.argv) != 2: logging.error("Incorrect usage: python script.py <technique_id>") print("Usage: python script.py <technique_id>") return technique_id = sys.argv[1] url = "http://192.168.142.146:5000/api/mitre_attack_execution" # Make sure your JWT token is complete and correctly formatted token = "token headers = { "Authorization": f "Bearer {token}" } params = { "technique_id": technique_id } response = requests.post(url, headers=headers, params=params) if response.status_code == 200: print("Request successful!") print("Server response:") print(response.json()) else: logging.error(f "Error: {response.status_code}, Response body: {response.text}") print(f "Error: {response.status_code}, Response body: {response.text}") if __name__ == "__main__": main()

the script works well when run by hand, for example :

python3 bin/mitrepurplelab.py T1059.003

but when I execute it via the dashboard I get this error:

in the panel search.log I get this:

02-09-2024 10:37:46.075 INFO dispatchRunner [1626 MainThread] - Search process mode: preforked (reused process by new user) (build 1fff88043d5f). 02-09-2024 10:37:46.075 INFO dispatchRunner [1626 MainThread] - registering build time modules, count=1 02-09-2024 10:37:46.075 INFO dispatchRunner [1626 MainThread] - registering search time components of build time module name=vix 02-09-2024 10:37:46.076 INFO BundlesSetup [1626 MainThread] - Setup stats for /opt/splunk/etc: wallclock_elapsed_msec=7, cpu_time_used=0.00727909, shared_services_generation=2, shared_services_population=1 02-09-2024 10:37:46.080 INFO UserManagerPro [1626 MainThread] - Load authentication: forcing roles="admin, power, user" 02-09-2024 10:37:46.080 INFO UserManager [10446 RunDispatch] - Setting user context: splunk-system-user 02-09-2024 10:37:46.080 INFO UserManager [10446 RunDispatch] - Done setting user context: NULL -> splunk-system-user 02-09-2024 10:37:46.080 INFO UserManager [10446 RunDispatch] - Unwound user context: splunk-system-user -> NULL 02-09-2024 10:37:46.080 INFO UserManager [10446 RunDispatch] - Setting user context: admin 02-09-2024 10:37:46.080 INFO UserManager [10446 RunDispatch] - Done setting user context: NULL -> admin 02-09-2024 10:37:46.080 INFO dispatchRunner [10446 RunDispatch] - search context: user="admin", app="Ta-Purplelab", bs-pathname="/opt/splunk/etc" 02-09-2024 10:37:46.080 INFO SearchParser [10446 RunDispatch] - PARSING: | makeresults | eval technique_id="T1059.003" | where isnotnull(technique_id) | mitrepurplelab T1059.003 02-09-2024 10:37:46.081 INFO dispatchRunner [10446 RunDispatch] - Search running in non-clustered mode 02-09-2024 10:37:46.081 INFO dispatchRunner [10446 RunDispatch] - SearchHeadInitSearchMs=0 02-09-2024 10:37:46.081 INFO dispatchRunner [10446 RunDispatch] - Executing the Search orchestrator and iterator model (dfs=false). 02-09-2024 10:37:46.081 INFO SearchOrchestrator [10446 RunDispatch] - SearchOrchestrator is constructed. sid=admin__admin_VGEtUHVycGxlbGFi__search1_1707475066.37, eval_only=0 02-09-2024 10:37:46.081 INFO SearchOrchestrator [10446 RunDispatch] - Initialized the SRI 02-09-2024 10:37:46.081 INFO SearchFeatureFlags [10446 RunDispatch] - Initializing feature flags from config. feature_seed=2135385444 02-09-2024 10:37:46.081 INFO SearchFeatureFlags [10446 RunDispatch] - Setting feature_flag=parallelreduce:enablePreview:true 02-09-2024 10:37:46.081 INFO SearchFeatureFlags [10446 RunDispatch] - Setting feature_flag=search:search_retry:false 02-09-2024 10:37:46.081 INFO SearchFeatureFlags [10446 RunDispatch] - Setting feature_flag=search:search_retry_realtime:false 02-09-2024 10:37:46.081 INFO SearchFeatureFlags [10446 RunDispatch] - Setting feature_flag=parallelreduce:autoAppliedPercentage:false 02-09-2024 10:37:46.081 INFO SearchFeatureFlags [10446 RunDispatch] - Setting feature_flag=subsearch:enableConcurrentPipelineProcessing:false 02-09-2024 10:37:46.081 INFO SearchFeatureFlags [10446 RunDispatch] - Setting feature_flag=subsearch:concurrent_pipeline_adhoc:false 02-09-2024 10:37:46.081 INFO SearchFeatureFlags [10446 RunDispatch] - Setting feature_flag=append:support_multiple_data_sources:false 02-09-2024 10:37:46.081 INFO SearchFeatureFlags [10446 RunDispatch] - Setting feature_flag=join:support_multiple_data_sources:false 02-09-2024 10:37:46.081 INFO SearchFeatureFlags [10446 RunDispatch] - Setting feature_flag=search_optimization::set_required_fields:stats:false 02-09-2024 10:37:46.081 INFO SearchFeatureFlags [10446 RunDispatch] - Setting feature_flag=searchresults:srs2:false 02-09-2024 10:37:46.081 INFO SearchFeatureFlags [10446 RunDispatch] - Setting feature_flag=search:read_final_results_from_timeliner:true 02-09-2024 10:37:46.081 INFO SearchFeatureFlags [10446 RunDispatch] - Setting feature_flag=search:fetch_remote_search_telemetry:true 02-09-2024 10:37:46.081 INFO SearchFeatureFlags [10446 RunDispatch] - Setting feature_flag=testing:boolean_flag:false 02-09-2024 10:37:46.081 INFO SearchFeatureFlags [10446 RunDispatch] - Setting feature_flag=testing:percent_flag:true 02-09-2024 10:37:46.081 INFO SearchFeatureFlags [10446 RunDispatch] - Setting feature_flag=testing:legacy_flag:true 02-09-2024 10:37:46.081 INFO SearchOrchestrator [10446 RunDispatch] - Search feature_flags={"v":1,"enabledFeatures":["parallelreduce:enablePreview","search:read_final_results_from_timeliner","search:fetch_remote_search_telemetry","testing:percent_flag","testing:legacy_flag"],"disabledFeatures":["search:search_retry","search:search_retry_realtime","parallelreduce:autoAppliedPercentage","subsearch:enableConcurrentPipelineProcessing","subsearch:concurrent_pipeline_adhoc","append:support_multiple_data_sources","join:support_multiple_data_sources","search_optimization::set_required_fields:stats","searchresults:srs2","testing:boolean_flag"]} 02-09-2024 10:37:46.081 INFO ISplunkDispatch [10446 RunDispatch] - Not running in splunkd. Bundle replication not triggered. 02-09-2024 10:37:46.081 INFO SearchOrchestrator [10449 searchOrchestrator] - Initialzing the run time settings for the orchestrator. 02-09-2024 10:37:46.081 INFO UserManager [10449 searchOrchestrator] - Setting user context: admin 02-09-2024 10:37:46.081 INFO UserManager [10449 searchOrchestrator] - Done setting user context: NULL -> admin 02-09-2024 10:37:46.081 INFO AdaptiveSearchEngineSelector [10449 searchOrchestrator] - Search execution_plan=classic 02-09-2024 10:37:46.082 INFO SearchOrchestrator [10449 searchOrchestrator] - Creating the search DAG. 02-09-2024 10:37:46.082 INFO SearchParser [10449 searchOrchestrator] - PARSING: | makeresults | eval technique_id="T1059.003" | where isnotnull(technique_id) | mitrepurplelab T1059.003 02-09-2024 10:37:46.082 INFO DispatchStorageManagerInfo [10449 searchOrchestrator] - Successfully created new dispatch directory for search job. sid=dc5edf3eebc8ccb6_tmp dispatch_dir=/opt/splunk/var/run/splunk/dispatch/dc5edf3eebc8ccb6_tmp 02-09-2024 10:37:46.082 INFO SearchParser [10449 searchOrchestrator] - PARSING: premakeresults 02-09-2024 10:37:46.082 INFO DispatchThread [10449 searchOrchestrator] - BatchMode: allowBatchMode: 1, conf(1): 1, timeline/Status buckets(0):0, realtime(0):0, report pipe empty(0):0, reqTimeOrder(0):0, summarize(0):0, statefulStreaming(0):0 02-09-2024 10:37:46.082 INFO DispatchThread [10449 searchOrchestrator] - required fields list to add to remote search = * 02-09-2024 10:37:46.082 INFO DispatchCommandProcessor [10449 searchOrchestrator] - summaryHash=f2df6493ea859e37 summaryId=A6ADAC30-27EC-4F28-BEB9-3BD2C7EC3E53_Ta-Purplelab_admin_f2df6493ea859e37 remoteSearch=premakeresults 02-09-2024 10:37:46.082 INFO DispatchCommandProcessor [10449 searchOrchestrator] - summaryHash=NSf2df6493ea859e37 summaryId=A6ADAC30-27EC-4F28-BEB9-3BD2C7EC3E53_Ta-Purplelab_admin_NSf2df6493ea859e37 remoteSearch=premakeresults 02-09-2024 10:37:46.082 INFO DispatchThread [10449 searchOrchestrator] - Getting summary ID for summaryHash=NSf2df6493ea859e37 02-09-2024 10:37:46.084 INFO DispatchThread [10449 searchOrchestrator] - Did not find a usable summary_id, setting info._summary_mode=none, not modifying input summary_id=A6ADAC30-27EC-4F28-BEB9-3BD2C7EC3E53_Ta-Purplelab_admin_NSf2df6493ea859e37 02-09-2024 10:37:46.085 INFO SearchParser [10449 searchOrchestrator] - PARSING: | makeresults | eval technique_id="T1059.003" | where isnotnull(technique_id) | mitrepurplelab T1059.003 02-09-2024 10:37:46.085 INFO ChunkedExternProcessor [10449 searchOrchestrator] - Running process: /opt/splunk/bin/python3.7 /opt/splunk/etc/apps/Ta-Purplelab/bin/mitrepurplelab.py 02-09-2024 10:37:46.155 ERROR ChunkedExternProcessor [10449 searchOrchestrator] - Failed attempting to parse transport header: Usage: python script.py <technique_id> 02-09-2024 10:37:46.161 ERROR ChunkedExternProcessor [10449 searchOrchestrator] - Error in 'mitrepurplelab' command: External search command exited unexpectedly. 02-09-2024 10:37:46.161 INFO ScopedTimer [10449 searchOrchestrator] - search.optimize 0.076785640 02-09-2024 10:37:46.161 WARN SearchPhaseGenerator [10449 searchOrchestrator] - AST processing error, exception=31SearchProcessorMessageException, error=Error in 'mitrepurplelab' command: External search command exited unexpectedly.. Fall back to 2 phase. 02-09-2024 10:37:46.161 INFO SearchPhaseGenerator [10449 searchOrchestrator] - Executing two phase fallback for the search=| makeresults | eval technique_id="T1059.003" | where isnotnull(technique_id) | mitrepurplelab T1059.003 02-09-2024 10:37:46.161 INFO SearchParser [10449 searchOrchestrator] - PARSING: | makeresults | eval technique_id="T1059.003" | where isnotnull(technique_id) | mitrepurplelab T1059.003 02-09-2024 10:37:46.162 INFO ChunkedExternProcessor [10449 searchOrchestrator] - Running process: /opt/splunk/bin/python3.7 /opt/splunk/etc/apps/Ta-Purplelab/bin/mitrepurplelab.py 02-09-2024 10:37:46.232 ERROR ChunkedExternProcessor [10449 searchOrchestrator] - Failed attempting to parse transport header: Usage: python script.py <technique_id> 02-09-2024 10:37:46.239 ERROR ChunkedExternProcessor [10449 searchOrchestrator] - Error in 'mitrepurplelab' command: External search command exited unexpectedly. 02-09-2024 10:37:46.239 ERROR SearchPhaseGenerator [10449 searchOrchestrator] - Fallback to two phase failed with SearchProcessorException: Error in 'mitrepurplelab' command: External search command exited unexpectedly. 02-09-2024 10:37:46.239 WARN SearchPhaseGenerator [10449 searchOrchestrator] - Failed to create search phases: exception=31SearchProcessorMessageException, error=Error in 'mitrepurplelab' command: External search command exited unexpectedly. 02-09-2024 10:37:46.240 INFO SearchStatusEnforcer [10449 searchOrchestrator] - sid=admin__admin_VGEtUHVycGxlbGFi__search1_1707475066.37, newState=BAD_INPUT_CANCEL, message=Error in 'mitrepurplelab' command: External search command exited unexpectedly. 02-09-2024 10:37:46.240 ERROR SearchStatusEnforcer [10449 searchOrchestrator] - SearchMessage orig_component=ChunkedExternProcessor sid=admin__admin_VGEtUHVycGxlbGFi__search1_1707475066.37 message_key=CHUNKED:UNEXPECTED_EXIT message=Error in 'mitrepurplelab' command: External search command exited unexpectedly. 02-09-2024 10:37:46.240 INFO SearchStatusEnforcer [10449 searchOrchestrator] - State changed to BAD_INPUT_CANCEL: Error in 'mitrepurplelab' command: External search command exited unexpectedly. 02-09-2024 10:37:46.240 INFO SearchStatusEnforcer [10449 searchOrchestrator] - Enforcing disk quota = 10485760000 02-09-2024 10:37:46.242 INFO DispatchManager [10449 searchOrchestrator] - DispatchManager::dispatchHasFinished(id='admin__admin_VGEtUHVycGxlbGFi__search1_1707475066.37', username='admin') 02-09-2024 10:37:46.242 INFO UserManager [10449 searchOrchestrator] - Unwound user context: admin -> NULL 02-09-2024 10:37:46.242 INFO SearchOrchestrator [10446 RunDispatch] - SearchOrchestrator is destructed. sid=admin__admin_VGEtUHVycGxlbGFi__search1_1707475066.37, eval_only=0 02-09-2024 10:37:46.242 INFO SearchStatusEnforcer [10446 RunDispatch] - SearchStatusEnforcer is already terminated 02-09-2024 10:37:46.242 INFO UserManager [10446 RunDispatch] - Unwound user context: admin -> NULL 02-09-2024 10:37:46.242 INFO LookupDataProvider [10446 RunDispatch] - Clearing out lookup shared provider map 02-09-2024 10:37:46.242 INFO dispatchRunner [1626 MainThread] - RunDispatch is done: sid=admin__admin_VGEtUHVycGxlbGFi__search1_1707475066.37, exit=0

the error seems to come from the fact that the argument went wrong:

02-09-2024 10:37:46.162 INFO ChunkedExternProcessor [10449 searchOrchestrator] - Running process: /opt/splunk/bin/python3.7 /opt/splunk/etc/apps/Ta-Purplelab/bin/mitrepurplelab.py 02-09-2024 10:37:46.232 ERROR ChunkedExternProcessor [10449 searchOrchestrator] - Failed attempting to parse transport header: Usage: python script.py <technique_id> 02-09-2024 10:37:46.239 ERROR ChunkedExternProcessor [10449 searchOrchestrator] - Error in 'mitrepurplelab' command: External search command exited unexpectedly.

I don't understand why, because you can see that the argument is well transmitted to the custom command.

and I can't retrieve the information about what is transmitted as an argument to the python script by the custom command

If you have any ideas, it would be a great help!

Re: [Error] Pass token values from a dashboard as parameters to python script

ITWhisperer — Fri, 09 Feb 2024 11:35:25 GMT

Does the argument need to be in quotes or passed as a field (so the SPL parser doesn't look for a field called T1059.003 and not find it so passes null?

<query>| makeresults | eval technique_id="$technique_id$" | where isnotnull(technique_id) | mitrepurplelab "$technique_id$"</query>

<query>| makeresults | eval technique_id="$technique_id$" | where isnotnull(technique_id) | mitrepurplelab technique_id</query>

Re: [Error] Pass token values from a dashboard as parameters to python script

mcayrol — Fri, 09 Feb 2024 13:05:11 GMT

It's not that because in the search log we saw that the $technique_id$ is well pass (T1059.003)

02-09-2024 10:37:46.161 INFO SearchParser [10449 searchOrchestrator] - PARSING: | makeresults | eval technique_id="T1059.003" | where isnotnull(technique_id) | mitrepurplelab T1059.003

And even when i'm doing this command, I have the same issue :

| mitrepurplelab T1059.003

I think the issue is with the commands.conf

When i put command.arg.1 = $technique_id$ on the commands.conf the script try to run with $technique_id$ as an argument but literraly $technique_id$ not 1059.003 so It doesn't work

Re: [Error] Pass token values from a dashboard as parameters to python script

ITWhisperer — Fri, 09 Feb 2024 13:18:01 GMT

What does your mitrepurplelab.log show as being passed in argv?

logging.basicConfig(filename='mitrepurplelab.log', level=logging.DEBUG, format='%(asctime)s - %(levelname)s - %(message)s') def main(): logging.debug(f "Arguments received: {sys.argv}")

Re: [Error] Pass token values from a dashboard as parameters to python script

mcayrol — Fri, 09 Feb 2024 13:44:59 GMT

ok so when i'm adding this to commands.conf :

command.arg.1 = T1059.003

The script work well but with the argument is fixed, we don't want that

Yeah i set mitrepurplelab.log to have more information and it is interesting because when I do :

| mitrepurplelab T1059.003

I have :

2024-02-09 13:29:43,221 - DEBUG - Arguments reçus: ['/opt/splunk/etc/apps/Ta-Purplelab/bin/mitrepurplelab.py'] 2024-02-09 13:29:43,221 - ERROR - Usage incorrect: python script.py <technique_id>

Like the T1059.003 was not pass

And when I launch the script by the dashboard I have the same output.

But when I remove chunked = true and add

enableheader = true outputheader = true requires_srinfo = true supports_getinfo = true supports_multivalues = true supports_rawargs = true python.version = python3

To commands.conf

I have this ouput :

2024-02-09 13:43:38,870 - DEBUG - Arguments reçus: ['/opt/splunk/etc/apps/Ta-Purplelab/bin/mitrepurplelab.py', '__GETINFO__', 'technique_id'] 2024-02-09 13:43:38,870 - ERROR - Usage incorrect: python script.py <technique_id>

We're getting close...

Re: [Error] Pass token values from a dashboard as parameters to python script

ITWhisperer — Fri, 09 Feb 2024 14:17:54 GMT

What did you get in the mitrepurplelab.log when you tried

<query>| makeresults | eval technique_id="$technique_id$" | where isnotnull(technique_id) | mitrepurplelab "$technique_id$"</query>

and

<query>| makeresults | eval technique_id="$technique_id$" | where isnotnull(technique_id) | mitrepurplelab technique_id</query>

Re: [Error] Pass token values from a dashboard as parameters to python script

mcayrol — Fri, 09 Feb 2024 14:26:44 GMT

I test

<query>| makeresults | eval technique_id="$technique_id$" | where isnotnull(technique_id) | mitrepurplelab "$technique_id$"</query>

and I got :

2024-02-09 14:24:52,100 - DEBUG - Arguments reçus: ['/opt/splunk/etc/apps/Ta-Purplelab/bin/mitrepurplelab.py', '__GETINFO__', '"T1059.003"'] 2024-02-09 14:24:52,100 - ERROR - Usage incorrect: python script.py <technique_id>

This time the Tehnique is well retreive but the syntax is not correct for the script I guess

Re: [Error] Pass token values from a dashboard as parameters to python script

ITWhisperer — Fri, 09 Feb 2024 14:29:16 GMT

Try removing the extra stuff you put in and set chunked to true again

Re: [Error] Pass token values from a dashboard as parameters to python script

mcayrol — Fri, 09 Feb 2024 14:40:48 GMT

When I do that I have this again

2024-02-09 14:39:40,578 - DEBUG - Arguments reçus: ['/opt/splunk/etc/apps/Ta-Purplelab/bin/mitrepurplelab.py'] 2024-02-09 14:39:40,578 - ERROR - Usage incorrect: python script.py <technique_id>

[mitrepurplelab] chunked = true python.version = python3 filename = mitrepurplelab.py

Re: [Error] Pass token values from a dashboard as parameters to python script

mcayrol — Fri, 09 Feb 2024 14:58:22 GMT

I've found a workaround in the meantime

Since I know what I'm getting, I clean up the arguments before loading them into my python script

Re: [Error] Pass token values from a dashboard as parameters to python script

ITWhisperer — Fri, 09 Feb 2024 15:05:36 GMT

Try adding

supports_rawargs = true

Other than that, do you have any documentation for the mitrepurplelab custom command that would indicate what values should be there?