Splunk dashboard design

[Karthikeya]

I have a requirement to create a dashboard with following Json data:

all_request_headers: { [-]
     Accept: */*
     Content-Length: 0
     Content-Type: text/plain
     Cookie: Cookie1=Salvin
     Host: wasphictst-wdc.hc.cloud.uk.sony
     User-Agent: insomnia/2021.5.3
   }
   all_response_headers: { [-]
     Connection: keep-alive
     Content-Length: 196
     Content-Type: text/html; charset=iso-8859-1
     Date: Fri, 14 Feb 2025 15:51:13 GMT
     Server: Apache/2.4.37 (Red Hat Enterprise Linux)
     Strict-Transport-Security: max-age=31536000; includeSubDomains
   }

waf_log: { [-]
     allowlist_configured: false
     allowlist_processed: false
     application_rules_configured: false
     application_rules_processed: false
     latency_request_body_phase: 1544
     latency_request_header_phase: 351
     latency_response_body_phase: 15
     latency_response_header_phase: 50
     memory_allocated: 71496
     omitted_app_rule_stats: { [+]
     }
     omitted_signature_stats: { [+]
     }
     psm_configured: false
     psm_processed: false
     rules_configured: true
     rules_processed: true
     status: PASSED
   }

Fields are getting auto extracted like waf_log.allowlist_configured ... etc.

They want a neat dashboard for request headers, response headers, waf log details etc. How to create this dashboard. I am confused. If we create based on fields then there will be so many panels right.

[ITWhisperer]

Start with your requirements. This is a very imprecise requirement. What does "neat" mean? What information are you being asked to show? Next, start creating searches to calculate that information from your logs. Then choose a way to display that information in an informative way. Next, if you want some more help here, please post your raw events i.e. unformatted, preferably using a code block using the </> button above. That way we can simulate your situation and suggest some searches.

[Karthikeya]

@ITWhisperer please find the raw event.

{ [-]
   adf: true
   all_request_headers: {[-]
     Accept: */*
     Content-Length: 0
     Content-Type: text/plain
     Cookie: Cookie1=Salvin
     Host: wasphictst-wdc.hc.cloud.uk.sony

    User-Agent: insomnia/2021.5.3
   }
   all_response_headers: { [-]
     Connection: keep-alive
     Content-Length: 196
     Content-Type: text/html; charset=iso-8859-1
     Date: Fri, 14 Feb 2025 15:51:13 GMT
     Server: Apache/2.4.37 (Red Hat Enterprise Linux)
     Strict-Transport-Security: max-age=31536000; includeSubDomains
   }
   avg_ingress_latency_be: 0
   avg_ingress_latency_fe: 0
   cacheable: true
   client_dest_port: 443
   client_insights:
   client_ip: 128.168.178.113
   client_rtt: 1
   client_src_port: 24487
   compression: NO_COMPRESSION_CAN_BE_COMPRESSED
   compression_percentage: 0
   conn_est_time_be: 0
   conn_est_time_fe: 0
   headers_received_from_server: {[+]
   }
   headers_sent_to_server: { [+] 

}
   host: xyz
   http_version: 1.1
   jwt_log: { [+]
   }
   log_id: 108692
   max_ingress_latency_be: 0
   max_ingress_latency_fe: 0
   method: GET
   persistent_session_id: 3472328296900352087
   pool: pool-cac2726e-acd1-4225-8ac8-72ebd82a57a6
   pool_name: xxxxx
   report_timestamp: 2025-02-14T15:51:13.176715Z
   request_content_type: text/plain
   request_headers: 833
   request_id: 9mY-Spaj-RgC9
   request_length: 193
   request_state: AVI_HTTP_REQUEST_STATE_SEND_RESPONSE_BODY_TO_CLIENT
   response_code: 404
   response_content_type: text/html; charset=iso-8859-1
   response_headers: 13
   response_length: 6148
   response_time_first_byte: 81
   response_time_last_byte: 81
   rewritten_uri_query: test=%26%26%20whoami
   server_conn_src_ip: 128.160.77.235
   server_dest_port: 80
   server_ip: 128.160.88.68
   server_name: 128.160.88.68
   server_response_code: 404
   server_response_length: 373
   server_response_time_first_byte: 78
   server_response_time_last_byte: 81
   server_rtt: 3
   server_src_port: 25921
   servers_tried: 1
   service_engine: GB-DRN-AB-Tier2-se-bmqhk
   significant: 0
   significant_log: [ [+]
   ]
   sni_hostname: xyx
   source_ip: xxxxxx
   ssl_cipher: TLS_AES_256_GCM_SHA384
   ssl_session_id: bc586cf2272c7130a6e90551566bf12c
   ssl_version: TLSv1.3
   tenant_name: admin
   udf: false
   uri_path: /cmd
   uri_query: test=&& whoami
   user_agent: insomnia/2021.5.3
   vcpu_id: 0
   virtualservice: virtualservice-e52d1117-b508-4a6d-9fb5-f03ca6319af7
   vs_ip: 128.160.71.101
   vs_name: xxx-443
   waf_log: { [-]
     allowlist_configured: false
     allowlist_processed: false
     application_rules_configured: false
     application_rules_processed: false
     latency_request_body_phase: 1544
     latency_request_header_phase: 351
     latency_response_body_phase: 15
     latency_response_header_phase: 50
     memory_allocated: 71496
     omitted_app_rule_stats: {[+]
     }
     omitted_signature_stats: {[+]
     }
     psm_configured: false
     psm_processed: false
     rules_configured: true
     rules_processed: true
     status: PASSED
   }
}

They want all_request_headers, all_response_headers, waf_log details to be viewed in a dashboard manner and any other important panels which makes sense.

[ITWhisperer]

This is not the raw unformatted event - it has been prettied for display. It should look something like 

{ 
   "adf": true,
   "all_request_headers": {
     "Accept": "*/*",
     "Content-Length": 0,

Please repost your event in this style

[Karthikeya]

{"adf":true,"significant":0,"udf":false,"virtualservice":"virtualservice-e52d1117-b508-4a6d-9fb5-f03ca6319af7","report_timestamp":"2025-02-14T15:51:13.176715Z","service_engine":"GB-DRN-AB-Tier2-se-bmqhk","vcpu_id":0,"log_id":108692,"client_ip":"128.168.178.113","client_src_port":24487,"client_dest_port":443,"client_rtt":1,"ssl_session_id":"bc586cf2272c7130a6e90551566bf12c","ssl_version":"TLSv1.3","ssl_cipher":"TLS_AES_256_GCM_SHA384","sni_hostname":"wasphictst-wdc.hc.cloud.uk.sony","http_version":"1.1","method":"GET","uri_path":"/cmd","uri_query":"test=&& whoami","rewritten_uri_query":"test=%26%26%20whoami","user_agent":"insomnia/2021.5.3","host":"wasphictst-wdc.hc.cloud.uk.sony","persistent_session_id":3472328296900352087,"request_content_type":"text/plain","response_content_type":"text/html; charset=iso-8859-1","request_length":193,"cacheable":true,"pool":"pool-cac2726e-acd1-4225-8ac8-72ebd82a57a6","pool_name":"p-wasphictst-wdc.hc.cloud.uk.sony-wdc-443","server_ip":"128.160.88.68","server_name":"128.160.88.68","server_conn_src_ip":"128.160.77.235","server_dest_port":80,"server_src_port":25921,"server_rtt":3,"server_response_length":373,"server_response_code":404,"server_response_time_first_byte":78,"server_response_time_last_byte":81,"response_length":6148,"response_code":404,"response_time_first_byte":81,"response_time_last_byte":81,"compression_percentage":0,"compression":"NO_COMPRESSION_CAN_BE_COMPRESSED","client_insights":"","request_headers":833,"response_headers":13,"request_state":"AVI_HTTP_REQUEST_STATE_SEND_RESPONSE_BODY_TO_CLIENT","all_request_headers":{"Host":"wasphictst-wdc.hc.cloud.uk.sony","User-Agent":"insomnia/2021.5.3","Cookie":"Cookie1=Jijin","Content-Type":"text/plain","Accept":"*/*","Content-Length":0},"all_response_headers":{"Content-Type":"text/html; charset=iso-8859-1","Content-Length":196,"Connection":"keep-alive","Date":"Fri, 14 Feb 2025 15:51:13 GMT","Server":"Apache/2.4.37 (Red Hat Enterprise Linux)","Strict-Transport-Security":"max-age=31536000; includeSubDomains"},"significant_log":["ADF_HTTP_CONTENT_LENGTH_HDR_WITH_UNSUPPORTED_METHOD","ADF_RESPONSE_CODE_4XX"],"headers_sent_to_server":{"X-Forwarded-For":"128.168.178.113","Host":"wasphictst-wdc.hc.cloud.uk.sony","Content-Length":0,"User-Agent":"insomnia/2021.5.3","Cookie":"Cookie1=Jijin","Content-Type":"text/plain","Accept":"*/*","X-Forwarded-Proto":"https"},"headers_received_from_server":{"Date":"Fri, 14 Feb 2025 15:51:13 GMT","Server":"Apache/2.4.37 (Red Hat Enterprise Linux)","Content-Length":196,"Content-Type":"text/html; charset=iso-8859-1"},"vs_ip":"128.160.71.101","waf_log":{"status":"PASSED","latency_request_header_phase":351,"latency_request_body_phase":1544,"latency_response_header_phase":50,"latency_response_body_phase":15,"rules_configured":true,"psm_configured":false,"application_rules_configured":false,"allowlist_configured":false,"allowlist_processed":false,"rules_processed":true,"psm_processed":false,"application_rules_processed":false,"memory_allocated":71496,"omitted_signature_stats":{"rules":0,"match_elements":0},"omitted_app_rule_stats":{"rules":0,"match_elements":0}},"request_id":"9mY-Spaj-RgC9","servers_tried":1,"jwt_log":{"is_jwt_verified":false},"max_ingress_latency_fe":0,"avg_ingress_latency_fe":0,"conn_est_time_fe":0,"max_ingress_latency_be":0,"avg_ingress_latency_be":0,"conn_est_time_be":0,"source_ip":"128.168.178.113","vs_name":"v-wasphictst-wdc.hc.cloud.uk.sony-443","tenant_name":"admin"}

[livehybrid]

Thanks for providing the raw example. It looks like some of the header fields have quite high entropy, meaning that it could create a lot of values for a dashboard/table.

Are they wanting to see rare or most frequent values for these headers, perhaps? 
Presume there are some headers such as "Date" which arent going to add much value?

As previously mentioned - I think its important to understand the purpose of the dashboard, otherwise the panels created might be meaningless and a waste of search time.

Please let me know how you get on and consider accepting this answer or adding karma this answer if it has helped.
Regards

Will

[ITWhisperer]

You could start with a dashboard with a couple of statistics tables using the following searches (assuming your events have already been ingested as JSON)

| fields _time all_request_headers
| spath input=all_request_headers
| fields - _raw all_request_headers

| fields _time all_response_headers
| spath input=all_response_headers
| fields - _raw all_response_headers

| fields _time waf_log
| spath input=waf_log
| fields - _raw waf_log

Or you could combine them into a single table

| fields _time all_request_headers all_response_headers waf_log
| spath input=all_request_headers
| spath input=all_response_headers
| spath input=waf_log
| fields - _raw all_request_headers all_response_headers waf_log

To be honest, these won't be very useful but it is what you/they asked for and might help you clarify what exactly they do want to see in the dashboard.

[Karthikeya]

| fields _time all_request_headers
| spath input=all_request_headers
| fields - _raw all_request_headers

Giving this search after my index and sourcetype, it is showing nothing in events, can you please help @ITWhisperer  

[ITWhisperer]

As I said, assuming your events have already been ingested as JSON. It looks like they aren't, or at least the fields you need aren't. Try this

| spath all_request_headers
| fields _time all_request_headers
| spath input=all_request_headers
| fields - _raw all_request_headers

[Karthikeya]

After giving this in search how to create a dashboard with single panel including all request headers?

[ITWhisperer]

Try using the Save as button and save it to a dashboard. There is a tutorial on how to do this https://docs.splunk.com/Documentation/Splunk/latest/SearchTutorial/WelcometotheSearchTutorial

[Karthikeya]

I understood that but once it added to dashboard and drilldown enabled for respective field values, if users click on any of the value search should be there. But here when I give above search it is empty events just showing time. I removed _raw from fields - . But I want to understand why we given spath command here?

[ITWhisperer]

Please share your dashboard configuration source

[livehybrid]

Hi @Karthikeya 

It sounds like you might need to work with the dashboard users to understand exactly what they want out of the dashboard - what is their main goal when they look at the dashboard? We do not want to overwhelm the users with charts they cannot read or make sense of. 

I'd start by understanding the main purpose of the dashboard, and then the top 3-4 statistics or details they want to be able to see.

Please let me know how you get on and consider accepting this answer or adding karma this answer if it has helped.
Regards

Will