Editing Dashboard and get this error 'Unexpected close tag'

[jwhughes58]

I've been asked to update 'Imperva Database Audit Analysis' and I'm running into issues trying to update the Audit Dashboard.  The sanitized data looks like this

Nov 10 23:20:52 syslog_server {"header":"Imperva Inc.|SecureSphere|version|Audit|Audit.DAM|Informative|", "dest-ip":"dip_address", "db-user":"db_user", "source-ip":"sip_address", "real-time":"Nov 10 2020 23:20:51 GMT",  "audit-policy":"["Policy - Policy_Name - Login/Logout"]", "server-group":"server_group", "service-name":"service_name", "application-name":"application_name", "source-application":"source_application", "os-user":"os_user", "host-name":"fqdn", "sql-error":"", "mx-ip":"mx_ip_address", "gw-ip":"gw_ip_address", "objects-list":"[]", "operation-name":"Login",  "schema-name":"schema_name", "object-name":"${Event.struct.operations.objects.name}", "agent-name":"agent_name", "success":"True", "os-user-chain":"root-->user", "db-name":"db_name" }
Nov 10 23:20:52 syslog_server {"header":"Imperva Inc.|SecureSphere|version|Audit|Audit.DAM|Informative|", "dest-ip":"dip_address", "db-user":"db_user", "source-ip":"sip_address", "real-time":"Nov 10 2020 23:20:50 GMT",  "audit-policy":"["Global Policy - Login/Logout"]",        "server-group":"server_group", "service-name":"service_name", "application-name":"application_name", "source-application":"source_application", "os-user":"os_user", "host-name":"fqdn", "sql-error":"", "mx-ip":"mx_ip_address", "gw-ip":"gw_ip_address", "objects-list":"[]", "operation-name":"Login",  "schema-name":"schema_name", "object-name":"${Event.struct.operations.objects.name}", "agent-name":"agent_name", "success":"True", "os-user-chain":"", "db-name":"db_name" }
Nov 10 23:20:52 syslog_server {"header":"Imperva Inc.|SecureSphere|version|Audit|Audit.DAM|Informative|", "dest-ip":"dip_address", "db-user":"db_user", "source-ip":"sip_address", "real-time":"Nov 10 2020 23:20:51 GMT",  "audit-policy":"["Policy - Login/Logout - SQL"]",         "server-group":"server_group", "service-name":"service_name", "application-name":"application_name", "source-application":"source_application", "os-user":"",        "host-name":"fqdn", "sql-error":"", "mx-ip":"mx_ip_address", "gw-ip":"gw_ip_address", "objects-list":"[]", "operation-name":"Login",  "schema-name":"schema_name", "object-name":"${Event.struct.operations.objects.name}", "agent-name":"agent_name", "success":"True", "os-user-chain":"\-->user", "db-name":"db_name" }
Nov 10 23:20:52 syslog_server {"header":"Imperva Inc.|SecureSphere|version|Audit|Audit.DAM|Informative|", "dest-ip":"dip_address", "db-user":"db_user", "source-ip":"sip_address", "real-time":"Nov 10 2020 23:20:51 GMT",  "audit-policy":"["Policy - Policy_Name - Login/Logout"]", "server-group":"server_group", "service-name":"service_name", "application-name":"application_name", "source-application":"source_application", "os-user":"os_user", "host-name":"fqdn", "sql-error":"", "mx-ip":"mx_ip_address", "gw-ip":"gw_ip_address", "objects-list":"[]", "operation-name":"Login",  "schema-name":"schema_name", "object-name":"${Event.struct.operations.objects.name}", "agent-name":"agent_name", "success":"True", "os-user-chain":"root-->user", "db-name":"db_name" }
Nov 10 23:20:52 syslog_server {"header":"Imperva Inc.|SecureSphere|version|Audit|Audit.DAM|Informative|", "dest-ip":"dip_address", "db-user":"db_user", "source-ip":"sip_address", "real-time":"Nov 10 2020 23:20:51 GMT",  "audit-policy":"["Global Policy - Login/Logout"]",        "server-group":"server_group", "service-name":"service_name", "application-name":"application_name", "source-application":"source_application", "os-user":"os_user", "host-name":"fqdn", "sql-error":"", "mx-ip":"mx_ip_address", "gw-ip":"gw_ip_address", "objects-list":"[]", "operation-name":"Logout", "schema-name":"schema_name", "object-name":"${Event.struct.operations.objects.name}", "agent-name":"agent_name", "success":"True", "os-user-chain":"", "db-name":"db_name" }
Nov 10 23:20:52 syslog_server {"header":"Imperva Inc.|SecureSphere|version|Audit|Audit.DAM|Informative|", "dest-ip":"dip_address", "db-user":"db_user", "source-ip":"sip_address", "real-time":"Nov 10 2020 23:20:50 GMT",  "audit-policy":"["Policy - Login/Logout - SQL"]",         "server-group":"server_group", "service-name":"service_name", "application-name":"application_name", "source-application":"source_application", "os-user":"",        "host-name":"fqdn", "sql-error":"", "mx-ip":"mx_ip_address", "gw-ip":"gw_ip_address", "objects-list":"[]", "operation-name":"Login",  "schema-name":"schema_name", "object-name":"${Event.struct.operations.objects.name}", "agent-name":"agent_name", "success":"True", "os-user-chain":"\-->user", "db-name":"db_name" }
Nov 10 23:20:52 syslog_server {"header":"Imperva Inc.|SecureSphere|version|Audit|Audit.DAM|Informative|", "dest-ip":"dip_address", "db-user":"db_user", "source-ip":"sip_address", "real-time":"Nov 10 2020 23:20:51 GMT",  "audit-policy":"["Global Policy - Login/Logout"]",        "server-group":"server_group", "service-name":"service_name", "application-name":"application_name", "source-application":"source_application", "os-user":"os_user", "host-name":"fqdn", "sql-error":"", "mx-ip":"mx_ip_address", "gw-ip":"gw_ip_address", "objects-list":"[]", "operation-name":"Logout", "schema-name":"schema_name", "object-name":"object_name",                             "agent-name":"agent_name", "success":"True", "os-user-chain":"", "db-name":"db_name" }
Nov 10 23:20:52 syslog_server {"header":"Imperva Inc.|SecureSphere|version|Audit|Audit.DAM|Informative|", "dest-ip":"dip_address", "db-user":"db_user", "source-ip":"sip_address", "real-time":"Nov 10 2020 23:20:50 GMT", "audit-policy":"["Policy - Login/Logout - SQL"]", "server-group":"server_group", "service-name":"service_name", "application-name":"application_name", "source-application":"source_application", "os-user":"", "host-name":"fqdn", "sql-error":"", "mx-ip":"mx_ip_address", "gw-ip":"gw_ip_address", "objects-list":"[]", "operation-name":"Login", "schema-name":"schema_name", "object-name":"object_name", "agent-name":"agent_name", "success":"True", "os-user-chain":"\-->user", "db-name":"db_nme" }

Since Splunk doesn't handle embedded [] and {} in json, I created this search to process the events.

index=my_index sourcetype=source:type
| rex field=_raw "(?<st_json>\{.*)" 
| eval st_json_1=replace(st_json, "\"\[\]\"", "\"Null\"") 
| eval st_json=replace(st_json_1, "\"\[", "") 
| eval st_json_1=replace(st_json, "\]\"", "") 
| eval st_json=replace(st_json_1, "\$\{", "") 
| eval st_json_1=replace(st_json, "\}\",", "\",") 
| spath input=st_json_1 
| eval dest_ip_db_name= 'dest-ip'."\\".'db-name' 
| chart count by dest_ip_db_name 
| sort limit=10 -count 
| rename dest_ip_db_name AS "Database Host \ Database Name" count AS "Number Of Events"

This works.  When I move it to the dashboard I get the "Unexpected close tag".  This is the query in the dashboard.

          <query>index=my_index sourcetype=source:type | rex field=_raw "(?<st_json>\{.*)" | eval st_json_1=replace(st_json, "\"\[\]\"", "\"Null\"") | eval st_json=replace(st_json_1, "\"\[", "") | eval st_json_1=replace(st_json, "\]\"", "") | eval st_json=replace(st_json_1, "\$\{", "") | eval st_json_1=replace(st_json, "\}\",", "\",") | spath input=st_json_1 | eval dest_ip_db_name= 'dest-ip'."\\".'db-name' | chart count by dest_ip_db_name | sort limit=10 -count | rename dest_ip_db_name AS "Database Host \ Database Name" count AS "Number Of Events"</query>

I don't see anything that would cause the 'Unexpected close tag'.  Is there an issue with doing the \ escapes in SimpleXML or something else that I'm not aware of?

TIA,

Joe

[jwhughes58 · Accepted solution]

Duh!  I didn't see it until I posted and saw that this line

rex field=_raw "(?<st_json>\{.*)"

is the cause.  SimpleXML is treating the <st_json> as a JSON tag.  I did a little more research and found a Splunk answer that said if using rex in a <query></query> you have to save the search and then call the saved search instead.  That is what I did and it is working.