topic Re: Rex field extraction in Splunk Search

How to extract the request type and instanceId fields values?

Sangamesh — Tue, 20 Jun 2023 02:45:16 GMT

{"log":"{\\"instanceId\\":\\"abc-fdh-48f-4432\\",\\"requestType\\":\\"ABC\\"}

Using the above sample log, how to extract the request type and instanceId fields values?

Re: Rex field extraction

gcusello — Mon, 19 Jun 2023 06:52:29 GMT

Hi @Sangamesh,

please try this regex

requestType\\\\\":\\\\\"(?<requestType>\w+)

that you can test at https://regex101.com/r/WUjcyz/1

Ciao

Giuseppe

Re: Rex field extraction

ITWhisperer — Mon, 19 Jun 2023 06:55:48 GMT

| foreach instanceId requestType [| rex "<<FIELD>>[^\"]+\"[^\"]+\"(?<<<FIELD>>>[^\\\]+)"]

Re: Rex field extraction

yuanliu — Mon, 19 Jun 2023 07:30:19 GMT

This is part of a JSON object. If the raw event is also JSON, you should have a field named "log". Is this correct?

You don't need rex to extract requestType. The best way to extract structured data is spath

| spath input=log

Your sample data should give you

instanceId	log	requestType
abc-fdh-48f-4432	{"instanceId":"abc-fdh-48f-4432","requestType":"ABC"}	ABC

If for some reason log is not available as a field, you should extract the full JSON object that contains "log" as a key, extract that JSON with spath, then extract fields contained in log using spath.

Re: Rex field extraction

Sangamesh — Mon, 19 Jun 2023 07:32:33 GMT

@ITWhisperer Could you please provide brief explanation about this regex

Re: Rex field extraction

Sangamesh — Mon, 19 Jun 2023 07:37:45 GMT

log is not available as a field. Solution didn't work

Re: Rex field extraction

ITWhisperer — Mon, 19 Jun 2023 07:44:01 GMT

The regex matches to the anchor, skips over the double quotes to get to the field contents and extracts the contents up to the next backslash (which is delimiting the contents in your example).

Re: Rex field extraction

yuanliu — Tue, 20 Jun 2023 07:55:24 GMT

Can you share some raw data (anonymize as needed)? Clearly "log" is one node in a larger JSON object. You should try to reach that object and work down the structure rather than trying to treat structured data as string.

Re: Rex field extraction

ITWhisperer — Tue, 20 Jun 2023 08:21:19 GMT

@yuanliu How many times do we have to ask for sample data? I may consider boycotting questions without suitable data 😀

Re: Rex field extraction

Sangamesh — Wed, 21 Jun 2023 11:28:32 GMT

Here is the sample _raw from the splunk but it has 2 different format log 1 format without double backslash(\\) and other one with double backslash(\\).How do I extract requestId,requestType,message,endpoint ?

{"host":"xx-1345a0196e.svr.am.abcde.net","app_id":"4324","source":"fstatabd","@version":"1",
"log":"{"timestamp":"2023-06-20T15:58:20.505Z","logger":"com.abcde.abcservices.service.ghstreamService",
"instanceId":"abc-atments-service-green-55dbdb5859-75vkl","namespace":"80458d64606-ct-abc-gkp-test",
"platform":"GKP","message":"Received xyz Callback Response for RequestId: 1436, Status code : 200 OK,
Response Payload : Successfully received the response","level":"INFO","traceId":"6491cc9c8bb8f102fafe7b07de24a457",
"spanId":"fafe7b07de24a457","remote-host":"123.027.16.57","protocol":"HTTP/1.1",
"http.endpoint":"GET /ament/p2CallBack","requestType":"RRR","requestId":"1436"}",
"s_sourcetype":"rost","tags":["_dateparsefailure"],
"Kubernetes.node":"xx-1535a0196e.svr.am.abcde.net","appId":"4324","@timestamp":"2023-06-20T15:58:31.263Z"}

{"log":"{\\"timestamp\\":\\"2023-06-21T07:45:54.528Z\\",\\"logger\\":\\"com.csa.fsces.service.3Service\\",
\\"instanceId\\":\\"unw-sthrage-service-green-58694cdbb8-smbjj\\",\\"namespace\\":\\"4248-ct-xrw-gkp-dev\\",
\\"platform\\":\\"GKP\\",\\"message\\":\\"Entity name: NDAN_POSITION_VUE,
Record type: NR, Request Id: 100185, Infdund context: 485640257604\\",
\\"level\\":\\"INFO\\",\\"traceId\\":\\"6492aa2326285332f853f16e2a49fd50\\",
\\"spanId\\":\\"9dc079cc5ef24811\\",\\"remote-host\\":\\"123.456.125.09\\",
\\"protocol\\":\\"HTTP/1.1\\",\\"http.endpoint\\":\\"POST /storage/e3/createfile/\\",
\\"RequestID\\":\\"143185\\",\\"user-agent\\":\\"Java/17.0.7\\",\\"RequestType\\":\\"TR\\"}",
"Kubernetes.node":"hh-10feb4b660.svr.us.sj.net","appId":"5352","hostname":"gg-10geb4b660.svr.us.sj.net"}

Re: Rex field extraction

ITWhisperer — Wed, 21 Jun 2023 12:12:04 GMT

Re: Rex field extraction

Sangamesh — Wed, 21 Jun 2023 12:34:36 GMT

Please provide the steps for creation of dynamic dashboard with using the previous solution output.It should include timestamp ,requestId,instanceId,message ,endpoint.

requestid,requesttype should be the dropdown menu and Date should be the calendar text field .based on these combination selections ,it should be populate the data in dashboard

Re: Rex field extraction

gcusello — Wed, 21 Jun 2023 13:11:43 GMT

Hi @Sangamesh ,

good for you, see next time!

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated by all the contributors 😉

Re: Rex field extraction

Sangamesh — Mon, 10 Jul 2023 07:54:37 GMT

@ITWhisperer How to extract RequestID,http.endpoint,message,RequestType from these below mentioned different types logs

{"@version":"1","tags":["_dateparsefailure"],"host":"wh-45sg.svr.us.xyz.net","s_sourcetype":"godn","log":"{\"timestamp\":\"2023-06-21T07:45:54.528Z\",\"logger\":\"com.csa.fsces.service.3Service\",
\"instanceId\":\"unw-sthrage-service-green-58694cdbb8-smbjj\",\"namespace\":\"4248-ct-xrw-gkp-dev\",
\"platform\":\"GKP\\",\"message\":\"Entity name: NDAN_POSITION_VUE,Record type: NR, Request Id: 100185, Infdund context: 485640257604\",
\"level\":\"INFO\",\"traceId\":\"6492aa2326285332f853f16e2a49fd50\",
\"spanId\":\"9dc079cc5ef24811\",\"remote-host\":\"123.456.125.09\",
\"protocol\":\"HTTP/1.1\",\"http.endpoint\":\"POST /storage/e3/createfile/\",
\"RequestID\":\"143185\",\"user-agent\":\"Java/17.0.7\",\"RequestType\":\"TR\"}",
"Kubernetes.node":"hh-10feb4b660.svr.us.sj.net","appId":"5352","hostname":"gg-10geb4b660.svr.us.sj.net"}

Re: Rex field extraction

ITWhisperer — Mon, 10 Jul 2023 08:34:04 GMT

Use the solution previously accepted.

Re: Rex field extraction

yuanliu — Tue, 11 Jul 2023 06:34:49 GMT

Again, if your developer has done their homework, they would have written conformant JSON, meaning that you can have more robust field extraction using simpler commands, like this:

| spath input=log ``` 1st format should have log.requestType as a field 2nd format's log field is a string containing escaped JSON containing a node RequestType ``` | eval RequestType = coalesce(RequestType, 'log.requestType')

There are some strange errors in the samples you cited. If they are actual errors, you should ask your developer to fix them.