topic Re: How do I extract a field from my raw stdout log data using rex? in Splunk Search

How do I extract a field from my raw stdout log data using rex?

rseri17 — Fri, 16 Apr 2021 16:51:03 GMT

Can you please help with extracting the fields from the below sample log. I am unable to escape the "'// &" '" in the log using regex.

I am trying to extract upstream_response_time and connection_requests.

{"log":"[1] api_access.log: [1618591069.220218866, { \"msec\": 1618591069.219, \"remote_addr\": \"10.248.32.1\", \"x_forwarded_for\": \"10.233.42.16, 10.248.32.1\", \"remote_port\": \"41474\", \"pipelined\": \".\", \"body_bytes_sent\": \"4554\", \"bytes_sent\": 5150, \"request_time\": 1.066, \"upstream_response_time\": \"1.066\", \"upstream_response_length\": \"18456\", \"upstream_status\": \"200\", \"kore_route\": \"-\", \"koreserver\": \"KoreServer/\", \"host\": \"app-artificial-intelligence-dev.t3-openshift1.*\", \"hostname\": \"kore-app-62-g6522\", \"server_name\": \"_\", \"request_completion\": \"OK\", \"status\": 200, \"connection_requests\": 2, \"request_uri\": \"/api/1.1/builder/streams/st-332b9e29-e487-567e-b382-56e0fa4beb9d/dialogs/dg-3558dfff-9932-5640-a364-5f7202d5dfc8/components?rnd=qdzsm9\", \"request_method\": \"GET\", \"request_content_type\": \"application/json;charset=UTF-8\", \"request_content_length\": \"-\", \"request_total_length\": 1735, \"args\": \"rnd=qdzsm9\",\"is_args\": \"?\", \"x-traceid\"😕"-\", \"http_user_agent\": \"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.150 Safari/537.36\" }\", \"podname\"=\u003e\"kore-app-62-g6522\"}]\n","stream":"stdout","time":"2021-04-16T16:37:49.39632196Z"}

Re: How do I extract a field from my raw stdout log data using rex?

alonsocaio — Fri, 16 Apr 2021 17:36:28 GMT

Hi @rseri17 ,

Maybe you can try the following regex for extracting the fields you need:

| rex field=FIELD_NAME "upstream_response_time\\"\:\s\\\"(?<upstream_response_time>[\d\.]+)\\\"" | rex field=FIELD_NAME "connection_requests\\\"\:\s(?<connection_requests>[\d]+)"

Re: How do I extract a field from my raw stdout log data using rex?

rseri17 — Mon, 19 Apr 2021 13:20:08 GMT

Thanks for your response . That did not work , I am unable to escape the ' " ' . It errors out parsing the regex.

Re: How do I extract a field from my raw stdout log data using rex?

ITWhisperer — Mon, 19 Apr 2021 13:47:12 GMT

It looks like @alonsocaio is just missing a backslash after upstream_response_time - try

| rex field=FIELD_NAME "upstream_response_time\\\"\:\s\\\"(?<upstream_response_time>[\d\.]+)\\\"" | rex field=FIELD_NAME "connection_requests\\\"\:\s(?<connection_requests>[\d]+)"

Re: How do I extract a field from my raw stdout log data using rex?

rseri17 — Mon, 19 Apr 2021 15:06:30 GMT

Thanks , I was able to extract the fields. However when i try to add that to my query to extract from the _raw its not working . could you please let me know , if I am missing something. I have attached the result.

Re: How do I extract a field from my raw stdout log data using rex?

ITWhisperer — Mon, 19 Apr 2021 15:22:20 GMT

You may need to add more (pairs of) backslashes to the splunk rex expression (not the regex101.com) particularly when trying to escape backslashes as it seems to need extra.

Re: How do I extract a field from my raw stdout log data using rex?

rseri17 — Mon, 19 Apr 2021 15:41:14 GMT

That worked !! Thanks a lot for your help guys. I didn't know about that, adding extra backsplashes worked.

|rex field=_raw "upstream_response_time\\\\\"\:\s\\\\\"(?<upstream_response_time>[\d\.]+)\\\\\""