Splunk Search

Fields are missing

mahesh27
Communicator

Here is the sample log:

{"date": "1/2/2022 00:12:22,124",  "DATA": "[http:nio-12567-exec-44] DIP: [675478-7655a-56778d-655de45565] Data: [7665-56767ed-5454656] MIM: [483748348-632637f-38648266257d] FLOW: [NEW] { SERVICE: AAP | Applicationid: iis-675456 | ACTION: START | REQ: GET data published/data/ui } DADTA -:TIME:<TIMESTAMP> (0) 1712721546785 to 1712721546885 ms GET /v8/wi/data/*, GET data/ui/wi/load/success", "tags": {"host": "GTU5656", "insuranceid": "8786578896667", "lib": "app"}}

 

We have around 10 services, by using below query i am getting 8 services and other 2 are not getting displayed in the table. But we can view them in events. Filed extraction is working correctly.
not sure why other 2 services are not showing up in the table.

index=test-index (data loaded) OR ("GET data published/data/ui" OR "GET /v8/wi/data/*" OR "GET data/ui/wi/load/success")
|rex field=_raw "DIP:\s+\[(?<dip>[^\]]+)."
|rex field=_raw "ACTION:\s+(?<actions>\w+)"
|rex dield=_raw "SERVICE:\s+(?<services>\S+)"
|search actions= start OR actions=done NOT service="null"
|eval split=services.":".actions
|timechart span=1d count by split
|eval _time=strftime(_time, "%d/%m/%Y")
|table _time *start *done


 Current output: (DCC:DONE &PIP:DONE  fields are missing)

_timeAAP:STARTACC:STARTABB:STARTDCC:STARTPIP:STARTAAP:DONEACC:DONEABB:DONE
1/2/20221100110011661
2/2/202250503303
3/2/20221001008708
4/2/2022100110019780180
5/2/20220505350040

 

Expected output:

_timeAAP:STARTACC:STARTABB:STARTDCC:STARTPIP:STARTAAP:DONEACC:DONEABB:DONEDCC:DONEPIP:DONE
1/2/20221100110011661991
2/2/20225050330302
3/2/2022100100870803
4/2/2022100110019780180190
5/2/202205053500405200

 

Labels (1)
0 Karma

yuanliu
SplunkTrust
SplunkTrust

This is exactly what I speculated in your previous question: that your developers have left a compliant JSON, while having some structure within DATA field.  Instead of rex individual elements as if DATA is made of random text, you should utilize the structure your developers intended.  Have you tried my suggestion yesterday?

 

index=test-index (data loaded) OR ("GET data published/data/ui" OR "GET /v8/wi/data/*" OR "GET data/ui/wi/load/success")
| rex field=DATA mode=sed "s/ *[\|}\]]/\"/g s/: *\[*/=\"/g"
| rename DATA AS _raw
| kv
|search ACTION= start OR ACTION=done NOT SERVICE="null"
|eval split=SERVICE.":".ACTION
|timechart span=1d count by split
|eval _time=strftime(_time, "%d/%m/%Y")
| table _time *START *DONE

 

(Since you are running timechart, there is no need to preserver _raw, so I omitted that.  I also don't see how your last table command could give you the result you illustrated because START and DONE are capitalized.) Your sample data (only one event) gives

_timeAAP:START
01/02/20221
11/04/20240

This is the data emulation including _time conversion

 

| makeresults
| eval _raw = "{\"date\": \"1/2/2022 00:12:22,124\",  \"DATA\": \"[http:nio-12567-exec-44] DIP: [675478-7655a-56778d-655de45565] Data: [7665-56767ed-5454656] MIM: [483748348-632637f-38648266257d] FLOW: [NEW] { SERVICE: AAP | Applicationid: iis-675456 | ACTION: START | REQ: GET data published/data/ui } DADTA -:TIME:<TIMESTAMP> (0) 1712721546785 to 1712721546885 ms GET /v8/wi/data/*, GET data/ui/wi/load/success\", \"tags\": {\"host\": \"GTU5656\", \"insuranceid\": \"8786578896667\", \"lib\": \"app\"}}"
| spath
| eval _time = strptime(date, "%d/%m/%Y %H:%M:%S,%f")
``` the above emulates
index=test-index (data loaded) OR ("GET data published/data/ui" OR "GET /v8/wi/data/*" OR "GET data/ui/wi/load/success")
```

 

Play with it and compare to real data.  If this doesn't work for select events, you need to post samples of those events.

 

Tags (1)
0 Karma

mahesh27
Communicator

@yuanliu , yes i tried the below query, but i getting 0 results

index=test-index (data loaded) OR ("GET data published/data/ui" OR "GET /v8/wi/data/*" OR "GET data/ui/wi/load/success")
| rex field=DATA mode=sed "s/ *[\|}\]]/\"/g s/: *\[*/=\"/g"
| rename DATA AS _raw
| kv
|search ACTION= start OR ACTION=done NOT SERVICE="null"
|eval split=SERVICE.":".ACTION
|timechart span=1d count by split
|eval _time=strftime(_time, "%d/%m/%Y")
| table _time *START *DONE
0 Karma

yuanliu
SplunkTrust
SplunkTrust

Have you compared emulation with real data?  Also, really get rid of that table command which can be in the way. (You can add some formatting after you verify that outputs are satisfactory.)  Is there some real data that you can share? (Anonymize as needed but take care to preserve precise structure.)  Using emulation, the output is not zero.  Clearly, actual data is different from what you posted above.

Run this:

 

| makeresults
| eval _raw = "{\"date\": \"1/2/2022 00:12:22,124\",  \"DATA\": \"[http:nio-12567-exec-44] DIP: [675478-7655a-56778d-655de45565] Data: [7665-56767ed-5454656] MIM: [483748348-632637f-38648266257d] FLOW: [NEW] { SERVICE: AAP | Applicationid: iis-675456 | ACTION: START | REQ: GET data published/data/ui } DADTA -:TIME:<TIMESTAMP> (0) 1712721546785 to 1712721546885 ms GET /v8/wi/data/*, GET data/ui/wi/load/success\", \"tags\": {\"host\": \"GTU5656\", \"insuranceid\": \"8786578896667\", \"lib\": \"app\"}}"
| spath
| eval _time = strptime(date, "%d/%m/%Y %H:%M:%S,%f")
``` the above emulates
index=test-index (data loaded) OR ("GET data published/data/ui" OR "GET /v8/wi/data/*" OR "GET data/ui/wi/load/success")
```
| rex field=DATA mode=sed "s/ *[\|}\]]/\"/g s/: *\[*/=\"/g"
| rename DATA AS _raw
| kv
|search ACTION= start OR ACTION=done NOT SERVICE="null"
|eval split=SERVICE.":".ACTION
|timechart span=1d count by split
|eval _time=strftime(_time, "%d/%m/%Y")
| table _time *START *DONE

 

Do you get the same results as I did in the previous comment? (I do not encourage use of screenshot to show search or results, but I had already shared them in text previously. So, here you go for a screenshot.)

Screen Shot 2024-04-11 at 2.27.15 PM.png

0 Karma
Get Updates on the Splunk Community!

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...

Edge Processor Scaling, Energy & Manufacturing Use Cases, and More New Articles on ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Get More Out of Your Security Practice With a SIEM

Get More Out of Your Security Practice With a SIEMWednesday, July 31, 2024  |  11AM PT / 2PM ETREGISTER ...