index=test-index (data loaded) OR ("GET data published/data/ui" OR "GET /v8/wi/data/*" OR "GET data/ui/wi/load/success")
|rex field=_raw "DIP:\s+\[(?<dip>[^\]]+)."
|rex field=_raw "ACTION:\s+(?<actions>\w+)"
|rex dield=_raw "SERVICE:\s+(?<services>\S+)"
|search actions= start OR actions=done NOT service="null"
|eval split=services.":".actions
|timechart span=1d count by split
|eval _time=strftime(_time, "%d/%m/%Y")
|table _time *start *done
When we run the above query , not all services getting captured, but we have data, attached the screen shot(highlighted ones are missing). can anyone let me know what is the issue with the query.
We don't know your data, we don't know what you're getting, we don't know if you match your data properly or extract the fields properly. We don't know anything except a search and some excel table.
Sample logs:
{"date": "1/2/2022 00:12:22,124" DATA [http:nio-12567-exec-44] DIP: [675478-7655a-56778d-655de45565] Data: [7665-56767ed-5454656] MIM: [483748348-632637f-38648266257d] FLOW: [NEW] { SERVICE: AAP | Applicationid: iis-675456 | ACTION: START | REQ: GET data published/data/ui } DADTA -:TIME:<TIMESTAMP> (0) 1712721546785 to 1712721546885 ms GET /v8/wi/data/*, GET data/ui/wi/load/success, "tags": {"host": "GTU5656", "insuranceid": "8786578896667", "lib": "app"}}
Sample logs:
{"date": "1/2/2022 00:12:22,124" DATA [http:nio-12567-exec-44] DIP: [675478-7655a-56778d-655de45565] Data: [7665-56767ed-5454656] MIM: [483748348-632637f-38648266257d] FLOW: [NEW] { SERVICE: AAP | Applicationid: iis-675456 | ACTION: DONE | REQ: GET data published/data/ui } DADTA -:TIME:<TIMESTAMP> (0) 1712721546785 to 1712721546885 ms GET /v8/wi/data/*, GET data/ui/wi/load/success, "tags": {"host": "GTU5656", "insuranceid": "8786578896667", "lib": "app"}}
Hi @PickleRick , added sample logs, let me know if u need any other details.
Are you sure that your raw event is not a valid JSON closer to
{"date": "1/2/2022 00:12:22,124", "DATA": "[http:nio-12567-exec-44] DIP: [675478-7655a-56778d-655de45565] Data: [7665-56767ed-5454656] MIM: [483748348-632637f-38648266257d] FLOW: [NEW] { SERVICE: AAP | Applicationid: iis-675456 | ACTION: START | REQ: GET data published/data/ui } DADTA -:TIME:<TIMESTAMP> (0) 1712721546785 to 1712721546885 ms GET /v8/wi/data/*, GET data/ui/wi/load/success", "tags": {"host": "GTU5656", "insuranceid": "8786578896667", "lib": "app"}}
instead? In other words, do you not have a field named "DATA" already? Because the overall structure of your illustration is very much compliant.
Assuming you have a field named DATA, a better strategy is trying to reconstruct a structure as your developers intended, instead of trying to extract individual tidbits as random text because your developers have clearly put in thoughts about data structure within DATA. I would propose something like
index=test-index (data loaded) OR ("GET data published/data/ui" OR "GET /v8/wi/data/*" OR "GET data/ui/wi/load/success")
| rex field=DATA mode=sed "s/ *[\|}\]]/\"/g s/: *\[*/=\"/g"
| rename _raw as temp
| rename DATA AS _raw
| kv
| rename temp as _raw
Your sample data should give you
ACTION | Applicationid | DIP | Data | FLOW | MIM | REQ | SERVICE | date | http | tags.host | tags.insuranceid | tags.lib |
START | iis-675456 | 675478-7655a-56778d-655de45565 | 7665-56767ed-5454656 | NEW | 483748348-632637f-38648266257d | GET data published/data/ui | AAP | 1/2/2022 00:12:22,124 | nio-12567-exec-44 | GTU5656 | 8786578896667 | app |
Here is an emulation that results in my hypothesized raw log:
| makeresults
| eval _raw = "{\"date\": \"1/2/2022 00:12:22,124\", \"DATA\": \"[http:nio-12567-exec-44] DIP: [675478-7655a-56778d-655de45565] Data: [7665-56767ed-5454656] MIM: [483748348-632637f-38648266257d] FLOW: [NEW] { SERVICE: AAP | Applicationid: iis-675456 | ACTION: START | REQ: GET data published/data/ui } DADTA -:TIME:<TIMESTAMP> (0) 1712721546785 to 1712721546885 ms GET /v8/wi/data/*, GET data/ui/wi/load/success\", \"tags\": {\"host\": \"GTU5656\", \"insuranceid\": \"8786578896667\", \"lib\": \"app\"}}"
| spath
``` the above emulates
index=test-index (data loaded) OR ("GET data published/data/ui" OR "GET /v8/wi/data/*" OR "GET data/ui/wi/load/success")
```
Play with the emulation and compare with real data.
Note: In the unimaginable case where your developers try really hard to mess up everybody's mind and inject semblance of JSON compliance while violating common sense, you can still apply the same principle against _raw. Like this:
index=test-index (data loaded) OR ("GET data published/data/ui" OR "GET /v8/wi/data/*" OR "GET data/ui/wi/load/success")
```
| rex mode=sed "s/ *[\|}\]]/\"/g s/: *\[*/=\"/g"
| kv
This is what the output would look like:
ACTION | Applicationid | DATA | DIP | Data | FLOW | MIM | REQ | SERVICE | host |
START | iis-675456 | http= | 675478-7655a-56778d-655de45565 | 7665-56767ed-5454656 | NEW | 483748348-632637f-38648266257d | GET data published/data/ui | AAP |
Without a better structure, you won't get subnodes embedded in tags; but your original question does not seem to care about tags.
Here is an emulation that resembles the actual sample you posted:
| makeresults
| eval _raw = "{\"date\": \"1/2/2022 00:12:22,124\", DATA: [http:nio-12567-exec-44] DIP: [675478-7655a-56778d-655de45565] Data: [7665-56767ed-5454656] MIM: [483748348-632637f-38648266257d] FLOW: [NEW] { SERVICE: AAP | Applicationid: iis-675456 | ACTION: START | REQ: GET data published/data/ui } DADTA -:TIME:<TIMESTAMP> (0) 1712721546785 to 1712721546885 ms GET /v8/wi/data/*, GET data/ui/wi/load/success\", \"tags\": {\"host\": \"GTU5656\", \"insuranceid\": \"8786578896667\", \"lib\": \"app\"}}"
``` the above emulates
index=test-index (data loaded) OR ("GET data published/data/ui" OR "GET /v8/wi/data/*" OR "GET data/ui/wi/load/success")
```
We have around 10 services, by using below query i am getting 8 services and other 2 are not getting displayed in the table. But we can view them in events. Filed extraction is working correctly.
not sure why other 2 services are not showing up in the table. please find the output.
index=test-index (data loaded) OR ("GET data published/data/ui" OR "GET /v8/wi/data/*" OR "GET data/ui/wi/load/success") |rex field=_raw "DIP:\s+\[(?<dip>[^\]]+)." |rex field=_raw "ACTION:\s+(?<actions>\w+)" |rex dield=_raw "SERVICE:\s+(?<services>\S+)" |search actions= start OR actions=done NOT service="null" |eval split=services.":".actions |timechart span=1d count by split |eval _time=strftime(_time, "%d/%m/%Y") |table _time *start *done
Current output: (DCC:DONE &PIP:DONE fields are missing)
_time | AAP:START | ACC:START | ABB:START | DCC:START | PIP:START | AAP:DONE | ACC:DONE | ABB:DONE |
1/2/2022 | 1 | 100 | 1 | 100 | 1 | 1 | 66 | 1 |
2/2/2022 | 5 | 0 | 5 | 0 | 3 | 3 | 0 | 3 |
3/2/2022 | 10 | 0 | 10 | 0 | 8 | 7 | 0 | 8 |
4/2/2022 | 100 | 1 | 100 | 1 | 97 | 80 | 1 | 80 |
5/2/2022 | 0 | 5 | 0 | 5 | 350 | 0 | 4 | 0 |
Expected output:
_time | AAP:START | ACC:START | ABB:START | DCC:START | PIP:START | AAP:DONE | ACC:DONE | ABB:DONE | DCC:DONE | PIP:DONE |
1/2/2022 | 1 | 100 | 1 | 100 | 1 | 1 | 66 | 1 | 99 | 1 |
2/2/2022 | 5 | 0 | 5 | 0 | 3 | 3 | 0 | 3 | 0 | 2 |
3/2/2022 | 10 | 0 | 10 | 0 | 8 | 7 | 0 | 8 | 0 | 3 |
4/2/2022 | 100 | 1 | 100 | 1 | 97 | 80 | 1 | 80 | 1 | 90 |
5/2/2022 | 0 | 5 | 0 | 5 | 350 | 0 | 4 | 0 | 5 | 200 |
Treating structured data as pure text is doomed to be unstable. Have you tried my suggestion of reconstructing events based on inherent structure?
As a side note - I suppose this is some sort of a typo and your search contains "search action=start", not "search action= start" (notice the space in the middle). Assuming that...
That's a bit strange because assuming all your events follow the same syntax, the search looks relatively sound.
The normal approach in debugging searches would be to either start from the beginning and verify whether each step gives you desired results so that after adding each subsequent step you can verify when it stops doing what you want or cutting the commands from the end and see when it starts working properly (for that stage of the pipeline).
I'd cut back to just after the rex commands and search for events that should match those results you lack in your final results.
Then add one command after another and see.
Two possible culprits:
1) default limit of results for timechart (but that's kinda unlikely because you'd get 10 results and "OTHER" by default, not 8 results)
2) case of field names - field names are case sensitive whereas field values are not so if your services field contains "done" in most cases but "DONE" for those missing ones, the whatever:DONE fields would _not_ get matched by the *done wildcard in the table command.