Not all fields showing up

[Santosh2]

index=test-index (data loaded) OR ("GET data published/data/ui" OR "GET /v8/wi/data/*" OR "GET data/ui/wi/load/success")
|rex field=_raw "DIP:\s+\[(?<dip>[^\]]+)."
|rex field=_raw "ACTION:\s+(?<actions>\w+)"
|rex dield=_raw "SERVICE:\s+(?<services>\S+)"
|search actions= start OR actions=done NOT service="null"
|eval split=services.":".actions
|timechart span=1d count by split
|eval _time=strftime(_time, "%d/%m/%Y")
|table _time *start *done

When we run the above query , not all services getting captured,  but we have data, attached the screen shot(highlighted ones are missing). can anyone let me know what is the issue with the query.

[PickleRick]

We don't know your data, we don't know what you're getting, we don't know if you match your data properly or extract the fields properly. We don't know anything except a search and some excel table.

[Santosh2]

Sample logs:
{"date": "1/2/2022 00:12:22,124"  DATA [http:nio-12567-exec-44] DIP: [675478-7655a-56778d-655de45565] Data: [7665-56767ed-5454656] MIM: [483748348-632637f-38648266257d] FLOW: [NEW] { SERVICE: AAP | Applicationid: iis-675456 | ACTION: START | REQ: GET data published/data/ui } DADTA -:TIME:<TIMESTAMP> (0) 1712721546785 to 1712721546885 ms GET /v8/wi/data/*, GET data/ui/wi/load/success, "tags": {"host": "GTU5656", "insuranceid": "8786578896667", "lib": "app"}}

Sample logs:
{"date": "1/2/2022 00:12:22,124"  DATA [http:nio-12567-exec-44] DIP: [675478-7655a-56778d-655de45565] Data: [7665-56767ed-5454656] MIM: [483748348-632637f-38648266257d] FLOW: [NEW] { SERVICE: AAP | Applicationid: iis-675456 | ACTION: DONE | REQ: GET data published/data/ui } DADTA -:TIME:<TIMESTAMP> (0) 1712721546785 to 1712721546885 ms GET /v8/wi/data/*, GET data/ui/wi/load/success, "tags": {"host": "GTU5656", "insuranceid": "8786578896667", "lib": "app"}}

Hi @PickleRick , added sample logs, let me know if u need any other details.

[yuanliu]

Are you sure that your raw event is not a valid JSON closer to

{"date": "1/2/2022 00:12:22,124",  "DATA": "[http:nio-12567-exec-44] DIP: [675478-7655a-56778d-655de45565] Data: [7665-56767ed-5454656] MIM: [483748348-632637f-38648266257d] FLOW: [NEW] { SERVICE: AAP | Applicationid: iis-675456 | ACTION: START | REQ: GET data published/data/ui } DADTA -:TIME:<TIMESTAMP> (0) 1712721546785 to 1712721546885 ms GET /v8/wi/data/*, GET data/ui/wi/load/success", "tags": {"host": "GTU5656", "insuranceid": "8786578896667", "lib": "app"}}

instead?  In other words, do you not have a field  named "DATA" already? Because the overall structure of your illustration is very much compliant.

Assuming you have a field named DATA, a better strategy is trying to reconstruct a structure as your developers intended, instead of trying to extract individual tidbits as random text because your developers have clearly put in thoughts about data structure within DATA.  I would propose something like

index=test-index (data loaded) OR ("GET data published/data/ui" OR "GET /v8/wi/data/*" OR "GET data/ui/wi/load/success")
| rex field=DATA mode=sed "s/ *[\|}\]]/\"/g s/: *\[*/=\"/g"
| rename _raw as temp
| rename DATA AS _raw
| kv
| rename temp as _raw

Your sample data should give you

ACTION

Applicationid

DIP

Data

FLOW

MIM

REQ

SERVICE

date

http

tags.host

tags.insuranceid

tags.lib

START

iis-675456

675478-7655a-56778d-655de45565

7665-56767ed-5454656

NEW

483748348-632637f-38648266257d

GET data published/data/ui

AAP

1/2/2022 00:12:22,124

nio-12567-exec-44

GTU5656

8786578896667

app

Here is an emulation that results in my hypothesized raw log:

| makeresults
| eval _raw = "{\"date\": \"1/2/2022 00:12:22,124\",  \"DATA\": \"[http:nio-12567-exec-44] DIP: [675478-7655a-56778d-655de45565] Data: [7665-56767ed-5454656] MIM: [483748348-632637f-38648266257d] FLOW: [NEW] { SERVICE: AAP | Applicationid: iis-675456 | ACTION: START | REQ: GET data published/data/ui } DADTA -:TIME:<TIMESTAMP> (0) 1712721546785 to 1712721546885 ms GET /v8/wi/data/*, GET data/ui/wi/load/success\", \"tags\": {\"host\": \"GTU5656\", \"insuranceid\": \"8786578896667\", \"lib\": \"app\"}}"
| spath
``` the above emulates
index=test-index (data loaded) OR ("GET data published/data/ui" OR "GET /v8/wi/data/*" OR "GET data/ui/wi/load/success")
```

Play with the emulation and compare with real data.

Note: In the unimaginable case where your developers try really hard to mess up everybody's mind and inject semblance of JSON compliance while violating common sense, you can still apply the same principle against _raw.  Like this:

index=test-index (data loaded) OR ("GET data published/data/ui" OR "GET /v8/wi/data/*" OR "GET data/ui/wi/load/success")
```
| rex mode=sed "s/ *[\|}\]]/\"/g s/: *\[*/=\"/g"
| kv

This is what the output would look like:

ACTION	Applicationid	DATA	DIP	Data	FLOW	MIM	REQ	SERVICE	host
START	iis-675456	http=	675478-7655a-56778d-655de45565	7665-56767ed-5454656	NEW	483748348-632637f-38648266257d	GET data published/data/ui	AAP

Without a better structure, you won't get subnodes embedded in tags; but your original question does not seem to care about tags.

Here is an emulation that resembles the actual sample you posted:

| makeresults
| eval _raw = "{\"date\": \"1/2/2022 00:12:22,124\",  DATA: [http:nio-12567-exec-44] DIP: [675478-7655a-56778d-655de45565] Data: [7665-56767ed-5454656] MIM: [483748348-632637f-38648266257d] FLOW: [NEW] { SERVICE: AAP | Applicationid: iis-675456 | ACTION: START | REQ: GET data published/data/ui } DADTA -:TIME:<TIMESTAMP> (0) 1712721546785 to 1712721546885 ms GET /v8/wi/data/*, GET data/ui/wi/load/success\", \"tags\": {\"host\": \"GTU5656\", \"insuranceid\": \"8786578896667\", \"lib\": \"app\"}}"
``` the above emulates
index=test-index (data loaded) OR ("GET data published/data/ui" OR "GET /v8/wi/data/*" OR "GET data/ui/wi/load/success")
```

[Santosh2]

We have around 10 services, by using below query i am getting 8 services and other 2 are not getting displayed in the table. But we can view them in events. Filed extraction is working correctly.
not sure why other 2 services are not showing up in the table. please find the output.

index=test-index (data loaded) OR ("GET data published/data/ui" OR "GET /v8/wi/data/*" OR "GET data/ui/wi/load/success")
|rex field=_raw "DIP:\s+\[(?<dip>[^\]]+)."
|rex field=_raw "ACTION:\s+(?<actions>\w+)"
|rex dield=_raw "SERVICE:\s+(?<services>\S+)"
|search actions= start OR actions=done NOT service="null"
|eval split=services.":".actions
|timechart span=1d count by split
|eval _time=strftime(_time, "%d/%m/%Y")
|table _time *start *done

 Current output: (DCC:DONE &PIP:DONE  fields are missing)

_time	AAP:START	ACC:START	ABB:START	DCC:START	PIP:START	AAP:DONE	ACC:DONE	ABB:DONE
1/2/2022	1	100	1	100	1	1	66	1
2/2/2022	5	0	5	0	3	3	0	3
3/2/2022	10	0	10	0	8	7	0	8
4/2/2022	100	1	100	1	97	80	1	80
5/2/2022	0	5	0	5	350	0	4	0

Expected output:

_time	AAP:START	ACC:START	ABB:START	DCC:START	PIP:START	AAP:DONE	ACC:DONE	ABB:DONE	DCC:DONE	PIP:DONE
1/2/2022	1	100	1	100	1	1	66	1	99	1
2/2/2022	5	0	5	0	3	3	0	3	0	2
3/2/2022	10	0	10	0	8	7	0	8	0	3
4/2/2022	100	1	100	1	97	80	1	80	1	90
5/2/2022	0	5	0	5	350	0	4	0	5	200

[yuanliu]

Treating structured data as pure text is doomed to be unstable.  Have you tried my suggestion of reconstructing events based on inherent structure?

[PickleRick]

As a side note - I suppose this is some sort of a typo and your search contains "search action=start", not "search action= start" (notice the space in the middle). Assuming that...

That's a bit strange because assuming all your events follow the same syntax, the search looks relatively sound.

The normal approach in debugging searches would be to either start from the beginning and verify whether each step gives you desired results so that after adding each subsequent step you can verify when it stops doing what you want or cutting the commands from the end and see when it starts working properly (for that stage of the pipeline).

I'd cut back to just after the rex commands and search for events that should match those results you lack in your final results.

Then add one command after another and see.

Two possible culprits:

1) default limit of results for timechart (but that's kinda unlikely because you'd get 10 results and "OTHER" by default, not 8 results)

2) case of field names - field names are case sensitive whereas field values are not so if your services field contains "done" in most cases but "DONE" for those missing ones, the whatever:DONE fields would _not_ get matched by the *done wildcard in the table command.