I have a datamodel lets say with a base constraint that returns the following two events
01-01-2019 01:00:00 type=VIEW_REQUEST duration=100 taskID=123456 01-01-2019 00:00:00 request=do_something count=5 taskID=123456
And I have a search built that was returning null results for taskID 123456 type and duration, but was returning the request and count fields, i.e. it was missing the fields from the VIEW_REQUEST event log. I ran the base search for the taskID 123456 and the fields are extracted properly and can be used without issue.
To summarize, this search:
("VIEW_REQUEST" "duration") OR ("do_something" "count") taskID="123456" | stats values(type) as type values(request) as request values(duration) as duration values(count) as count by taskID
type --------| request -----| duration | count | taskID VIEW_REQUEST | do_something | 100 -----| 5 ----| 123456
While this search (with """("VIEW_REQUEST" "duration") OR ("do_something" "count") taskID""" as the constraint for the datamodel):
| tstats values(request.type) as type values(request.request) as request values(request.duration) as duration values(request.count) as count from datamodel=request by request.taskID
type | request -----| duration | count | taskID NULL | do_something | NULL ----| 5 ----| 123456
The field extraction is working correctly for approximately 99.7% of all results, but there are 297 out of approximately 80,000 results where duration and type are null for some reason when searched via the data model, but the fields are extracted properly in the events that are returned by the base constraint. How do I fix this?
I can't remember why, but I had a span=1s at the end of the tstats that was removing an extremely small subset of events. Removing that didn't seem to have an effect on the end results that I was expecting other than correctly reporting the durations I was expecting. I believe this is resolved although I'm unsure why it was failing with the span in there.
I remember now why I had the span in there, without the span=1s, the span was defaulting to daily. I need to have it on a transactional level
Which I think I've now resolved by adding latest(_time) to the aggregations rather than the by clause. I'll update if I find other issues with it.