Missing field values in report

[pdantuuri0411]

We have logs in the following format[1]. We created a report with few fields like time, service, operation, method, principle, systemid and count.

But when ever a field is missing in the log, the report is not picking up the log to display it in the report.

[1]

2019-09-18T19:41:20.624+0000 INFO 855 com.l7tech.log.custom.splunk : -4: |https://apiqa.flatball.com/fmpsp2p/service|getReleaseInfo|SOAP|10.88.30.32|p2p_client_esmartct||87d1...; charset=UTF-8

In the above example, the system id is missing and the report didnt pickup this log.

Search used for the report is as below:

| pivot servicestrafficcaapimodel servicesapiqads count(servicesapiqads) AS count SPLITROW _time AS _time PERIOD day SPLITROW service AS service SPLITROW operation AS operation SPLITROW method AS method SPLITROW principal AS principal SPLITROW systemid AS systemid TOP 100 count(servicesapiqads) ROWSUMMARY 0 COLSUMMARY 0 SHOWOTHER 1 | SORT -_time

[jkat54]

Sounds like a usecase for fillnull search command:

https://docs.splunk.com/Documentation/SplunkCloud/7.2.7/SearchReference/Fillnull

[pdantuuri0411]

thank you for the reference @jkat54. I tried adding fillnull with different options but it did not resolve my issue.

 | pivot servicestrafficcaapimodel servicesapiqads count(servicesapiqads) AS count SPLITROW _time AS _time PERIOD day SPLITROW service AS service SPLITROW operation AS operation SPLITROW method AS method SPLITROW principal AS principal SPLITROW systemid AS systemid TOP 100 count(servicesapiqads) ROWSUMMARY 0 COLSUMMARY 0 SHOWOTHER 1 | SORT -_time |  fillnull value=NULL

[jkat54]

How about an eval on the datamodels definition instead..

 | eval field=if(isnull(field),"NULL",field)

[pdantuuri0411]

Thank you for your answer. I tried to add it in my data model and it works fine. But I couldn't save it there. So this is not reflecting in the report.

my data model

| from datamodel:"servicestrafficcaapimodel.servicesapiqads" | eval systemid=if(isnull(systemid),"NULL",systemid)

I don't have an option to save it. How do I reflect it in the report?

Thank you again

[jkat54]

Why can't you save the edits to your data model?

You go to datasets, open the dataset/datamodels, click edit (or manage then edit), click add field, select eval expression from add field drop down, and you add the expression save and rebuild any summaries/access you have.

[pdantuuri0411]

I am trying the exact same thing. I put "systemid=if(isnull(systemid),"NULL",systemid)" in eval expression bar and I get the below error while previewing. Name of the field is systemid.

"Error in 'eval' command: Fields cannot be assigned a boolean result. Instead, try if([bool expr], [expr], [expr])."

[jkat54]

Maybe it doesn't like "NULL"? Never seen that before.

[richgalloway]

It looks like the missing field is causing the event to be skipped over in the data model. Check the constraints on the data model to make sure they are correct.

[pdantuuri0411]

In the constraints for this data model, I just have the index. SO every event from this index should be in the report.

The issue is the logs that have a missing field determined in the report query is not getting pickedup.