Why are some of my fields for a given sourcetype not recognized for some logs in JSON format?

[bruceclarke]

Hey all,

I'm seeing an issue with almost half of my logs for a given sourcetype not having fields recognized. So, I run a search like sourcetype=exceptions | table _time Machine and I see about half of the entries with empty table cells for the "Machine" column. When I look at the raw logs, however, I see that every log has a non-empty "Machine" field.

I'm assuming something must be happening at index time to affect this. Does anyone know why some of my logs wouldn't be recognizing the field? I'm running Splunk 6.1.4 and the logs are in JSON format. I've specified this in the configuration files (and it seems to be working for at least half of the logs).

Thanks!

[vasanthmss]

Try the following steps,

1. check the extracted fields - [ what are the fields got extracted?]
2. Validate you JSON by copy and paste the _raw - http://jsonviewer.stack.hu/ or any
3. share the sample JSON and Extracted fields. May be like, events.{}

I guess there is some issue with the data. JSON object was not be formed correctly. Ensure that members / Objects are separated by comma(,).

[bruceclarke]

Can someone shed some light on this? I'm still seeing it after upgrading to Splunk 6.2, and I have no idea how to begin debugging!

[vasanthmss]

could you provide supportive sample data?

[bruceclarke]

I'm not entirely sure what you want to see, but if I run sourcetype=exceptions | table _time Machine, I see the following raw data for every log:

{ [-] 
    Exception: { [+] 
   } 
    Machine:  SOME_VM_MACHINE 
    SessionInfo: { [+] 
   } 
    Timestamp:  2014-12-08T22:39:51.3375753+00:00 
}

But in the table I get a NULL value for Machine for some of the logs. It appears to be completely random (i.e. it's not just one Machine or one indexer that misses the Machine field consistently.