I've seen related questions on this subject, but I'm a total newb to splunk so I can't figure out if the problem they're having is the same or not. To make it worse, I have no idea where to put their answers in the config tree!
The semi-problem:
We're dumping JSON into log files on our servers. Everything is properly escaped and 1 JSON object appears per line in the logs.
However, we noticed that Splunk has less records that our other logging stack.
In searching, we realized that 80% or so of our events has a linecount of 1, while others have 2-10 on average and one has 2000.
I'm told I should create a new sourcetype for my log files with a custom LINE_BREAK so that Splunk correctly parses our 1 JSON object per Event.
I'd like to know the following:
1) What could be causing Splunk to not be able to parse my JSON logs, when others (td-agent/logstash) are able to no problem?
2) Is there a key/value format (lots of text like stacktraces) which Splunk could do better with?
3) If JSON is fine, what do I need to modify so that Splunk is able to parse my logs correctly?
Thank you and apologies for the potential dups!
... View more