Hi everyone,
Currently I am trying to use the helm chart generated by Splunk App for Infrastructure, to monitor log files other than container logs.
Here is what I add to the helm chart (.\rendered-charts\splunk-connect-for-kubernetes\charts\splunk-kubernetes-logging\templates\configMap.yaml)
source.files.conf: |-
# This fluentd conf file contains sources for log files other than container logs.
<source>
@id files.log
@type tail
@label @SPLUNK
tag tail.logfiles.messages
path /var/log/test.json
pos_file /var/log/splunk-fluentd-message-files.log.pos
path_key source
read_from_head true
<parse>
@type json
</parse>
</source>
I put some random json events on the log files:
cat /var/log/test.json
{"cpu/usage": 0.5, "cpu/rate": 10, "memory/usage": 100, "memory/rss": 90}
{"cpu/usage": 0.5, "cpu/rate": 10, "memory/usage": 100, "memory/rss": 100}
{"cpu/usage": 0.5, "cpu/rate": 10, "memory/usage": 100, "memory/rss": 98}
{"cpu/usage": 0.5, "cpu/rate": 10, "memory/usage": 100, "memory/rss": 93}
And I got "Event field cannot be blank" error.
2019-08-19 04:42:17 +0000 [info]: #0 [files.log] detected rotation of /var/log/test.json; waiting 5 seconds
2019-08-19 04:42:17 +0000 [info]: #0 [files.log] following tail of /var/log/test.json
2019-08-19 04:42:19 +0000 [error]: #0 Failed POST to https://<my-hec-ip>:8088/services/collector , response: {"text":"Event field cannot be blank","code":13,"invalid-event-number":40}
Is there something I need to add to the "filter" and the "match" settings?
Cheers,
Vincent
Hi everyone, thanks for the help from @mmodestino_splunk ,
At the output section of the configMap.yaml, there is a "format" stanza that use a single_value type as the output format. See below code snippet:
<format>
# we just want to keep the raw logs, not the structure created by docker or journald
@type single_value
message_key log
add_newline false
</format>
This single_value output type only send the value of a single field instead of the whole record. You can see this link for more information: https://docs.fluentd.org/formatter/single_value
Two way we can fix this issue:
1. include a fields named "log" in the json payload. (I go for this option because I am not a fluentd expert, so I try to only use the given configurations )
2. Create a new "match" and "format" in the output section, for the particular log files.
Here is the sample of my test log file, which will work with the the existing output plugin of Splunk App for Infrastructure.
{"pod": "pod1", "namespace": "mynamespace", "cluster_name": "minishift", "log": "You can put your payload in here"}
Hope this can help.
Cheers,
Vincent
Hi everyone, thanks for the help from @mmodestino_splunk ,
At the output section of the configMap.yaml, there is a "format" stanza that use a single_value type as the output format. See below code snippet:
<format>
# we just want to keep the raw logs, not the structure created by docker or journald
@type single_value
message_key log
add_newline false
</format>
This single_value output type only send the value of a single field instead of the whole record. You can see this link for more information: https://docs.fluentd.org/formatter/single_value
Two way we can fix this issue:
1. include a fields named "log" in the json payload. (I go for this option because I am not a fluentd expert, so I try to only use the given configurations )
2. Create a new "match" and "format" in the output section, for the particular log files.
Here is the sample of my test log file, which will work with the the existing output plugin of Splunk App for Infrastructure.
{"pod": "pod1", "namespace": "mynamespace", "cluster_name": "minishift", "log": "You can put your payload in here"}
Hope this can help.
Cheers,
Vincent
Nothing wrong, per-se, probably just an empty line in your file...
I upvoted @dhihoriya_splunk, because ideally a filter that strips any event thats blank is a good way to ensure you don’t run into this in other logs
@mmodestino_splunk Thank you for upvoting my answer.
np, will check to see if there is a built in mitigation to this in newer images. Pretty sure we added one....
@season88481 what version of the fluentd-hec images are you using?
Thanks @mmodestino_splunk , I am using the default helm chart built by the Splunk App for Infrastructure. By looking at the daemonset.yaml, the image is "splunk/fluentd-hec:1.1.1".
Also, I cannot see any blank line on the log file.
Add these to the output.conf, but still have the same "Event field cannot be blank error":
<filter tail.logfiles.**>
@type jq_transformer
jq 'if .record.log == "\n" then .record.log = "E" else .record.log = .record.log end | .record'
</filter>
I haven't changed anything on the "match *" section though. In order to monitor other log files, is there anything I need to update on the "match *" section?
Many thanks.
Cheers,
Vincent
yeah i think theres a few things that could be biting you.
can you please dump the config of your logging pod? We dump the rendered configs in the first 1000 lines when the pod spins up and is helpful to debug the filters.
Also, if you want to hit me up on slack i can probably help more real time. splk.it/slack (my handle in mattymo)
I think we just need to ensure your source, filters, and output config yield the right record.
Hi @season88481
It is the issue in which event
field passed to Splunk is empty and which is probably caused by a log record containing a blank message
value.
I think you need to add the filter as below in output.conf: |-
of configMap, It will resolve the issue of blank event as It will filter the logs with empty value also: (empty logs are shown as E
in Splunk)
# ensure we do not have empty line logs, they cannot be ingested by Splunk and result in 400 response from
# the Splunk HEC
<filter tail.containers.**>
@type jq_transformer
jq 'if .record.log == "\n" then .record.log = "E" else .record.log = .record.log end | .record'
</filter>
Thanks for your reply dhihoriya. I find the issue actually with my tail input doesn't match the format at the output settings.
I accepted my own answer but they give the karma back to myself. I will reward this to you.
Thank you @season88481