All Apps and Add-ons

fluentd monitor log files and send to Splunk HEC

season88481
Contributor

Hi everyone,

Currently I am trying to use the helm chart generated by Splunk App for Infrastructure, to monitor log files other than container logs.

Here is what I add to the helm chart (.\rendered-charts\splunk-connect-for-kubernetes\charts\splunk-kubernetes-logging\templates\configMap.yaml)

  source.files.conf: |-
    # This fluentd conf file contains sources for log files other than container logs.
    <source>
      @id files.log
      @type tail
      @label @SPLUNK
      tag tail.logfiles.messages
      path /var/log/test.json
      pos_file /var/log/splunk-fluentd-message-files.log.pos
      path_key source
      read_from_head true
      <parse>
        @type json
      </parse>
    </source>

I put some random json events on the log files:

cat /var/log/test.json
{"cpu/usage": 0.5, "cpu/rate": 10, "memory/usage": 100, "memory/rss": 90}
{"cpu/usage": 0.5, "cpu/rate": 10, "memory/usage": 100, "memory/rss": 100}
{"cpu/usage": 0.5, "cpu/rate": 10, "memory/usage": 100, "memory/rss": 98}
{"cpu/usage": 0.5, "cpu/rate": 10, "memory/usage": 100, "memory/rss": 93}  

And I got "Event field cannot be blank" error.

2019-08-19 04:42:17 +0000 [info]: #0 [files.log] detected rotation of /var/log/test.json; waiting 5 seconds
2019-08-19 04:42:17 +0000 [info]: #0 [files.log] following tail of /var/log/test.json
2019-08-19 04:42:19 +0000 [error]: #0 Failed POST to https://<my-hec-ip>:8088/services/collector , response: {"text":"Event field cannot be blank","code":13,"invalid-event-number":40}

Is there something I need to add to the "filter" and the "match" settings?

Cheers,
Vincent

0 Karma
1 Solution

season88481
Contributor

Hi everyone, thanks for the help from @mmodestino_splunk ,

At the output section of the configMap.yaml, there is a "format" stanza that use a single_value type as the output format. See below code snippet:

        <format>
          # we just want to keep the raw logs, not the structure created by docker or journald
          @type single_value
          message_key log
          add_newline false
        </format>

This single_value output type only send the value of a single field instead of the whole record. You can see this link for more information: https://docs.fluentd.org/formatter/single_value

Two way we can fix this issue:
1. include a fields named "log" in the json payload. (I go for this option because I am not a fluentd expert, so I try to only use the given configurations )
2. Create a new "match" and "format" in the output section, for the particular log files.

Here is the sample of my test log file, which will work with the the existing output plugin of Splunk App for Infrastructure.

 {"pod": "pod1", "namespace": "mynamespace", "cluster_name": "minishift",  "log": "You can put your payload in here"}

Hope this can help.

Cheers,
Vincent

View solution in original post

0 Karma

season88481
Contributor

Hi everyone, thanks for the help from @mmodestino_splunk ,

At the output section of the configMap.yaml, there is a "format" stanza that use a single_value type as the output format. See below code snippet:

        <format>
          # we just want to keep the raw logs, not the structure created by docker or journald
          @type single_value
          message_key log
          add_newline false
        </format>

This single_value output type only send the value of a single field instead of the whole record. You can see this link for more information: https://docs.fluentd.org/formatter/single_value

Two way we can fix this issue:
1. include a fields named "log" in the json payload. (I go for this option because I am not a fluentd expert, so I try to only use the given configurations )
2. Create a new "match" and "format" in the output section, for the particular log files.

Here is the sample of my test log file, which will work with the the existing output plugin of Splunk App for Infrastructure.

 {"pod": "pod1", "namespace": "mynamespace", "cluster_name": "minishift",  "log": "You can put your payload in here"}

Hope this can help.

Cheers,
Vincent

View solution in original post

0 Karma

mattymo
Splunk Employee
Splunk Employee

Nothing wrong, per-se, probably just an empty line in your file...

I upvoted @dhihoriya_splunk, because ideally a filter that strips any event thats blank is a good way to ensure you don’t run into this in other logs

0 Karma

dhihoriya_splun
Splunk Employee
Splunk Employee

@mmodestino_splunk Thank you for upvoting my answer.

0 Karma

mattymo
Splunk Employee
Splunk Employee

np, will check to see if there is a built in mitigation to this in newer images. Pretty sure we added one....

@season88481 what version of the fluentd-hec images are you using?

0 Karma

season88481
Contributor

Thanks @mmodestino_splunk , I am using the default helm chart built by the Splunk App for Infrastructure. By looking at the daemonset.yaml, the image is "splunk/fluentd-hec:1.1.1".

Also, I cannot see any blank line on the log file.

Add these to the output.conf, but still have the same "Event field cannot be blank error":

          <filter tail.logfiles.**>
             @type jq_transformer
             jq 'if .record.log == "\n" then .record.log = "E" else .record.log = .record.log end | .record'
           </filter>

I haven't changed anything on the "match *" section though. In order to monitor other log files, is there anything I need to update on the "match *" section?

Many thanks.

Cheers,
Vincent

0 Karma

mattymo
Splunk Employee
Splunk Employee

yeah i think theres a few things that could be biting you.

can you please dump the config of your logging pod? We dump the rendered configs in the first 1000 lines when the pod spins up and is helpful to debug the filters.

Also, if you want to hit me up on slack i can probably help more real time. splk.it/slack (my handle in mattymo)

I think we just need to ensure your source, filters, and output config yield the right record.

0 Karma

dhihoriya_splun
Splunk Employee
Splunk Employee

Hi @season88481

It is the issue in which event field passed to Splunk is empty and which is probably caused by a log record containing a blank message value.

I think you need to add the filter as below in output.conf: |- of configMap, It will resolve the issue of blank event as It will filter the logs with empty value also: (empty logs are shown as E in Splunk)

# ensure we do not have empty line logs, they cannot be ingested by Splunk and result in 400 response from
      # the Splunk HEC
      <filter tail.containers.**>
        @type jq_transformer
        jq 'if .record.log == "\n" then .record.log = "E" else .record.log = .record.log end | .record'
      </filter>

season88481
Contributor

Thanks for your reply dhihoriya. I find the issue actually with my tail input doesn't match the format at the output settings.

I accepted my own answer but they give the karma back to myself. I will reward this to you.

0 Karma

dhihoriya_splun
Splunk Employee
Splunk Employee

Thank you @season88481

0 Karma
Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.