All Apps and Add-ons

We are thinking of moving to Azure Kontainer Service (AKS), is there any splunk API plugin for fluentD to push data onto Splunk?

codeRelix
New Member

We are thinking of moving to Azure K(C)ontainer Service (AKS), is there any splunk API plugin for fluentD to push data onto Splunk? We don't want to run a native splunk process that does so today.

0 Karma

mattymo
Splunk Employee
Splunk Employee

UPDATE: Check out Splunk Connect for Kubernetes here: https://github.com/splunk/splunk-connect-for-kubernetes

Hi codeRelix,

We are currently working on many items with regards to k8s and the various cloud providers. In fact, I will be working with the Azure team in the coming days to get Azure + k8s + Splunk all sorted out.

I'd be glad to talk with you further on this topic and get you involved in some of our early access programs when they are ready.

What do you mean by native Splunk process? Is it installed directly on the host?

In the meantime,

I have had success with this fluentd plugin, here:

https://github.com/cmeerbeek/fluent-plugin-splunkhec

I am also working with our PMs/ENG to look at the possibility of an officially supported fluentd plugin. No guarantees though. 🙂

All that said, I will tell you that I have had much better success with the Splunk UF running as a Daemonset in my research.

Especially, when it comes to the shape of the data that you will be indexing.

Docker logs wrapped in JSON or fluentd logs wrapped in EVEN MORE JSON are a nightmare for searching once in Splunk, and won't allow you to use our exising TAs, don't support multiline logs, and cost you more at index time.

I have pushed an early prototype of the UF with props.conf that will unwrap the json logs to our github:

https://github.com/splunk/docker-itmonitoring/blob/7.0.0-k8s/README-k8s.md

Should be merged to master shortly, and I will follow up with some how-tos on supporting multiline logs etc.

Anyways, feel free to come join us in the slack chat (splk.it/slack) in the #kubernetes room. my handle is @mattymo, id be glad to set up some time to show you what I am doing in my lab 🙂

Matt

- MattyMo
0 Karma

mallempatisreed
Explorer

HI Matthew,

We are planning to migrate to cloud Azure and considering the above option for forwarding logs to SPlunk. Does the solution you have mentioned above worked without any issues.

Thanks,
Sree

0 Karma

abdulc
New Member

actually setting the port to SSL (443) works

0 Karma

abdulc
New Member

But im getting alot of trash logs with a lot of trailing slashes

0 Karma

mattymo
Splunk Employee
Splunk Employee

lets move that to it's own post, or come chat in the slack chat! splk.it/slack

my username is @mattymo.

- MattyMo
0 Karma

abdulc
New Member

made a request to join your slack, thanks

0 Karma

abdulc
New Member
Splunk addon for Kubernetes fails to connect to Splunk using fluentd Splunk plugin 

    2018-07-01 22:39:04 +0000 [debug]: #0 Sending 305313 bytes to Splunk.
    2018-07-01 22:39:10 +0000 [debug]: #0 Received new chunk, size=1186
    2018-07-01 22:39:16 +0000 [debug]: #0 Received new chunk, size=19231
    2018-07-01 22:40:04 +0000 [warn]: #0 thread exited by unexpected error plugin=Fluent::Plugin::SplunkHecOutput title=:"hec_worker_https://MyCloudInstance.cloud.splunk.com:8088/services/collector" error_class=Net::OpenTimeout error="execution expired"
    2018-07-01 22:40:04 +0000 [error]: #0 unexpected error error_class=Net::OpenTimeout error="execution expired"
    2018-07-01 22:40:04 +0000 [error]: #0 /usr/local/lib/ruby/2.5.0/net/http.rb:937:in initialize' 2018-07-01 22:40:04 +0000 [error]: #0 /usr/local/lib/ruby/2.5.0/net/http.rb:937:inopen'
    2018-07-01 22:40:04 +0000 [error]: #0 /usr/local/lib/ruby/2.5.0/net/http.rb:937:in block in connect' 2018-07-01 22:40:04 +0000 [error]: #0 /usr/local/lib/ruby/2.5.0/timeout.rb:103:intimeout'
    2018-07-01 22:40:04 +0000 [error]: #0 /usr/local/lib/ruby/2.5.0/net/http.rb:935:in connect' 2018-07-01 22:40:04 +0000 [error]: #0 /usr/local/lib/ruby/2.5.0/net/http.rb:920:indo_start'
    2018-07-01 22:40:04 +0000 [error]: #0 /usr/local/lib/ruby/2.5.0/net/http.rb:915:in start' 2018-07-01 22:40:04 +0000 [error]: #0 /usr/local/bundle/gems/net-http-persistent-3.0.0/lib/net/http/persistent.rb:692:instart'
    2018-07-01 22:40:04 +0000 [error]: #0 /usr/local/bundle/gems/net-http-persistent-3.0.0/lib/net/http/persistent.rb:622:in connection_for' 2018-07-01 22:40:04 +0000 [error]: #0 /usr/local/bundle/gems/net-http-persistent-3.0.0/lib/net/http/persistent.rb:927:inrequest'
    2018-07-01 22:40:04 +0000 [error]: #0 /usr/local/bundle/gems/fluent-plugin-splunk-hec-1.0.0/lib/fluent/plugin/out_splunk_hec.rb:343:in send_to_hec' 2018-07-01 22:40:04 +0000 [error]: #0 /usr/local/bundle/gems/fluent-plugin-splunk-hec-1.0.0/lib/fluent/plugin/out_splunk_hec.rb:311:inblock in start_worker_threads'
    2018-07-01 22:40:04 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.2.0/lib/fluent/plugin_helper/thread.rb:78:in block in thread_create' #<Thread:0x00007f0b0f5c91f0@/usr/local/bundle/gems/fluentd-1.2.0/lib/fluent/plugin_helper/thread.rb:70 run> terminated with exception (report_on_exception is true): /usr/local/lib/ruby/2.5.0/net/http.rb:937:ininitialize': execution expired (Net::OpenTimeout)
    from /usr/local/lib/ruby/2.5.0/net/http.rb:937:in open' from /usr/local/lib/ruby/2.5.0/net/http.rb:937:inblock in connect'
    from /usr/local/lib/ruby/2.5.0/timeout.rb:103:in timeout' from /usr/local/lib/ruby/2.5.0/net/http.rb:935:inconnect'
    from /usr/local/lib/ruby/2.5.0/net/http.rb:920:in do_start' from /usr/local/lib/ruby/2.5.0/net/http.rb:915:instart'
    from /usr/local/bundle/gems/net-http-persistent-3.0.0/lib/net/http/persistent.rb:692:in start' from /usr/local/bundle/gems/net-http-persistent-3.0.0/lib/net/http/persistent.rb:622:inconnection_for'
    from /usr/local/bundle/gems/net-http-persistent-3.0.0/lib/net/http/persistent.rb:927:in request' from /usr/local/bundle/gems/fluent-plugin-splunk-hec-1.0.0/lib/fluent/plugin/out_splunk_hec.rb:343:insend_to_hec'
    from /usr/local/bundle/gems/fluent-plugin-splunk-hec-1.0.0/lib/fluent/plugin/out_splunk_hec.rb:311:in block in start_worker_threads' from /usr/local/bundle/gems/fluentd-1.2.0/lib/fluent/plugin_helper/thread.rb:78:inblock in thread_create'
    2018-07-01 22:40:04 +0000 [info]: fluent/log.rb:322:info: Worker 0 finished unexpectedly with status 1
0 Karma

mattymo
Splunk Employee
Splunk Employee

looks like reachability issues, try hitting hec with this test from a pod that has curl:

curl -k https://<host>:8088/services/collector -H 'Authorization: Splunk <token>' -d '{"sourcetype": "mysourcetype", "event":"Hello, World!"}'

- MattyMo
0 Karma

mattymo
Splunk Employee
Splunk Employee

Hey mallempatisreedhar!

We have recently announce splunk connect for kubernetes, which is based on fluentd and heapster.

I would suggest that it should be able to run in AKS, give it a shot!

https://github.com/splunk/splunk-connect-for-kubernetes

- MattyMo
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...