topic Re: Fetching logs from Elasticsearch in Getting Data In

Fetching logs from Elasticsearch

Mojal — Wed, 14 Aug 2024 09:50:46 GMT

Hi,

I have an Elastic DB that receive logs from various services directly and I want to send these logs to Splunk Enterprise.
Is there any documentation about install instruction of the Elasticsearch Data Integrator?
I couldn't config it to make it work and I don't find any documentation on how to install and configure this add-on.

Please help me with that.@larmesto

Kind Regards,
Mohammad

Re: Fetching logs from Elasticsearch

marnall — Wed, 14 Aug 2024 20:35:47 GMT

Are you able to find working values for the inputs of the app? It seems like you can enter in your Elasticsearch domain name, port, user, secret, interval, etc, then theoretically it should pull data from your elasticsearch instance.

If you enter in the values but it does not work, then you could try searching your _internal index for keywords like "elasticsearch" to see if the app generates any errors that would explain why it is not pulling data from your elasticsearch instance.

Re: Fetching logs from Elasticsearch

Mojal — Sun, 18 Aug 2024 06:39:22 GMT

Thank you for your help @marnall

You are correct, I did enter my elastic search information in the app but it did not pull any data.

When I go thorough _Internal logs, I see some error logs that contains users like proxy and root, but I dont have any of this users in my configs nor in my database credentials and also I didnt active the proxy option in the Elasticsearch Data Integrator add-on.

I could mention that I can connect to elastic database via curl from splunk server which means the connection is open.

Re: Fetching logs from Elasticsearch

marnall — Sun, 18 Aug 2024 08:26:07 GMT

As a test, does the app still complain when you add a filler proxy user+password combination in the settings?

There is also a different app that is often suggested for the use case of searching Elasticsearch data from Splunk. If it is not strictly necessary for you to migrate the data from Elasticsearch into Splunk, then this may be an option: https://github.com/brunotm/elasticsplunk

Re: Fetching logs from Elasticsearch

Mojal — Sun, 18 Aug 2024 12:46:00 GMT

Yes, still it does generate proxy logs even when fill fake settings.

The problem with those apps you mentioned is that they dont support authentication.

My Elasticsearch database is protected by authentication.

Re: Fetching logs from Elasticsearch

canoop — Tue, 27 Aug 2024 22:11:51 GMT

Hi @Mojal @marnall

I am facing the same issue with my Splunk Cluster. Were y'all able to find any workarounds/solutions?

P.S: I have deployed the splunk cluster via splunk-operator in my kubernetes environment.