Accessing Elasticsearch Data from Splunk Using SPL (Without Data Duplication)

[kn450]

Hi Splunk Community,

I’m working on a use case where data is stored in Elasticsearch, and I’d like to use Splunk solely as an interface for visualizing and querying the data using SPL (Search Processing Language) — without ingesting or storing the data again in Splunk, to avoid duplication and unnecessary storage costs.

My main questions are:

• Is there a way to connect Splunk directly to Elasticsearch as an external data source?

• Can Splunk query external data (like from Elasticsearch) using SPL, without indexing it?

• Are there any available add-ons, modular inputs, or scripted solutions that allow this type of integration?

• Is this approach officially supported by Splunk, or would it require a custom integration?

I’m aware that tools like Logstash or Kafka can be used to bring data into Splunk, but that’s exactly what I’m trying to avoid — I don’t want to duplicate the data storage.

If anyone has experience with a similar setup, or any recommendations, I’d greatly appreciate your input.

Thanks in advance!

[tscroggins]

Hi @kn450,

With respect to your prior comments: "... it's important to note that the queries used are not written in Splunk’s native SPL language; instead, they rely on Elasticsearch queries. This limits the integration with some of Splunk’s core functionalities and does not provide the desired level of efficiency in terms of performance and deep analysis."

I use custom generating commands to run Elasticsearch searches, and I treat the results as if they came from a similar base SPL command. I agree the ideal would be a virtual index or federated search that compiles a search command into equivalent Elasticsearch Query DSL, for example, but that isn't presently feasible.

What Splunk functionality would you like to use with custom search commands, including those from apps on Splunkbase, that you cannot use? Do you have specific use cases in mind?

[richgalloway]

Splunk cannot natively query Elasticsearch or use it as an external data source.  There is, however, an add-on to make some of that possible.  See https://splunkbase.splunk.com/app/6477

[kn450]

Thank you for your input.

We have indeed used the mentioned add-on and were able to successfully retrieve data from Elasticsearch.
However, it's important to note that the queries used are not written in Splunk’s native SPL language; instead, they rely on Elasticsearch queries. This limits the integration with some of Splunk’s core functionalities and does not provide the desired level of efficiency in terms of performance and deep analysis.

We are currently looking for best practices and would prefer to adopt a solution that has been widely used over a long period without issues, offering better integration and higher performance with Splunk.

If you have any proven experiences or reliable recommendations, we would appreciate you sharing them.

[livehybrid]

Hi @kn450 

You might want to look at ElasticSPL Add-on for Splunk app on Splunkbase which allows this, the mean features include:

Query Elasticsearch in an ad-hoc fashion using DSL, Lucene or ES|QL search statements for time-series data using elasticadhoc and elasticquery
Query Elasticsearch in an ad-hoc fashion using DSL search statements for aggregated data using elasticadhocstats and elasticquerystats

For more info and docs please see https://docs.datapunctum.com/elasticspl

The app contains custom commands that allow you to search Elastic from Splunk without having to use a modular input to ingest the data into Splunk.

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[kn450]

We have indeed used the mentioned add-on and were able to successfully retrieve data from Elasticsearch.
However, it's important to note that the queries used are not written in Splunk’s native SPL language; instead, they rely on Elasticsearch queries. This limits the integration with some of Splunk’s core functionalities and does not provide the desired level of efficiency in terms of performance and deep analysis.

We are currently looking for best practices and would prefer to adopt a solution that has been widely used over a long period without issues, offering better integration and higher performance with Splunk.

If you have any proven experiences or reliable recommendations, we would appreciate you sharing them

[livehybrid]

Hi @kn450 

Apologies I didnt realise you wanted to search Elastic in Native SPL, I inferred the requirement as being able to use DSL within SPL. 

It sounds like what you are looking for is Federated Search (" to search datasets outside of your local Splunk platform deployment.") against Elastic, which is not currently possible. 

There are currently no apps/add-ons which translate SPL into DSL for searching Elastic.

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing