All Apps and Add-ons

Working of Elastic search Add on ... Reindexing of same data ?

nawazns5038
Builder

Hi @gaurav_maniar @larmesto

I have enabled the modular input for getting data from the Elasticsearch into Splunk and it is working and getting the data in.

But I am not sure how the interval is suppose to work.

It is pulling in the same data again and again with everytime the input runs in the interval.

Is it supposed to work that way ?

Thanks,
Nawaz.

sittipornbaycom
Observer

Hi Nawaz

 

Please you can share python code and step config for get log ELK.

I'd like get log ELK and found issue same you.

 

Thanks

0 Karma

harshpatel
Contributor

Hi @nawazns5038

The Add-On is using always the same range you specify during the input creation. I found this piece of code in the app which is executed each time of input invocation to create filter for API call.

body={"query": {
    "bool": {"filter": [ 
        { "range":  { opt_date_field_name: {"gte": opt_greater_or_equal, "lte": opt_lower_or_equal}}}
        ]
        }}
      }

There is no checkpoint mechanism. It will pull the same time range data on each invocation. If you want to increment the time range on each invocation you may have to modify the script to store epoch time in KVStore as a checkpoint and use that on next invocation for getting new data.

Hope this was helpful,
Harsh

0 Karma

highsplunker
Contributor

Hi Nawaz,
Why did you decide you yous this ELK Connector instead of e.g. standard Splunk Forwarder?
What's your use case?

Best regards.

0 Karma

nawazns5038
Builder

I am trying to get data from the elastic search instance . I am not sure if we can install a forwarder there .

Thanks
Nawaz

0 Karma

highsplunker
Contributor

Thanks for your quick response!

Ok, so are satisfied with the App? Reliable? Bugs, issues?
What's the format of files where your ELK instance resides?
Plain files? Log/CSV/txt...

I'm just wondering, why do I need this connector if I can install Splunk Forwarder on my ELK server... Advantages of FW: online input, strong reliability, flexible filtering. Advantages of the Connector: no installation.

0 Karma

nawazns5038
Builder

I am not 100% satisfied with the Addon but it is working okay after making some python changes.

The only disadvantage is it not real time , It can only work for pulling all the data once and not updating on a regular basis.

The files are in JSON format.

I cannot install the forwarder on the ELK server. I am not sure if it works the same way like Splunk like storing the data in buckets or something., I can look into installing only if the data on the ELK side is stored in the form of logs.

0 Karma

ebb
Observer

Hello @nawazns5038 ,

it's been a long time since your post but I'm facing the same problem.

It seems to me that the App can not be use to continually ingest data (give a start and "now" as end date) at given intervals. Did you find a workaround for this or got some reply from the creator? 

Thanks in advanced for sharing this info!

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!