All Apps and Add-ons

How to get data in whit elasticsearch-data-integrator app?

Expl
Loves-to-Learn

hi,

We need to configure the TA-elasticsearch-data-integrator---modular-input app and we receive data.
The problem is : we do receive data, but not all...

here is the app conf:

Name ALogName
Intervalle 3600
Index MyIndex
Statut Activated
Elasticsearch instance URL: MyName
Port #: MyPort
Use SSL 1
Verify Certs 1
CA Certs Path: /my/ca.pem
User: MyUser
Secret / Password: MyPassword
Elasticsearch Indice: MyIndice
Elasticsearch Date field name: @timestamp
Time Preset: 30d
Custom Source Type: json

If i use CLI, with the exact same configuration, except i use match, I receive the good datas.

curl -u "MyUser:MyPassword" -k "https://MyName:MyPort/MyIndice/_search?&scroll=1m&size=1000" -H 'Content-Type: application/json' -d'{"query": {"match": {"message": "MyMessage"}}, "sort": { "@timestamp": "desc" }}'
{"_scroll_id":"[...]","took":695,"timed_out":false,"_shards":{"total":8,"successful":8,"skipped":0,"failed":0},"hits":{"total":{"value":3,"relation":"eq"},"max_score":null,"hits":[...MyData...]

here is the logs of the app:

2021-12-06 13:29:00,073 INFO pid=26584 tid=MainThread file=base.py:log_request_success:271 | POST https://MyName:MyPort/MyIndice/_search?scroll=2m&size=1000 [status:200 request:0.870s]
2021-12-06 13:37:12,701 WARNING pid=26584 tid=MainThread file=base.py:log_request_fail:299 | POST https://MyName:MyPort/_search/scroll [status:404 request:0.076s]
2021-12-06 13:37:12,703 INFO pid=26584 tid=MainThread file=base.py:log_request_success:271 | DELETE https://MyName:MyPort/_search/scroll [status:404 request:0.002s]
2021-12-06 13:37:12,705 ERROR pid=26584 tid=MainThread file=base_modinput.py:log_error:309 | Get error when collecting events.
Traceback (most recent call last):
File "/opt/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/aob_py3/modinput_wrapper/base_modinput.py", line 128, in stream_events
self.collect_events(ew)
File "/opt/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/elasticsearch_json.py", line 104, in collect_events
input_module.collect_events(self, ew)
File "/opt/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/input_module_elasticsearch_json.py", line 109, in collect_events
for doc in res:
File "/opt/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/elasticsearch/helpers/actions.py", line 589, in scan
body={"scroll_id": scroll_id, "scroll": scroll}, **scroll_kwargs
File "/opt/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/elasticsearch/client/utils.py", line 168, in _wrapped
return func(*args, params=params, headers=headers, **kwargs)
File "/opt/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/elasticsearch/client/__init__.py", line 1513, in scroll
"POST", "/_search/scroll", params=params, headers=headers, body=body
File "/opt/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/elasticsearch/transport.py", line 415, in perform_request
raise e
File "/opt/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/elasticsearch/transport.py", line 388, in perform_request
timeout=timeout,
File "/opt/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/elasticsearch/connection/http_urllib3.py", line 275, in perform_request
self._raise_error(response.status, raw_data)
File "/opt/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/elasticsearch/connection/base.py", line 331, in _raise_error
status_code, error_message, additional_info
elasticsearch.exceptions.NotFoundError: NotFoundError(404, 'search_phase_execution_exception', 'No search context found for id [9884105]')

Any help would be great, thanks!

Labels (2)
0 Karma

Brenny
New Member

Hi, we facing the same problem.

Did you get any help ?

0 Karma
Get Updates on the Splunk Community!

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...

Splunkbase | Splunk Dashboard Examples App for SimpleXML End of Life

The Splunk Dashboard Examples App for SimpleXML will reach end of support on Dec 19, 2024, after which no new ...

Understanding Generative AI Techniques and Their Application in Cybersecurity

Watch On-Demand Artificial intelligence is the talk of the town nowadays, with industries of all kinds ...