How to get data in whit elasticsearch-data-integrator app?

Expl — Wed, 09 Aug 2023 16:09:11 GMT

hi,

We need to configure the TA-elasticsearch-data-integrator---modular-input app and we receive data.
The problem is : we do receive data, but not all...

here is the app conf:

Name ALogName
Intervalle 3600
Index MyIndex
Statut Activated
Elasticsearch instance URL: MyName
Port #: MyPort
Use SSL 1
Verify Certs 1
CA Certs Path: /my/ca.pem
User: MyUser
Secret / Password: MyPassword
Elasticsearch Indice: MyIndice
Elasticsearch Date field name: @timestamp
Time Preset: 30d
Custom Source Type: json

If i use CLI, with the exact same configuration, except i use match, I receive the good datas.

curl -u "MyUser:MyPassword" -k "https://MyName:MyPort/MyIndice/_search?&scroll=1m&size=1000" -H 'Content-Type: application/json' -d'{"query": {"match": {"message": "MyMessage"}}, "sort": { "@timestamp": "desc" }}'
{"_scroll_id":"[...]","took":695,"timed_out":false,"_shards":{"total":8,"successful":8,"skipped":0,"failed":0},"hits":{"total":{"value":3,"relation":"eq"},"max_score":null,"hits":[...MyData...]

here is the logs of the app:

2021-12-06 13:29:00,073 INFO pid=26584 tid=MainThread file=base.py:log_request_success:271 | POST https://MyName:MyPort/MyIndice/_search?scroll=2m&size=1000 [status:200 request:0.870s]
2021-12-06 13:37:12,701 WARNING pid=26584 tid=MainThread file=base.py:log_request_fail:299 | POST https://MyName:MyPort/_search/scroll [status:404 request:0.076s]
2021-12-06 13:37:12,703 INFO pid=26584 tid=MainThread file=base.py:log_request_success:271 | DELETE https://MyName:MyPort/_search/scroll [status:404 request:0.002s]
2021-12-06 13:37:12,705 ERROR pid=26584 tid=MainThread file=base_modinput.py:log_error:309 | Get error when collecting events.
Traceback (most recent call last):
File "/opt/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/aob_py3/modinput_wrapper/base_modinput.py", line 128, in stream_events
self.collect_events(ew)
File "/opt/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/elasticsearch_json.py", line 104, in collect_events
input_module.collect_events(self, ew)
File "/opt/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/input_module_elasticsearch_json.py", line 109, in collect_events
for doc in res:
File "/opt/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/elasticsearch/helpers/actions.py", line 589, in scan
body={"scroll_id": scroll_id, "scroll": scroll}, **scroll_kwargs
File "/opt/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/elasticsearch/client/utils.py", line 168, in _wrapped
return func(*args, params=params, headers=headers, **kwargs)
File "/opt/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/elasticsearch/client/__init__.py", line 1513, in scroll
"POST", "/_search/scroll", params=params, headers=headers, body=body
File "/opt/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/elasticsearch/transport.py", line 415, in perform_request
raise e
File "/opt/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/elasticsearch/transport.py", line 388, in perform_request
timeout=timeout,
File "/opt/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/elasticsearch/connection/http_urllib3.py", line 275, in perform_request
self._raise_error(response.status, raw_data)
File "/opt/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/elasticsearch/connection/base.py", line 331, in _raise_error
status_code, error_message, additional_info
elasticsearch.exceptions.NotFoundError: NotFoundError(404, 'search_phase_execution_exception', 'No search context found for id [9884105]')

Any help would be great, thanks!

Re: Getting data in whit elasticsearch-data-integrator app

Brenny — Wed, 09 Aug 2023 07:03:36 GMT

Hi, we facing the same problem.

Did you get any help ?

topic How to get data in whit elasticsearch-data-integrator app? in All Apps and Add-ons

How to get data in whit elasticsearch-data-integrator app?

Re: Getting data in whit elasticsearch-data-integrator app