Hi @tuts, Use Elasticsearch Data Integrator - Module Input if your requirements match the following: Simple index list or pattern Single date field Less than or equal to 10,000 documents per sea...
See more...
Hi @tuts, Use Elasticsearch Data Integrator - Module Input if your requirements match the following: Simple index list or pattern Single date field Less than or equal to 10,000 documents per search The add-on uses the Python Elasticsearch client search() method, which wraps the Elasticsearch Search API. The add-on will search for all documents in the configured index list with configured date field values greater than or equal to now minus the configured offset and less than or equal to now. E.g. Given logs-*,metrics-*, @timestamp, and -24h, respectively, the add-on will retrieve documents in pages of 1,000: GET /logs-*,metrics-*/_search?from=0&size=1000
{
"query": {
"bool": {
"must": [
{
"range": {
"@timestamp": {
"gte": "now-24h",
"lte": "now"
}
}
}
]
}
}
} Elasticsearch limits scrolling using the from and size parameters to 10,000 results (10 pages of 1,000 documents). If you need to retrieve more documents per interval or need more control over how search results are presented prior to entering the Splunk ingest pipeline, you should evaluate REST API Module Input or similar solutions. You might also consider writing your own modular input or scripted input. A custom solution would allow to control the query language (Query DSL, ES|QL, SQL, etc.), scrolling, checkpointing, etc. If you have more specific questions, members of the community like me with experience in both Splunk and Elasticsearch can assist.