Hi, I am not sure the purpose of roll forward existing index. How do I use "collect" to write data in a scheduled report?  My understanding collect is a manual push. I am looking for automatic update daily. Where does KV store lookup save the data?  How do I move the DBXquery to KV store?  Does it require "admin"? What do you think about the "append command" in the previous post? Thank you so much for your help.

Hi, What I meant by splitting the data is to split the number of rows. So, if my query has 200k rows, splitting into 2, it becomes 100k rows. Or @marnall suggested split into a smaller queries..  not sure if it's possible since I have a large query involving multiple DBs I don't know how to do this in a scheduled report and write it into the same summary index with the same _time setting.  Please suggest. Thanks

If the data is in an index, it must be placed there with a timestamp, so if an app was updated 45 days ago that info ingested to Splunk and the Splunk _time timestamp is 45 days ago, the only way you can find that data is to search that data with a time range that encompasses that time. But of course you don't know when it was updated. I have used a technique in the past where I roll forward existing index data by running a search at say 1am, that will search for data from yesterday earliest=-d@d and latest=@d and does a stats latest(*) as * by X Y Z Then use collect to write that data to the index with the current timestamp (1am) so effectively all rolled forward items from the previous day PLUS any new items that are added in the same day and collected to the same index. Naturally you would need to massage the data so that any updates would then shift previous->discard, current->previous,  new->current. That means your previous day's data is always the latest view of all versions. Not sure if this helps. Have you tried using kv store for the lookup - that's another story and you can use some accelerated keys for that data that may make it perform faster than standard lookup.  

Hi, Thanks for your help. No I do not care the order. I am afraid if I split the data and re-combine them it will return duplicate/missing data as it doesn't have a unique identifier. Also I don't know how to split the data and keeping the same _time.  Please help answer this. Thanks How do I split my query from DBXquery (eg. 200k rows)and push it into a Summary Index at the same time?

Hello, Thank you again for your help. Just to clarify,  I cannot set _time to the exact time every time I query the data, , correct? So, I need to filter the data last update, if I want to get the most recent copy. I currently are using CSV as a lookup, but the limitation is the size like you mentioned. I am trying to replace CSV lookup by doing the following:   Please let me know what you think. https://community.splunk.com/t5/Splunk-Search/How-to-perform-lookup-from-index-search-with-dbxquery/m-p/650654 | index=vulnerability_index | table ip_address, vulnerability, score | append [| dbxquery query="select * from tableCompany"] | stats values(*) as * by ip_address

I'm looking for a particular string in the list of strings.   The "a" in the first part is not from a field,  it's just a string that I'm trying to compare against.     I'm trying to implement the following logic in python "word1" in ['word1', 'word2', ..., 'word_x', ]

OK, so I assume the list of strings are a, b and c, but what is the FIRST "a" in your statement "a" in ("a","b","c") as I said in my first reply, the search statement is comparing FIELD "a" with the string values of the IN part. Is your field "a" something that has those values? For you to get all results simply adding that AND statement, would imply field "a" has "a" in all your events.

|search (Check_Feature_Availability=false) AND NOT ("Choice1" IN ("Choice1", "Choice2", "Choice3")) OR (Check_Feature_Availability=true) AND ("Choice1" IN ("Choice1", "Choice2", "Choice3"))   The list is really all the options from a multi-valued dropdown menu.  The values are all different.  

I am trying to dynamically alter my searched data, by utilizing a value from my data source (Check_Feature_Availability - boolean data) with a selected value from a multi-dropdown in the dashboard (list of strings).