How to collect a huge volume of data?

[Thulasinathan_M]

Hello Splunk Experts,

I'm searching for ERRORS and WARN in the application from different servers and trying to collect these log lines to a stored area(Summary Index - may be Sourcetype) to avoid searching again & again on a huge volume of data. I don't want to use lookup because of the data volume. What is the procedure to get this done. Could someone please assist. Thanks in advance!!

[Thulasinathan_M]

Hi @ITWhisperer . Thanks for the info, using same index would be fine just I want it to get written in either a new file in new soucetype or a new log file on same sourcetype. I've checked with our Admins, they're advising to do it application level. So just trying to understand why it's not feasible from splunk.

[ITWhisperer]

You can use the collect command and specify the index and sourcetype (although I think this may add to your license usage, whereas if you default the source type it becomes stash which I think avoids additional license usage, but you should check that). You might want to check back with your admins as to why you can't use a different index (as this also gives you the potential for different retention periods and storage options).

[Thulasinathan_M]

Hi @ITWhisperer ,

Thanks again!! License usage is not a problem in our case. I just need these to stored in a dedicated path, instead of storing in stash. Is there any article/ links I could refere to, could you please kindly help!!

[ITWhisperer]

collect - Splunk Documentation

[ITWhisperer]

You need to create (or ask your administrators to create) an index. You can then schedule a report to extract a subset of the results and add them to the summary index with the collect command. Depending on what you want in your summary index and how you are going to process it afterwards, you may have to consider making the update to the summary index idempotent to avoid adding the same information to the summary index multiple times.