topic Re: How to regularly write filtered events to a new index? in Getting Data In

How to regularly write filtered events to a new index?

xsstest — Wed, 11 Oct 2017 15:55:37 GMT

if I have an index test, the index has too many events, I need to filter by keyword and write the result to the index Useful_logs.

for example:

Filter conditions:

index=test sourcetype=abc "login" "user" "deviceId"

then at the zero of every day，filter the events of the previous day

Write the filtered event to index Useful_logs

Finally, I can use index=Useful_logs to search for the log I want.

Of course, maybe some friends will let me configure the "transforms.conf" file.
But I want to keep all the logs of the test index, but also write useful logs to the new index （the Useful_logs index）.

So what should I do?

Re: How to regularly write filtered events to a new index?

xsstest — Wed, 11 Oct 2017 15:57:58 GMT

Use the summary index?

Re: How to regularly write filtered events to a new index?

gcusello — Wed, 11 Oct 2017 16:14:09 GMT

Hi xsstest,
reindexing already indexed logs isn't a good idea because you have double license consuption!

If you can identify the logs you want using source or sourcetype, you can see that, also having many different events in your index, your search will be very quick.

If you want anyway to put the selected logs in another index you could do two things:

put the selected logs in a summary index,
extract the selected logs and reindex them in the new index.

First solution has the advantage that you don't have double license consuption and that your searches will be very performant, but in this way you have ro rebuild all your searches and field extraction because you have to use a differtent syntax in searches.
To have this you have to run something like this:

index=test sourcetype=abc "login" "user" "deviceId"
| table _time _raw
| tscollect namespace=Useful_logs

to access this logs use

| tstats count FROM Useful_logs GROUPBY _time _raw | ...

To run the second solution run your search index=test sourcetype=abc "login" "user" "deviceId" and then download results in raw format that you can newly ingest in the new index.
This second solution has double license consuption but your searches will be the same (only different index).

Bye.
Giuseppe

Re: How to regularly write filtered events to a new index?

s2_splunk — Wed, 11 Oct 2017 16:54:58 GMT

Why not simply setup an eventtype that returns just your useful data?

Re: How to regularly write filtered events to a new index?

xsstest — Fri, 13 Oct 2017 11:12:37 GMT

What should I do ?

Re: How to regularly write filtered events to a new index?

s2_splunk — Fri, 13 Oct 2017 23:26:41 GMT

The simplest is to define an eventtype, as suggested, and use that for searching your useful logs. I'd recommend reading up on it and applying it to your problem to see if that helps.
If that doesn't solve your problem, please describe your problem in more detail.