topic Re: Need Help in filtering data ad ingest few of the data in Getting Data In

Need Help in filtering data ad ingest few of the data

satyaallaparthi — Wed, 30 Sep 2020 02:42:12 GMT

Hello,

I have tons of data that are ingesting to some index="abc".

But I want to filter the whole data and want to ingest the log with the words "Events,Transaction,Payment" and then want to route that data to index=event_logs

I wrote the below props and transforms. But no luck.

Props:

TRANSFORMS-filter = null, IQ,Events

Transforms:

[null]
REGEX= .
DEST_KEY = Queue
FORMAT = nullQueue

[IQ]
REGEX= .+(Event|Payment|Transaction).+
DEST_KEY = Queue
FORMAT = indexQueue

[Events]
REGEX= .+(Event|Payment|Transaction).+
DEST_KEY = _MetaData:Index
FORMAT = Event_log

Please do help me with the issue.

Thanks in Advance

Re: Need Help in filtering data ad ingest few of the data

gcusello — Fri, 25 Oct 2019 14:41:20 GMT

Hi satyaallaparthi,
Three notes:
at first, where do you have these conf files?
You have to put them on Indexers or (when present) on Heavy Forwarders, not on Universal Forwarder.
For more information see at https://docs.splunk.com/Documentation/Splunk/7.3.2/Forwarding/Routeandfilterdatad .

Then the regex should be a little different:

REGEX=Event|Payment|Transaction

.
At least, why do you need to override index? is it not possible to set it on inputs.conf?

Ciao.
Giuseppe

Re: Need Help in filtering data ad ingest few of the data

satyaallaparthi — Fri, 25 Oct 2019 14:51:46 GMT

Hello,

Yes, I did tried that before doing this.. but no luck.. that is why I changed the regex..

No, that is not possible to set in Inputs. Because I am getting the data from other instance.

Thanks,

Re: Need Help in filtering data ad ingest few of the data

gcusello — Fri, 25 Oct 2019 14:58:52 GMT

Hi satyaallaparthi,
let me understand:

you have these conf files on Indexer, not on Universal Forwarders and you don't have intermediate Heavy Forwarders;
you tried my rex and you rex both without success;
you restarted Splunk after conf files updates.

Could you try without the third command in props.conf?

TRANSFORMS-filter = null, IQ

Ciao.
Giuseppe

Re: Need Help in filtering data ad ingest few of the data

satyaallaparthi — Fri, 25 Oct 2019 17:21:25 GMT

But I want the third one to route data to new index.

And yes, I have heavy forwarder is between indexer and UF. I restarted the server after I placed. But no luck

Thanks,

Re: Need Help in filtering data ad ingest few of the data

woodcock — Sat, 26 Oct 2019 02:51:50 GMT

Casing matters; use this EXACTLY:

Props:

TRANSFORMS-filter = null_all_my_stuff, unnull_IQ_stuff, stuff_to_different_index

Transforms:

[null_all_my_stuff]
REGEX= .
DEST_KEY = queue
FORMAT = nullQueue

[unnull_IQ_stuff]
REGEX= (Event|Payment|Transaction)
DEST_KEY = queue
FORMAT = indexQueue

[stuff_to_different_index]
REGEX= (Event|Payment|Transaction)
DEST_KEY = _MetaData:Index
FORMAT = Event_log

Re: Need Help in filtering data ad ingest few of the data

satyaallaparthi — Sat, 26 Oct 2019 03:13:57 GMT

Yes, but what about the 3rd one for routing data to another index ?

As I mentioned, I want to route data to new index. Which is events_logs. But that is not working..

That is the reason I posted for help.

Thanks,

Re: Need Help in filtering data ad ingest few of the data

woodcock — Sat, 26 Oct 2019 04:10:56 GMT

There is no description of such a requirement in the OP. Edit it and add those details.

Re: Need Help in filtering data ad ingest few of the data

gcusello — Sat, 26 Oct 2019 06:19:35 GMT

Hi satyaallaparthi,
If you have an Heavy Forwarder, you have to put conf files on Heavy Forwarder.

I hint to use only the first two commands to debug the situation, I know that you need also the third command, but usually the correct approach is to debug problem by problem.

Ciao.
Giuseppe

Re: Need Help in filtering data ad ingest few of the data

satyaallaparthi — Wed, 30 Sep 2020 02:45:18 GMT

Yes, I did edited now.

[Events]
REGEX= .+(Event|Payment|Transaction).+
DEST_KEY = _MetaData:Index
FORMAT = Event_log

Thanks,

Re: Need Help in filtering data ad ingest few of the data

satyaallaparthi — Sat, 26 Oct 2019 13:10:39 GMT

Sure,

I will do the the step by step process and will let u know.

Thanks,

Re: Need Help in filtering data ad ingest few of the data

woodcock — Sat, 26 Oct 2019 15:21:34 GMT

If you are sure that your settings are correct, then it must be something else. If you are doing a sourcetype override/overwrite, you must use the ORIGINAL value, NOT the new value. You must deploy your settings to the first full instance(s) of Splunk that handle the events (usually either the HF tier if you use one, or else your Indexer tier), restart all Splunk instances there, send in new events (old events will stay broken), then test using _index_earliest=-5m to be absolutely certain that you are only examining the newly indexed events.

Re: Need Help in filtering data ad ingest few of the data

woodcock — Sat, 26 Oct 2019 15:41:32 GMT

OK, I updated my original answer at the top of this thread. The main thing is that Queue must be queue. See my other answer if it still doesn't work.

Re: Need Help in filtering data ad ingest few of the data

satyaallaparthi — Wed, 30 Sep 2020 02:43:32 GMT

Hello,

I really don't know what is going on in my splunk.

I tried to route data from main index to iis_nonprod. That is not working.

I kept props and transforms in HF. Because the data is touting HF before going to IDX.

source: e:\IISLogs\W3SVC1\u_ex191029.log

props:
[source::e:\IISLogs\W3SVC1*.log]
TRANSFORMS-filter = routeData

tried both REGEX = . and REGEX= . and REGEX =. but nothing is working.

transforms:
[routeData]
REGEX = .
DEST_KEY = _MetaData:Index
FORMAT = iis_nonprod

but still the data is not routing.

By the way. the above filterings are not working. 😞

can you please help me with that?

Re: Need Help in filtering data ad ingest few of the data

satyaallaparthi — Wed, 30 Sep 2020 02:43:35 GMT

Hello,

I really don't know what is going on in my splunk.

I tried to route data from main index to iis_nonprod. That is not working.

I kept props and transforms in HF. Because the data is touting HF before going to IDX.

source: e:\IISLogs\W3SVC1\u_ex191029.log

props:
[source::e:\IISLogs\W3SVC1*.log]
TRANSFORMS-filter = routeData

tried both REGEX = . and REGEX= . and REGEX =. but nothing is working.

transforms:
[routeData]
REGEX = .
DEST_KEY = _MetaData:Index
FORMAT = iis_nonprod

but still the data is not routing.

By the way. the above filterings are not working. 😞

can any one please help me with that?