Getting Data In

Need Help in filtering data ad ingest few of the data

satyaallaparthi
Communicator

Hello,

I have tons of data that are ingesting to some index="abc".

But I want to filter the whole data and want to ingest the log with the words "Events,Transaction,Payment" and then want to route that data to index=event_logs

I wrote the below props and transforms. But no luck.

Props:

TRANSFORMS-filter = null, IQ,Events

Transforms:

[null]
REGEX= .
DEST_KEY = Queue
FORMAT = nullQueue

[IQ]
REGEX= .+(Event|Payment|Transaction).+
DEST_KEY = Queue
FORMAT = indexQueue

[Events]
REGEX= .+(Event|Payment|Transaction).+
DEST_KEY = _MetaData:Index
FORMAT = Event_log

Please do help me with the issue.

Thanks in Advance

Tags (2)
0 Karma

satyaallaparthi
Communicator

Hello,

I really don't know what is going on in my splunk.

I tried to route data from main index to iis_nonprod. That is not working.

I kept props and transforms in HF. Because the data is touting HF before going to IDX.

source: e:\IISLogs\W3SVC1\u_ex191029.log

props:
[source::e:\IISLogs\W3SVC1*.log]
TRANSFORMS-filter = routeData

tried both REGEX = . and REGEX= . and REGEX =. but nothing is working.

transforms:
[routeData]
REGEX = .
DEST_KEY = _MetaData:Index
FORMAT = iis_nonprod

but still the data is not routing.

By the way. the above filterings are not working. 😞

can any one please help me with that?

0 Karma

woodcock
Esteemed Legend

If you are sure that your settings are correct, then it must be something else. If you are doing a sourcetype override/overwrite, you must use the ORIGINAL value, NOT the new value. You must deploy your settings to the first full instance(s) of Splunk that handle the events (usually either the HF tier if you use one, or else your Indexer tier), restart all Splunk instances there, send in new events (old events will stay broken), then test using _index_earliest=-5m to be absolutely certain that you are only examining the newly indexed events.

0 Karma

satyaallaparthi
Communicator

Hello,

I really don't know what is going on in my splunk.

I tried to route data from main index to iis_nonprod. That is not working.

I kept props and transforms in HF. Because the data is touting HF before going to IDX.

source: e:\IISLogs\W3SVC1\u_ex191029.log

props:
[source::e:\IISLogs\W3SVC1*.log]
TRANSFORMS-filter = routeData

tried both REGEX = . and REGEX= . and REGEX =. but nothing is working.

transforms:
[routeData]
REGEX = .
DEST_KEY = _MetaData:Index
FORMAT = iis_nonprod

but still the data is not routing.

By the way. the above filterings are not working. 😞

can you please help me with that?

0 Karma

woodcock
Esteemed Legend

Casing matters; use this EXACTLY:

Props:

TRANSFORMS-filter = null_all_my_stuff, unnull_IQ_stuff, stuff_to_different_index

Transforms:

[null_all_my_stuff]
REGEX= .
DEST_KEY = queue
FORMAT = nullQueue

[unnull_IQ_stuff]
REGEX= (Event|Payment|Transaction)
DEST_KEY = queue
FORMAT = indexQueue

[stuff_to_different_index]
REGEX= (Event|Payment|Transaction)
DEST_KEY = _MetaData:Index
FORMAT = Event_log
0 Karma

satyaallaparthi
Communicator

Yes, but what about the 3rd one for routing data to another index ?

As I mentioned, I want to route data to new index. Which is events_logs. But that is not working..

That is the reason I posted for help.

Thanks,

0 Karma

woodcock
Esteemed Legend

There is no description of such a requirement in the OP. Edit it and add those details.

0 Karma

satyaallaparthi
Communicator

Yes, I did edited now.

[Events]
REGEX= .+(Event|Payment|Transaction).+
DEST_KEY = _MetaData:Index
FORMAT = Event_log

Thanks,

0 Karma

woodcock
Esteemed Legend

OK, I updated my original answer at the top of this thread. The main thing is that Queue must be queue. See my other answer if it still doesn't work.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi satyaallaparthi,
Three notes:
at first, where do you have these conf files?
You have to put them on Indexers or (when present) on Heavy Forwarders, not on Universal Forwarder.
For more information see at https://docs.splunk.com/Documentation/Splunk/7.3.2/Forwarding/Routeandfilterdatad .

Then the regex should be a little different:

REGEX=Event|Payment|Transaction

.
At least, why do you need to override index? is it not possible to set it on inputs.conf?

Ciao.
Giuseppe

0 Karma

satyaallaparthi
Communicator

Hello,

Yes, I did tried that before doing this.. but no luck.. that is why I changed the regex..

No, that is not possible to set in Inputs. Because I am getting the data from other instance.

Thanks,

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi satyaallaparthi,
let me understand:

  • you have these conf files on Indexer, not on Universal Forwarders and you don't have intermediate Heavy Forwarders;
  • you tried my rex and you rex both without success;
  • you restarted Splunk after conf files updates.

Could you try without the third command in props.conf?

TRANSFORMS-filter = null, IQ

Ciao.
Giuseppe

0 Karma

satyaallaparthi
Communicator

But I want the third one to route data to new index.

And yes, I have heavy forwarder is between indexer and UF. I restarted the server after I placed. But no luck

Thanks,

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi satyaallaparthi,
If you have an Heavy Forwarder, you have to put conf files on Heavy Forwarder.

I hint to use only the first two commands to debug the situation, I know that you need also the third command, but usually the correct approach is to debug problem by problem.

Ciao.
Giuseppe

0 Karma

satyaallaparthi
Communicator

Sure,

I will do the the step by step process and will let u know.

Thanks,

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...