Hello,
I have tons of data that are ingesting to some index="abc".
But I want to filter the whole data and want to ingest the log with the words "Events,Transaction,Payment" and then want to route that data to index=event_logs
I wrote the below props and transforms. But no luck.
Props:
TRANSFORMS-filter = null, IQ,Events
Transforms:
[null]
REGEX= .
DEST_KEY = Queue
FORMAT = nullQueue
[IQ]
REGEX= .+(Event|Payment|Transaction).+
DEST_KEY = Queue
FORMAT = indexQueue
[Events]
REGEX= .+(Event|Payment|Transaction).+
DEST_KEY = _MetaData:Index
FORMAT = Event_log
Please do help me with the issue.
Thanks in Advance
Hello,
I really don't know what is going on in my splunk.
I tried to route data from main index to iis_nonprod. That is not working.
I kept props and transforms in HF. Because the data is touting HF before going to IDX.
source: e:\IISLogs\W3SVC1\u_ex191029.log
props:
[source::e:\IISLogs\W3SVC1*.log]
TRANSFORMS-filter = routeData
tried both REGEX = . and REGEX= . and REGEX =. but nothing is working.
transforms:
[routeData]
REGEX = .
DEST_KEY = _MetaData:Index
FORMAT = iis_nonprod
but still the data is not routing.
By the way. the above filterings are not working. 😞
can any one please help me with that?
If you are sure that your settings are correct, then it must be something else. If you are doing a sourcetype override
/overwrite, you must use the ORIGINAL value, NOT the new value. You must deploy your settings to the first full instance(s)
of Splunk that handle the events (usually either the HF tier
if you use one, or else your Indexer tier
), restart all Splunk instances there, send in new events (old events will stay broken), then test using _index_earliest=-5m
to be absolutely certain that you are only examining the newly indexed events.
Hello,
I really don't know what is going on in my splunk.
I tried to route data from main index to iis_nonprod. That is not working.
I kept props and transforms in HF. Because the data is touting HF before going to IDX.
source: e:\IISLogs\W3SVC1\u_ex191029.log
props:
[source::e:\IISLogs\W3SVC1*.log]
TRANSFORMS-filter = routeData
tried both REGEX = . and REGEX= . and REGEX =. but nothing is working.
transforms:
[routeData]
REGEX = .
DEST_KEY = _MetaData:Index
FORMAT = iis_nonprod
but still the data is not routing.
By the way. the above filterings are not working. 😞
can you please help me with that?
Casing matters; use this EXACTLY:
TRANSFORMS-filter = null_all_my_stuff, unnull_IQ_stuff, stuff_to_different_index
[null_all_my_stuff]
REGEX= .
DEST_KEY = queue
FORMAT = nullQueue
[unnull_IQ_stuff]
REGEX= (Event|Payment|Transaction)
DEST_KEY = queue
FORMAT = indexQueue
[stuff_to_different_index]
REGEX= (Event|Payment|Transaction)
DEST_KEY = _MetaData:Index
FORMAT = Event_log
Yes, but what about the 3rd one for routing data to another index ?
As I mentioned, I want to route data to new index. Which is events_logs. But that is not working..
That is the reason I posted for help.
Thanks,
There is no description of such a requirement in the OP. Edit it and add those details.
Yes, I did edited now.
[Events]
REGEX= .+(Event|Payment|Transaction).+
DEST_KEY = _MetaData:Index
FORMAT = Event_log
Thanks,
OK, I updated my original answer at the top of this thread. The main thing is that Queue
must be queue
. See my other answer if it still doesn't work.
Hi satyaallaparthi,
Three notes:
at first, where do you have these conf files?
You have to put them on Indexers or (when present) on Heavy Forwarders, not on Universal Forwarder.
For more information see at https://docs.splunk.com/Documentation/Splunk/7.3.2/Forwarding/Routeandfilterdatad .
Then the regex should be a little different:
REGEX=Event|Payment|Transaction
.
At least, why do you need to override index? is it not possible to set it on inputs.conf?
Ciao.
Giuseppe
Hello,
Yes, I did tried that before doing this.. but no luck.. that is why I changed the regex..
No, that is not possible to set in Inputs. Because I am getting the data from other instance.
Thanks,
Hi satyaallaparthi,
let me understand:
Could you try without the third command in props.conf?
TRANSFORMS-filter = null, IQ
Ciao.
Giuseppe
But I want the third one to route data to new index.
And yes, I have heavy forwarder is between indexer and UF. I restarted the server after I placed. But no luck
Thanks,
Hi satyaallaparthi,
If you have an Heavy Forwarder, you have to put conf files on Heavy Forwarder.
I hint to use only the first two commands to debug the situation, I know that you need also the third command, but usually the correct approach is to debug problem by problem.
Ciao.
Giuseppe
Sure,
I will do the the step by step process and will let u know.
Thanks,