Creating a new sourcetype and performing transforms on it in the same TA

[ra__22]

If I have a transforms.conf like the below:

[ORIGIN2]
REGEX = (?:"id":"32605")
FORMAT = sourcetype::test-2
DEST_KEY = MetaData:Sourcetype

[aa]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[bb]
REGEX =(?=.*successfully)
DEST_KEY = queue
FORMAT = indexQueue

and I call the props like the following:

[test]
TRANSFORMS-rename_sourcetype = ORIGIN2
SHOULD_LINEMERGE = false
EVAL-ok = "ok"

[aslaof:test-2]
EVAL-action2 = "whatt"
TRANSFORMS-eliminate_unwanted_data = aa,bb
EVAL-action = "nooooo"

I cant seem to figure out why Im not allowed to perform a transform on my newly created sourcetype. Oddly, Splunk registers my 2 EVAL commands, but my transforms are not performed. Am I not allowed to perform transforms on a sourcetype I just created?

Also tried combining the initial transform that creates the sourcetype into one piece:

REGEX = (?=.*"id":"32605")(?=.*successfully), but this does not seem to work either. 

[livehybrid]

Hi @ra__22 

The TRANSFORMS is done as part of the typingQueue parsing process when data is ingested into Splunk. The data only goes through that once, subsequent changes to the sourcetype will not trigger re-evaluation of props.conf rules for the new sourcetype.

The EVAL statements are run at search-time, so these will apply to the new sourcetype when you search the data, which is why you are seeing the eval fields working.

To fix your issue with "eliminate_unwanted_data" not running, try moving this transform call to the original sourcetype name, perhaps run it before you change the sourcetype to remove ambiguity.

Please let me know how you get on and consider adding karma to this or any other answer if it has helped.
Regards

Will

[ra__22]

Thanks @livehybrid .

I've written some new props/transforms to try to get the same result, however now Im running into trouble again. 

So my real issue is that there are alot of logs coming in to ID: 32605 that do not have 'successfully' in them that I need to send to the null queue/get rid of. But I dont seem to be able to both get the logs I want to the new sourcetype and get rid of these unwanted ones. Seems no matter the order I put the transforms in below, it does not work.
I also tried creating a transform where I specifically target id = 32605 and the log not having the word 'successfully'. That doesnt seem to work either. 

Transforms:

[ORIGIN1]
REGEX = (?:\"id\":\"32605\".*successfully)
FORMAT = sourcetype::test-2
DEST_KEY = MetaData:Sourcetype

[ORIGIN2]
REGEX = (?:successfully.*\"id\":\"32605\")
FORMAT = sourcetype::test-2
DEST_KEY = MetaData:Sourcetype

[SAVE_OTHERS2]
REGEX =(?:"id":"(?!32605\b)\d+")
DEST_KEY = queue
FORMAT = indexQueue

[JUNK]
REGEX = (?:"id":"32605")
DEST_KEY = queue
FORMAT = nullQueue


Props:

[test]
TRANSFORMS-rename_sourcetype = ORIGIN1, ORIGIN2, JUNK, SAVE_OTHERS2
SHOULD_LINEMERGE = false

[PickleRick]

Wait a second.

1. Transforms are called in a specific order:

a) Transform classes are called in an alphabetical order

b) Transforms within a single class are called left to right.

2. All transforms are called (that's the important part!)

So if you want to keep just part of your data and filter the remaining events out you have to first redirect all events to nullQueue and then match the part of your events you want to keep and send them to indexQueue.

So you should to this like this - first transform should send all events matching your "ID":\s*"?32605 to nullQueue. Then you should have a transform sending the "successful" events to indexQueue.

BTW, you don't need to use non-capturing groups with your REGEX in a transform.

[ra__22]

Hey @PickleRick 

Thanks for the response. Thats how I understand it as well, but it doesnt seem to be working for me. Even if I re-arrange how Im calling the transforms, I am not able to stop id:32605 from appearing in my sourcetype.

The below configurations result in 
sourcetype = test : all ids are appearing

sourcetype = test-2: populated with (id = 32605 & the word successfully )

Even though Im certain I should be filtering out id = 32605. Ive tested the regex and its certainly matching my logs. Are there any tools to see more in-depthly how transforms are working/being applied on the backend?

My props now looks like this:

[ORIGIN1]
REGEX = \"id\":\"32605\".*successfully
FORMAT = sourcetype::test-2
DEST_KEY = MetaData:Sourcetype

[ORIGIN2]
REGEX = successfully.*\"id\":\"32605\"
FORMAT = sourcetype::test-2
DEST_KEY = MetaData:Sourcetype

[SAVE_OTHERS2]
REGEX ="id":"(?!32605\b)\d+"
DEST_KEY = queue
FORMAT = indexQueue

[null]
REGEX = \"id\":\"32605\"
FORMAT = queue
DEST_KEY = nullQueue

Transforms:
[test]
TRANSFORMS-rename_sourcetype = null, ORIGIN1, ORIGIN2, SAVE_OTHERS2
SHOULD_LINEMERGE = false

[PickleRick]

You seem to be overthinking it with your negative lookaheads.

props.conf:

[test]
TRANSFORMS-01-filterout-most = drop_all_events, retain_32605
TRANSFORMS-02-recast-successful = recast_sourcetype_successful

transforms.conf:

[drop_all_events]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[retain_32605]
REGEX = "id"\s*:\s*"32605"
DEST_KEY = queue
FORMAT = indexQueue

[recast_sourcetype_successful]
REGEX = successful
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::test-2
WRITE_META = true

(I'm never sure whether WRITE_META is needed or not).