SOURCE_Key Extraction

[AliMaher]

I Have used the below two events to test the SOURCE_KEY =  

<132>1 2023-12-24T09:48:05+00:00 DCSECIDKOASV02 ikeyserver 8244 - [meta sequenceId="2850227"] {Warning}, {RADIUS}, {W-006001}, {An invalid RADIUS packet has been received.}, {0x0C744774DF59FC530462C92D2781B102}, {Source Location:10.240.86.6:1812 (Authentication)}, {Client Location:10.240.86.18:42923}, {Reason:The packet is smaller than minimum size allowed for RADIUS}, {Request ID:101}, {Input Details:0x64656661756C742073656E6420737472696E67}, {Request Type:Indeterminate}

<132>1 2023-12-24T09:48:05+00:00 DCSECIDKOASV02 ikeyserver 8244 - [meta sequenceId="2850228"] {Warning}, {RADIUS}, {W-006001}, {An invalid RADIUS packet has been received.}, {0xBA42228CB3604ECFDEEBC274D3312187}, {Source Location:10.240.86.6:1812 (Authentication)}, {Client Location:10.240.86.19:18721}, {Reason:The packet is smaller than minimum size allowed for RADIUS}, {Request ID:101}, {Input Details:0x64656661756C742073656E6420737472696E67}, {Request Type:Indeterminate}

Using the below Regex:

[xmlExtractionIDX]
REGEX = .*?"]\s+\{(?<Severity>\w+)\},\s+\{\w+\},\s+\{(?<DeviceID>[^}]*)\},(.*)
FORMAT = Severity::$1 DeviceID::$2 Last_Part::$3

WRITE_META = true

till that it's working fine then i want to add more precise extraction and want to extarct more info from the Last_Part field using the SOURCE_KEY = 

[xmlExtractionIDX]
REGEX = .*?"]\s+\{(?<Severity>\w+)\},\s+\{\w+\},\s+\{(?<DeviceID>[^}]*)\},(.*)
FORMAT = Severity::$1 DeviceID::$2 Last_Part::$3
SOURCE_KEY = MetaData:Last_Part
REGEX = Reason:(.*?)\}
FORMAT = Reason::$1
WRITE_META = true

But it doesn't work now, Is there any advice to do that using SOURCE_KEY 

[PickleRick]

Apart from the direct technical answer - you can't have two same settings (two FORMAT entries) in the same stanza. The latter overwrittes the former.

But there are more issues here - why are you trying to use index-time extractions in the first place?

[AliMaher]

I am trying to test the Index Time field extraction, 

and want to know how to refine the field extraction using source_key Keyword.

Then how can i refine my Field extraction if i cant use the SOURCE_KEY twice?

[PickleRick]

OK. If it's just for testing the functionality, I won't be bugging you about it too much 😉

Just remember that apart from very specific cases index-time extractions are best avoided.

But back to the point - if you want to extract a field from a previously extracted field, you need to have two separate transforms and make sure they are triggered in a proper order.

So you need to first define a transform which extracts a field (or set of fields) from raw data. And then define another transform which extracts your field from an already extracted field. As a bonus you might (if you don't need it indexed) add yet another transform to "delete" (by setting it to null() using INGEST_EVAL) the field extracted in the first step.

Example:

transforms.conf:

[test_extract_payload]
REGEX = payload:\s"([^"]+)"
FORMAT = payload::$1
WRITE_META = true

[test_extract_site]
REGEX = site:\s(\S)+
FORMAT = site::$1
WRITE_META = true
SOURCE_KEY = payload

props.conf:

[my_sourcetype]
TRANSFORMS-extract-site-from-payload = test_extract_payload, test_extract_site

 This way you'll get your site field extracted from an event containing

payload: "whatever whatever site: site1 whatever"

but not from just

"whatever whatever site: site1 whatever"

or

payload: "whatever whatever" site: site1