topic Re: INGEST_EVAL in Splunk Enterprise

INGEST_EVAL

splunkpersona — Wed, 31 Jul 2024 17:08:36 GMT

I am working on ingesting the WSJT-X log. I got to where I have the basic fields in Splunk and wanted to create a date and time stamp from the poorly formatted data. I started with a very basic eval statement to test this and I am not seeing the new field. So, what did I miss?

I created the following:

transforms.conf [wsjtx_log] REGEX = (\d{2})(\d{2})(\d{2})_(\d{2})(\d{2})(\d{2})\s+(\d+\.\d+)\s+(\w+)\s+(\w+)\s+(\d+|-\d+)\s+(-\d+\.\d+|\d+\.\d+)\s+(\d+)\s+(.+) FORMAT = year::$1 month::$2 day::$3 hour::$4 min::$5 sec::$6 freqMhz::$7 action::$8 mode::$9 rxDB::$10 timeOffset::$11 freqOffSet::$12 remainder::$13 [add20] INGEST_EVAL = fyear="20" . $year$ props.conf [wsjtx_log] REPORT-wsjtx_all = wsjtx_log TRANSFORMS = add20 fields.conf fyear] INDEXED = TRUE

Re: INGEST_EVAL

PickleRick — Wed, 31 Jul 2024 04:46:57 GMT

The fields extracted with REPORT are eztracted in search time so they're not available inindex time for INGEST_EVAL.

Re: INGEST_EVAL

splunkpersona — Wed, 31 Jul 2024 17:09:15 GMT

That makes sense, the docs mentioned the order of operations but sometimes that doesnt sink in. It was easy enough to transition but I am still not seeing the field. I do see the fields as parsed from the props.conf-

Props:

[wsjtx_log] #REPORT-wsjtx_all = wsjtx_log EXTRACT-wsjtx = (?<year>\d{2})(?<month>\d{2})(?<day>\d{2})_(?<hour>\d{2})(?<min>\d{2})(?<sec>\d{2})\s+(?<freqMhz>\d+\.\d+)\s+(?<action>\w+)\s+(?<mode>\w+)\s+(?<rxDB>\d+|-\d+)\s+(?<timeOffset>-\d+\.\d+|\d+\.\d+)\s+(?<freqOffSet>\d+)\s+(?<remainder>.+) TRANSFORMS = add20 Transform: [wsjtx_log] #REGEX = (\d{2})(\d{2})(\d{2})_(\d{2})(\d{2})(\d{2})\s+(\d+\.\d+)\s+(\w+)\s+(\w+)\s+(\d+|-\d+)\s+(-\d+\.\d+|\d+\.\d+)\s+(\d+)\s+(.+) #FORMAT = year::$1 month::$2 day::$3 hour::$4 min::$5 sec::$6 freqMhz::$7 action::$8 mode::$9 rxDB::$10 timeOffset::$11 freqOffSet::$12 remainder::$13 [add20] INGEST_EVAL = fyear="20" . $year$ Fields: [fyear] INDEXED = TRUE

Re: INGEST_EVAL

PickleRick — Wed, 31 Jul 2024 16:49:16 GMT

No. It's not about the order of operations.

It's about search-time vs. index-time.

REPORT and EXTRACT are two operations that are done on the event in search time - when the event is being read from the index and processed before presenting to the user. INGEST_EVAL is an operation which is done in index-time - when the event is initially received from the source and before it's written to the index. You search-time operations are not performed in index-time (and vice-versa).

So regardless of whether you define your search-time operations inline or with transform (in other words - as REPORT or EXTRACT), they are not active in index-time. You can only operate on indexed fields with INGEST_EVAL. So if you want to extract a part of your event in order to use it in INGEST_EVAL, you have to first extract it with TRANSFORM as indexed field (if you don't need it stored later, you can afterwards rewrite it with another INGEST_EVAL to null()).

Re: INGEST_EVAL

splunkpersona — Wed, 31 Jul 2024 17:27:59 GMT

I removed the $ signs from the field (I copied from the web UI). I also used this as a guide but still no go.

https://docs.splunk.com/Documentation/Splunk/9.2.2/Data/IngestEval

Re: INGEST_EVAL

PickleRick — Wed, 31 Jul 2024 17:31:21 GMT

OK. From the start.

Your INGEST_EVAL looks like this:

INGEST_EVAL = fyear="20" . year

Right?

Where does the "year" field come from?

Re: INGEST_EVAL

splunkpersona — Wed, 31 Jul 2024 17:37:21 GMT

From the EXTRACT in props.conf.

EXTRACT-wsjtx = (?<year>\d{2})(?<month>\d{2})(?<day>\d{2})_(?<hour>\d{2})(?<min>\d{2})(?<sec>\d{2})\s+(?<freqMhz>\d+\.\d+)\s+(?<action>\w+)\s+(?<mode>\w+)\s+(?<rxDB>\d+|-\d+)\s+(?<timeOffset>-\d+\.\d+|\d+\.\d+)\s+(?<freqOffSet>\d+)\s+(?<remainder>.+)

Re: INGEST_EVAL

PickleRick — Wed, 31 Jul 2024 17:42:34 GMT

As I wrote before - EXTRACT and REPORT are run in search-time. TRANSFORM (including INGEST_EVAL) is run in index-time. You don't have search-time stuff in index-time. So you don't have your "year" field when you're trying to run INGEST_EVAL.

Re: INGEST_EVAL

splunkpersona — Wed, 31 Jul 2024 22:32:47 GMT

Ok, between your commentary and my re-write of the documentation I got this working. I will post my re write of the splunk instructions and the confs ASAP. Wish I could attach docs..