Getting Data In

Is it possible to ingest-time eval _indextime field?

Path Finder

Hello!

I have a distributed deployment of Splunk Enterprise. All my UFs send raw events to two HFs, these send cooked data to three-node IDX cluster. My search interface is three-node SH cluster. I plan to use a few ingest-time eval fields, first of all I tested how it works, placing:

props.conf to HFs:

props.conf
------------------------------------------------------------
[airwatch_iis_flogs]
TRANSFORMS = ingest-eval-rule-size_bytes, ingest-eval-rule-orig_host, ingest-eval-rule-orig_time

transforms.conf to HFs:

transforms.conf
------------------------------------------------------------
[ingest-eval-rule-size_bytes]
INGEST_EVAL = size_bytes=len(_raw)

[ingest-eval-rule-orig_host]
INGEST_EVAL = orig_host=upper(host)

[ingest-eval-rule-orig_time]
INGEST_EVAL = orig_time=_time

fields.conf to SHs:

fields.conf
------------------------------------------------------------
[size_bytes]
INDEXED = True

[orig_host]
INDEXED = True

[orig_time]
INDEXED = True

I put props.conf/transforms.conf to HFs (not to IDXs) as these servers process all the raw events and cook the data for indexers. This configuration works like a charm: querying

index=* sourcetype=airwatch_iis_flogs

I get the events having expected indexed fields sizebytes (calculated), orighost (uppercase) and orig_host (match _time).

Now, field of interest to me is _indextime. I want to index the latency:

transforms.conf
 ------------------------------------------------------------
 [ingest-eval-rule-latency_sec]
 INGEST_EVAL = latency_sec=_indextime-_time

I also put the relevant changes to props.conf and fields.conf, but unfortunately this configuration doesn't work. Is it maybe because _indextime is empty while cooking events on HFs, and actually filled up while writing events to disk on indexers (not sure, where IndexQueue lives - on IDXs or HFs)?

What should I do to use this _indextime field in ingest-time eval - maybe put:

outputs.conf
------------------------------------------------------------
sendCookedData = false

to my HFs, move all props.conf/transforms.conf to IDXs? I feel myself, there are more drawbacks then benefts from this decision.

Are there any more limitations using _indextime ingest-time eval?

0 Karma

Communicator

Hi @oshirnin ,
the eval statement now() is not available for INGESTEVAL. But you can use time(), which does pretty much the same and which can be used for INGESTEVAL. So please try the following:

[ingest-eval-rule-latency_sec]
INGEST_EVAL = latency_sec= time() - _time

I had a similar requirement yesterday and tested that successfully. This will also give more accuracy than _indextime, since time() includes subseconds, which are not available in _indextime, potentially leading to negative latency values when _indextime and _time are in the same second.

Cheers

Communicator

Please ignore my question. Heavy Forwarder seems able to get time() as well. My Splunk version is 7.3.1

0 Karma

Communicator

Hi norbert_hamel

 

I try to get time() in my Heavy Forwarder, doesn't seem working.

Are you putting the ingest_eval at your heavy forwarder as well?

And what is your Splunk version

Many thanks.

Cheers,

S

0 Karma

SplunkTrust
SplunkTrust

I believe it's not possible to reference the _indextime field at index time since it is not defined until after all other props and transforms are processed. You'll need to calculate latency at search time.
I'm waiting to be corrected on this.

---
If this reply helps you, an upvote would be appreciated.

Path Finder

Hello, @richgalloway ! I feel the same. I think it would be enough to ingest-eval now() but strange - it doesn't work! Do you maybe see errors in the configs?

transforms.conf
------------------------------------------------------------
[ingest-eval-rule-time_now]
INGEST_EVAL = time_now = now()

props.conf
------------------------------------------------------------
[airwatch_iis_flogs]
TRANSFORMS = ingest-eval-rule-time_now

fields.conf
------------------------------------------------------------
[time_now]
INDEXED = True
0 Karma

SplunkTrust
SplunkTrust

Why ingest now()? _indextime is already available and it pretty much the same thing.

---
If this reply helps you, an upvote would be appreciated.
0 Karma