Is it possible to ingest-time eval _indextime field?

[oshirnin]

Hello!

I have a distributed deployment of Splunk Enterprise. All my UFs send raw events to two HFs, these send cooked data to three-node IDX cluster. My search interface is three-node SH cluster. I plan to use a few ingest-time eval fields, first of all I tested how it works, placing:

props.conf to HFs:

props.conf
------------------------------------------------------------
[airwatch_iis_flogs]
TRANSFORMS = ingest-eval-rule-size_bytes, ingest-eval-rule-orig_host, ingest-eval-rule-orig_time

transforms.conf to HFs:

transforms.conf
------------------------------------------------------------
[ingest-eval-rule-size_bytes]
INGEST_EVAL = size_bytes=len(_raw)

[ingest-eval-rule-orig_host]
INGEST_EVAL = orig_host=upper(host)

[ingest-eval-rule-orig_time]
INGEST_EVAL = orig_time=_time

fields.conf to SHs:

fields.conf
------------------------------------------------------------
[size_bytes]
INDEXED = True

[orig_host]
INDEXED = True

[orig_time]
INDEXED = True

I put props.conf/transforms.conf to HFs (not to IDXs) as these servers process all the raw events and cook the data for indexers. This configuration works like a charm: querying

index=* sourcetype=airwatch_iis_flogs

I get the events having expected indexed fields size_bytes (calculated), orig_host (uppercase) and orig_host (match _time).

Now, field of interest to me is _indextime. I want to index the latency:

transforms.conf
 ------------------------------------------------------------
 [ingest-eval-rule-latency_sec]
 INGEST_EVAL = latency_sec=_indextime-_time

I also put the relevant changes to props.conf and fields.conf, but unfortunately this configuration doesn't work. Is it maybe because _indextime is empty while cooking events on HFs, and actually filled up while writing events to disk on indexers (not sure, where IndexQueue lives - on IDXs or HFs)?

What should I do to use this _indextime field in ingest-time eval - maybe put:

outputs.conf
------------------------------------------------------------
sendCookedData = false

to my HFs, move all props.conf/transforms.conf to IDXs? I feel myself, there are more drawbacks then benefts from this decision.

Are there any more limitations using _indextime ingest-time eval?

[BDein]

Hi Guys,

This is a very interesting topic, for which I've made an app, not only for HF's but also for the IDX's, however I do see some challenges here...
First of all I took a little different approach in props.conf, as I used the [default] stanza, with the aim to hit ALL events coming in.
It works - to a curtain extend, but it's really wired (to me) how, as most UF's sends their data through the HF's, and some directly to the IDX's, and for some strange reason, that I've spend quite some time investigating - without a solution (nor explanation) yet.

Some hosts "hits" the transforms (props) and gets the new values, and some don't, even though the all pass the same HF's.

Of those host sending directly to the IDX's none of them "hits" the transforms.

transforms.conf
---------------------------
[ingest-eval-rule-timeskew]
INGEST_EVAL = timeSkew=time() - _time
WRITE_META=true

[ingest-eval-rule-evtsize]
INGEST_EVAL = evtSizeB=len(_raw)
WRITE_META=true

[ingest-eval-rule-tierhost]
INGEST_EVAL = tierHost=replace(splunk_server,"\..*","")
WRITE_META = true

props.conf
---------------------------
[default]
TRANSFORMS = ingest-eval-rule-timeskew, ingest-eval-rule-evtsize, ingest-eval-rule-tierhost

fields.conf
---------------------------
[timeSkew]
INDEXED = True

[evtSizeB]
INDEXED = True

[tierHost]
INDEXED = True

Does anyone of you have an idea about:

1. Why none of the data parsing direct to the IDX's don't "hit" this transform?
2. Why some hosts (UF's) sending data through the HF's "hits" and some don't?

[BDein]

I actually just got this one into place by simply adding a wildcard to props.conf, so it now looks like this:

[default]
TRANSFORMS = ingest-eval-rule-timeskew, ingest-eval-rule-evtsize, ingest-eval-rule-tierhost

[(?::){0}*]
TRANSFORMS = ingest-eval-rule-timeskew, ingest-eval-rule-evtsize, ingest-eval-rule-tierhost

And now all - except for a few internal sourcetypes, it's great.

[norbert_hamel]

Hi @oshirnin ,
the eval statement now() is not available for INGEST_EVAL. But you can use time(), which does pretty much the same and which can be used for INGEST_EVAL. So please try the following:

[ingest-eval-rule-latency_sec]
INGEST_EVAL = latency_sec= time() - _time

I had a similar requirement yesterday and tested that successfully. This will also give more accuracy than _indextime, since time() includes subseconds, which are not available in _indextime, potentially leading to negative latency values when _indextime and _time are in the same second.

Cheers

[season88481]

Please ignore my question. Heavy Forwarder seems able to get time() as well. My Splunk version is 7.3.1

[season88481]

Hi norbert_hamel

I try to get time() in my Heavy Forwarder, doesn't seem working.

Are you putting the ingest_eval at your heavy forwarder as well?

And what is your Splunk version

Many thanks.

Cheers,

S

[richgalloway]

I believe it's not possible to reference the _indextime field at index time since it is not defined until after all other props and transforms are processed. You'll need to calculate latency at search time.
I'm waiting to be corrected on this.

[oshirnin]

Hello, @richgalloway ! I feel the same. I think it would be enough to ingest-eval now() but strange - it doesn't work! Do you maybe see errors in the configs?

transforms.conf
------------------------------------------------------------
[ingest-eval-rule-time_now]
INGEST_EVAL = time_now = now()

props.conf
------------------------------------------------------------
[airwatch_iis_flogs]
TRANSFORMS = ingest-eval-rule-time_now

fields.conf
------------------------------------------------------------
[time_now]
INDEXED = True

[richgalloway]

Why ingest now()? _indextime is already available and it pretty much the same thing.