Getting Data In

Is it possible to ingest-time eval _indextime field?

oshirnin
Path Finder

Hello!

I have a distributed deployment of Splunk Enterprise. All my UFs send raw events to two HFs, these send cooked data to three-node IDX cluster. My search interface is three-node SH cluster. I plan to use a few ingest-time eval fields, first of all I tested how it works, placing:

props.conf to HFs:

props.conf
------------------------------------------------------------
[airwatch_iis_flogs]
TRANSFORMS = ingest-eval-rule-size_bytes, ingest-eval-rule-orig_host, ingest-eval-rule-orig_time

transforms.conf to HFs:

transforms.conf
------------------------------------------------------------
[ingest-eval-rule-size_bytes]
INGEST_EVAL = size_bytes=len(_raw)

[ingest-eval-rule-orig_host]
INGEST_EVAL = orig_host=upper(host)

[ingest-eval-rule-orig_time]
INGEST_EVAL = orig_time=_time

fields.conf to SHs:

fields.conf
------------------------------------------------------------
[size_bytes]
INDEXED = True

[orig_host]
INDEXED = True

[orig_time]
INDEXED = True

I put props.conf/transforms.conf to HFs (not to IDXs) as these servers process all the raw events and cook the data for indexers. This configuration works like a charm: querying

index=* sourcetype=airwatch_iis_flogs

I get the events having expected indexed fields size_bytes (calculated), orig_host (uppercase) and orig_host (match _time).

Now, field of interest to me is _indextime. I want to index the latency:

transforms.conf
 ------------------------------------------------------------
 [ingest-eval-rule-latency_sec]
 INGEST_EVAL = latency_sec=_indextime-_time

I also put the relevant changes to props.conf and fields.conf, but unfortunately this configuration doesn't work. Is it maybe because _indextime is empty while cooking events on HFs, and actually filled up while writing events to disk on indexers (not sure, where IndexQueue lives - on IDXs or HFs)?

What should I do to use this _indextime field in ingest-time eval - maybe put:

outputs.conf
------------------------------------------------------------
sendCookedData = false

to my HFs, move all props.conf/transforms.conf to IDXs? I feel myself, there are more drawbacks then benefts from this decision.

Are there any more limitations using _indextime ingest-time eval?

0 Karma

BDein
Explorer

Hi Guys,

This is a very interesting topic, for which I've made an app, not only for HF's but also for the IDX's, however I do see some challenges here...
First of all I took a little different approach in props.conf, as I used the [default] stanza, with the aim to hit ALL events coming in.
It works - to a curtain extend, but it's really wired (to me) how, as most UF's sends their data through the HF's, and some directly to the IDX's, and for some strange reason, that I've spend quite some time investigating - without a solution (nor explanation) yet.

Some hosts "hits" the transforms (props) and gets the new values, and some don't, even though the all pass the same HF's.

Of those host sending directly to the IDX's none of them "hits" the transforms.

transforms.conf
---------------------------
[ingest-eval-rule-timeskew]
INGEST_EVAL = timeSkew=time() - _time
WRITE_META=true

[ingest-eval-rule-evtsize]
INGEST_EVAL = evtSizeB=len(_raw)
WRITE_META=true

[ingest-eval-rule-tierhost]
INGEST_EVAL = tierHost=replace(splunk_server,"\..*","")
WRITE_META = true

props.conf
---------------------------
[default]
TRANSFORMS = ingest-eval-rule-timeskew, ingest-eval-rule-evtsize, ingest-eval-rule-tierhost

fields.conf
---------------------------
[timeSkew]
INDEXED = True

[evtSizeB]
INDEXED = True

[tierHost]
INDEXED = True

Does anyone of you have an idea about:

  1. Why none of the data parsing direct to the IDX's don't "hit" this transform?
  2. Why some hosts (UF's) sending data through the HF's "hits" and some don't?
0 Karma

BDein
Explorer

I actually just got this one into place by simply adding a wildcard to props.conf, so it now looks like this:

[default]
TRANSFORMS = ingest-eval-rule-timeskew, ingest-eval-rule-evtsize, ingest-eval-rule-tierhost

[(?::){0}*]
TRANSFORMS = ingest-eval-rule-timeskew, ingest-eval-rule-evtsize, ingest-eval-rule-tierhost

 

And now all - except for a few internal sourcetypes, it's great.

0 Karma

norbert_hamel
Communicator

Hi @oshirnin ,
the eval statement now() is not available for INGEST_EVAL. But you can use time(), which does pretty much the same and which can be used for INGEST_EVAL. So please try the following:

[ingest-eval-rule-latency_sec]
INGEST_EVAL = latency_sec= time() - _time

I had a similar requirement yesterday and tested that successfully. This will also give more accuracy than _indextime, since time() includes subseconds, which are not available in _indextime, potentially leading to negative latency values when _indextime and _time are in the same second.

Cheers

season88481
Contributor

Please ignore my question. Heavy Forwarder seems able to get time() as well. My Splunk version is 7.3.1

0 Karma

season88481
Contributor

Hi norbert_hamel

 

I try to get time() in my Heavy Forwarder, doesn't seem working.

Are you putting the ingest_eval at your heavy forwarder as well?

And what is your Splunk version

Many thanks.

Cheers,

S

0 Karma

richgalloway
SplunkTrust
SplunkTrust

I believe it's not possible to reference the _indextime field at index time since it is not defined until after all other props and transforms are processed. You'll need to calculate latency at search time.
I'm waiting to be corrected on this.

---
If this reply helps you, Karma would be appreciated.

oshirnin
Path Finder

Hello, @richgalloway ! I feel the same. I think it would be enough to ingest-eval now() but strange - it doesn't work! Do you maybe see errors in the configs?

transforms.conf
------------------------------------------------------------
[ingest-eval-rule-time_now]
INGEST_EVAL = time_now = now()

props.conf
------------------------------------------------------------
[airwatch_iis_flogs]
TRANSFORMS = ingest-eval-rule-time_now

fields.conf
------------------------------------------------------------
[time_now]
INDEXED = True
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Why ingest now()? _indextime is already available and it pretty much the same thing.

---
If this reply helps you, Karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...