Getting Data In

Is it possible to ingest-time eval _indextime field?

oshirnin
Path Finder

Hello!

I have a distributed deployment of Splunk Enterprise. All my UFs send raw events to two HFs, these send cooked data to three-node IDX cluster. My search interface is three-node SH cluster. I plan to use a few ingest-time eval fields, first of all I tested how it works, placing:

props.conf to HFs:

props.conf
------------------------------------------------------------
[airwatch_iis_flogs]
TRANSFORMS = ingest-eval-rule-size_bytes, ingest-eval-rule-orig_host, ingest-eval-rule-orig_time

transforms.conf to HFs:

transforms.conf
------------------------------------------------------------
[ingest-eval-rule-size_bytes]
INGEST_EVAL = size_bytes=len(_raw)

[ingest-eval-rule-orig_host]
INGEST_EVAL = orig_host=upper(host)

[ingest-eval-rule-orig_time]
INGEST_EVAL = orig_time=_time

fields.conf to SHs:

fields.conf
------------------------------------------------------------
[size_bytes]
INDEXED = True

[orig_host]
INDEXED = True

[orig_time]
INDEXED = True

I put props.conf/transforms.conf to HFs (not to IDXs) as these servers process all the raw events and cook the data for indexers. This configuration works like a charm: querying

index=* sourcetype=airwatch_iis_flogs

I get the events having expected indexed fields size_bytes (calculated), orig_host (uppercase) and orig_host (match _time).

Now, field of interest to me is _indextime. I want to index the latency:

transforms.conf
 ------------------------------------------------------------
 [ingest-eval-rule-latency_sec]
 INGEST_EVAL = latency_sec=_indextime-_time

I also put the relevant changes to props.conf and fields.conf, but unfortunately this configuration doesn't work. Is it maybe because _indextime is empty while cooking events on HFs, and actually filled up while writing events to disk on indexers (not sure, where IndexQueue lives - on IDXs or HFs)?

What should I do to use this _indextime field in ingest-time eval - maybe put:

outputs.conf
------------------------------------------------------------
sendCookedData = false

to my HFs, move all props.conf/transforms.conf to IDXs? I feel myself, there are more drawbacks then benefts from this decision.

Are there any more limitations using _indextime ingest-time eval?

0 Karma

BDein
Engager

Hi Guys,

This is a very interesting topic, for which I've made an app, not only for HF's but also for the IDX's, however I do see some challenges here...
First of all I took a little different approach in props.conf, as I used the [default] stanza, with the aim to hit ALL events coming in.
It works - to a curtain extend, but it's really wired (to me) how, as most UF's sends their data through the HF's, and some directly to the IDX's, and for some strange reason, that I've spend quite some time investigating - without a solution (nor explanation) yet.

Some hosts "hits" the transforms (props) and gets the new values, and some don't, even though the all pass the same HF's.

Of those host sending directly to the IDX's none of them "hits" the transforms.

transforms.conf
---------------------------
[ingest-eval-rule-timeskew]
INGEST_EVAL = timeSkew=time() - _time
WRITE_META=true

[ingest-eval-rule-evtsize]
INGEST_EVAL = evtSizeB=len(_raw)
WRITE_META=true

[ingest-eval-rule-tierhost]
INGEST_EVAL = tierHost=replace(splunk_server,"\..*","")
WRITE_META = true

props.conf
---------------------------
[default]
TRANSFORMS = ingest-eval-rule-timeskew, ingest-eval-rule-evtsize, ingest-eval-rule-tierhost

fields.conf
---------------------------
[timeSkew]
INDEXED = True

[evtSizeB]
INDEXED = True

[tierHost]
INDEXED = True

Does anyone of you have an idea about:

  1. Why none of the data parsing direct to the IDX's don't "hit" this transform?
  2. Why some hosts (UF's) sending data through the HF's "hits" and some don't?
0 Karma

BDein
Engager

I actually just got this one into place by simply adding a wildcard to props.conf, so it now looks like this:

[default]
TRANSFORMS = ingest-eval-rule-timeskew, ingest-eval-rule-evtsize, ingest-eval-rule-tierhost

[(?::){0}*]
TRANSFORMS = ingest-eval-rule-timeskew, ingest-eval-rule-evtsize, ingest-eval-rule-tierhost

 

And now all - except for a few internal sourcetypes, it's great.

0 Karma

norbert_hamel
Communicator

Hi @oshirnin ,
the eval statement now() is not available for INGEST_EVAL. But you can use time(), which does pretty much the same and which can be used for INGEST_EVAL. So please try the following:

[ingest-eval-rule-latency_sec]
INGEST_EVAL = latency_sec= time() - _time

I had a similar requirement yesterday and tested that successfully. This will also give more accuracy than _indextime, since time() includes subseconds, which are not available in _indextime, potentially leading to negative latency values when _indextime and _time are in the same second.

Cheers

season88481
Contributor

Please ignore my question. Heavy Forwarder seems able to get time() as well. My Splunk version is 7.3.1

0 Karma

season88481
Contributor

Hi norbert_hamel

 

I try to get time() in my Heavy Forwarder, doesn't seem working.

Are you putting the ingest_eval at your heavy forwarder as well?

And what is your Splunk version

Many thanks.

Cheers,

S

0 Karma

richgalloway
SplunkTrust
SplunkTrust

I believe it's not possible to reference the _indextime field at index time since it is not defined until after all other props and transforms are processed. You'll need to calculate latency at search time.
I'm waiting to be corrected on this.

---
If this reply helps you, an upvote would be appreciated.

oshirnin
Path Finder

Hello, @richgalloway ! I feel the same. I think it would be enough to ingest-eval now() but strange - it doesn't work! Do you maybe see errors in the configs?

transforms.conf
------------------------------------------------------------
[ingest-eval-rule-time_now]
INGEST_EVAL = time_now = now()

props.conf
------------------------------------------------------------
[airwatch_iis_flogs]
TRANSFORMS = ingest-eval-rule-time_now

fields.conf
------------------------------------------------------------
[time_now]
INDEXED = True
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Why ingest now()? _indextime is already available and it pretty much the same thing.

---
If this reply helps you, an upvote would be appreciated.
0 Karma
Get Updates on the Splunk Community!

Improve Your Security Posture

Watch NowImprove Your Security PostureCustomers are at the center of everything we do at Splunk and security ...

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...