transforms.conf conditional statement for INGEST_EVAL

[whunterj]

I am attempting to setup an INGEST_EVAL for the _time field. My goal is to check if the _time field is in the future and prevent any future timestamps from being indexed. The INGEST_EVAL is configured correctly in the props.conf, fields.conf and transforms.conf, but is failing when I attempt to use a conditional statement. My goal is to do something like this in my transforms.conf:

[ingest_time_timestamp]
INGEST_EVAL = ingest_time_stamp:=if(_time > time(), time(), _time)

If _time is in the future, then I want it set to the current time, otherwise I want to leave it alone.

Anyone have any ideas?

[whunterj · Accepted solution]

Update:

I found a solution to this. The : before the = appears to have been problematic. Any timestamps that come in greater than the current time are now set to the system time, effectively preventing Splunk from indexing future timestamps. I also added an additional index time eval statement to set a flag variable called timestamp_status to "CRITICAL" if future timestamps are found and reset to the current time. The idea behind that was to allow me to search for any "CRITICAL" values for the timestamp_status field and see where I need to make adjustments to timestamp parsing for data sources that may start to send future timestamps. For anyone interested, I will paste the configs below.

props.conf:

[default]
TRANSFORMS-check_for_future_timestamp = check_for_future_timestamp

fields.conf:

[check_for_future_timestamp]
INDEXED = True

transforms.conf:

[check_for_future_timestamp]
INGEST_EVAL = timestamp_status=if(_time > time(), "CRITICAL", "OK"), _time=if(_time > time(), time(), _time)

[PickleRick]

This is something you can relatively easily find after indexing by comparing values of _time and _indextime so creating an indexed field just to check if the timestamp is correct seems kinda like an overkill. And manipulating _time (apart from possibly some format conversions which can't be resolved on simple props.conf parameters) is - as a rule of thumb - a very bad idea.

Also look at the MAX_DAYS_HENCE parameter in props.conf.