splunk trial / HEC / props & transforms

[pck_npluyaud]

Hye.

I have 2 platforms : all-in-one for testing, enterprise for the production.

I submit a new HEC flow on the first, defined in /splunk_httpinput app/. The source of the flow is from a Kubernetes platform.

2 problems :

• get the time inside the flow to become the splunk' timestamp => don't work
• get the value of a field to push the flow on the good index => don't work

dataflow example

{"ENV":"rci_omega-backend","timestamp":"2020-07-09T17:13:40.998+02:00","level":"INFO","class":"pacifica.bo.common.logger.LoggerInterceptor","method":"afterCompletion","file":"LoggerInterceptor.java","line":59,"correlationId":"da4c4d57-f8db-464b-9765-fb115e16cd88","user":"toto","request_path":"/app-web/health-check","request_method":"GET","response_status":200,"request_elapsed_time":18,"msg":"HTTP_REQUESTS"}

[http://k8s_events]

inputs.conf

[http://k8s_events]

disabled = 0

index = k8s_events
token = 71a0604c-fce7-4831-abae-2bbc32ee0295
sourcetype = kube:events

props.conf

[kube:events]
INDEXED_EXTRACTIONS = json
pulldown_type = true
KV_MODE = none
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
AUTO_KV_JSON = false
TIME_PREFIX = /"timestamp":"/
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N+%2N:%2N
MAX_TIMESTAMP_LOOKAHEAD=29
TRANSFORMS-SetIndex = kube_index_dev,kube_index_rci,kube_index_rcu

transforms.conf

[kube_index_dev]
REGEX = "ENV":"dev_omega-backend"
DEST_KEY = _MetaData:Index
FORMAT = dev_omega

[kube_index_rci]
REGEX = "ENV":"rci_omega-backend"
DEST_KEY = _MetaData:Index
FORMAT = rci_omega

[kube_index_rcu]
REGEX = "ENV":"rcu_omega-backend"
DEST_KEY = _MetaData:Index
FORMAT = rcu_omega

Any help is welcome...

[richgalloway]

I can answer the timestamp problem, but not the other one.  The settings for TIME_PREFIX and TIME_FORMAT are incorrect.  Try these, instead.

TIME_PREFIX = "timestamp":"
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N+%:z

[pck_npluyaud]

Thanks, but I have always any milliseconds of delay..

timestamp: 2020-07-09T18:11:42.507+02:00 => 09/07/2020 18:11:42,508

timestamp: 2020-07-09T18:11:42.483+02:00 => 09/07/2020 18:11:42,484

timestamp: 2020-07-09T18:13:48.163+02:00 => 09/07/2020 18:13:48,166

[richgalloway]

What do you mean? The TIME_FORMAT setting reads the timestamp string as it is presented. It doesn't (and can't) account for delay. That's a separate topic. How are you detecting this delay?

[pck_npluyaud]

How ? In my 3 examples, I have until 3ms of difference between the timestamp in the _raw and the timestamp from the log captured by Splunk. This is why I say that the code didn't work.

(or I don't understand something ...)

timestamp _raw: 2020-07-09T18:11:42.507+02:00 => timestamp splunk 09/07/2020 18:11:42,508
timestamp _raw: 2020-07-09T18:11:42.483+02:00 => timestamp splunk 09/07/2020 18:11:42,484
timestamp _raw: 2020-07-09T18:13:48.163+02:00 => timestamp splunk 09/07/2020 18:13:48,166

I verify with this request : index=k8s* | rex field=_raw "\"timestamp\":\"(?<timestamp>[^\"]*)" | eval timestamp_s=strptime(timestamp, "%Y-%m-%dT%H:%M:%S.%3Q%:z") | eval time_s=strftime(_time,"%s.%3Q") | eval diff=time_s-timestamp_s | stats count by diff

At this time, on 30 events, 18 equal, 10 with 1ms difference, 1 with 2ms, 1 with 3ms.

[richgalloway]

I ran your query using the example event above and those three timestamps. Each showed diff=zero.

The only suggestion I have is to change TIME_FORMAT to use the string you have in the test query.

TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3Q%:z