Hi at all,
I have a new doubt about the sequence of activities during indextime.
I have a data flow, arriving from HEC on an HF that I need to elaborate it because these data arrive from a concentrator and are relative to many different data flows (linux, oracle, etc...), so I have to assign the correct sourcetype to these data and I have to elaborate logs because they are modified by securelog: the original logs are inserted in a field of json adding some metadata.
I configured the following flow:
in props.conf:
[source::http:logstash*]
TRANSFORMS-000 = global_set_metadata
TRANSFORMS-001 = set_sourcetype_by_regex
TRANSFORMS-001 = set_index_by_sourcetype
in transforms.conf:
[global_set_metadata]
INGEST_EVAL = host := coalesce(json_extract(_raw, "host.name"), json_extract(_raw, "host.hostname")), relay_hostname := json_extract(_raw, "hub"), source := "http:logstash".coalesce("::".json_extract(_raw, "log.file.path"), "")
[set_sourcetype_by_regex]
INGEST_EVAL = sourcetype := case(searchmatch("/var/log/audit/audit.log"), "linux_audit", true(), "logstash")
[set_index_by_sourcetype]
INGEST_EVAL = index:=case(sourcetype=linux, "index_linux", sourcetype=logstash, "index_logstash")
in which:
the first transformation extract (using INGEST_EVAL) metadata as host, source and relay_hostname (the concentrator from which the logs arrive),
the second one assign the correct sourcetype based on a regex.
the third one assign the correct index based on sourcetype and usig INGEST_EVAL to avoid to re-run a regex,
the first two transformations are correctly executed, but the third doesn't use the sourcetype assigned by the second one.
I also tried a different approach using CLONE_SOURCETYPE in the second one (instead of INGEST_EVAL) and it runs, but I'm verifying if the above flow can run because it's more linear and should be less heavy for the system.
Where could I search the issue?
is there something wrong in the activity flow?
Thank you to all.
Ciao.
Giuseppe
