Hi at all, I have a new doubt about the sequence of activities during indextime. I have a data flow, arriving from HEC on an HF that I need to elaborate it because these data arrive from a concentr...
See more...
Hi at all, I have a new doubt about the sequence of activities during indextime. I have a data flow, arriving from HEC on an HF that I need to elaborate it because these data arrive from a concentrator and are relative to many different data flows (linux, oracle, etc...), so I have to assign the correct sourcetype to these data and I have to elaborate logs because they are modified by securelog: the original logs are inserted in a field of json adding some metadata. I configured the following flow: in props.conf: [source::http:logstash*] TRANSFORMS-000 = global_set_metadata TRANSFORMS-001 = set_sourcetype_by_regex TRANSFORMS-001 = set_index_by_sourcetype in transforms.conf: [global_set_metadata] INGEST_EVAL = host := coalesce(json_extract(_raw, "host.name"), json_extract(_raw, "host.hostname")), relay_hostname := json_extract(_raw, "hub"), source := "http:logstash".coalesce("::".json_extract(_raw, "log.file.path"), "") [set_sourcetype_by_regex] INGEST_EVAL = sourcetype := case(searchmatch("/var/log/audit/audit.log"), "linux_audit", true(), "logstash") [set_index_by_sourcetype] INGEST_EVAL = index:=case(sourcetype=linux, "index_linux", sourcetype=logstash, "index_logstash") in which: the first transformation extract (using INGEST_EVAL) metadata as host, source and relay_hostname (the concentrator from which the logs arrive), the second one assign the correct sourcetype based on a regex. the third one assign the correct index based on sourcetype and usig INGEST_EVAL to avoid to re-run a regex, the first two transformations are correctly executed, but the third doesn't use the sourcetype assigned by the second one. I also tried a different approach using CLONE_SOURCETYPE in the second one (instead of INGEST_EVAL) and it runs, but I'm verifying if the above flow can run because it's more linear and should be less heavy for the system. Where could I search the issue? is there something wrong in the activity flow? Thank you to all. Ciao. Giuseppe