Sequence of activities at index time

gcusello — Thu, 04 Jul 2024 06:26:58 GMT

Hi at all,

I have a new doubt about the sequence of activities during indextime.
I have a data flow, arriving from HEC on an HF that I need to elaborate it because these data arrive from a concentrator and are relative to many different data flows (linux, oracle, etc...), so I have to assign the correct sourcetype to these data and I have to elaborate logs because they are modified by securelog: the original logs are inserted in a field of json adding some metadata.

I configured the following flow:

in props.conf:

[source::http:logstash*]
TRANSFORMS-000 = global_set_metadata
TRANSFORMS-001 = set_sourcetype_by_regex
TRANSFORMS-001 = set_index_by_sourcetype

in transforms.conf:

[global_set_metadata]
INGEST_EVAL = host := coalesce(json_extract(_raw, "host.name"), json_extract(_raw, "host.hostname")), relay_hostname := json_extract(_raw, "hub"), source := "http:logstash".coalesce("::".json_extract(_raw, "log.file.path"), "")

[set_sourcetype_by_regex]
INGEST_EVAL = sourcetype := case(searchmatch("/var/log/audit/audit.log"), "linux_audit", true(), "logstash")

[set_index_by_sourcetype]
INGEST_EVAL = index:=case(sourcetype=linux, "index_linux", sourcetype=logstash, "index_logstash")

in which:
the first transformation extract (using INGEST_EVAL) metadata as host, source and relay_hostname (the concentrator from which the logs arrive),
the second one assign the correct sourcetype based on a regex.
the third one assign the correct index based on sourcetype and usig INGEST_EVAL to avoid to re-run a regex,
the first two transformations are correctly executed, but the third doesn't use the sourcetype assigned by the second one.

I also tried a different approach using CLONE_SOURCETYPE in the second one (instead of INGEST_EVAL) and it runs, but I'm verifying if the above flow can run because it's more linear and should be less heavy for the system.

Where could I search the issue?
is there something wrong in the activity flow?

Thank you to all.
Ciao.
Giuseppe

Re: Sequence of activities at index time

isoutamo — Thu, 04 Jul 2024 08:39:13 GMT

You propably used raw endpoint on HEC?

Re: Sequence of activities at index time

gcusello — Fri, 05 Jul 2024 06:15:39 GMT

Hi @isoutamo ,

nice to hear you!

yes, I'm using HEC on premise, so I cannot use Edge.

Ciao.

Giuseppe

Re: Sequence of activities at index time

isoutamo — Fri, 05 Jul 2024 14:12:03 GMT

But are you using HEC's raw endpoint instead of event?

Also you have two same TRANSFORMS
TRANSFORMS-001 = set_sourcetype_by_regex
TRANSFORMS-001 = set_index_by_sourcetype

Which means that only one of those are used!

Re: Sequence of activities at index time

gcusello — Mon, 08 Jul 2024 06:03:06 GMT

Hi @isoutamo ,

thank you for your support.

it was a mistyping, the issue was that the searchmatch() function doesn't run in INGEST_EVAL, ising the match() function, my INGEST_EVAL is working.

Thank you again for your support.

Ciao.

Giuseppe

topic Re: Sequence of activities at index time in Getting Data In

Sequence of activities at index time

Re: Sequence of activities at index time

Re: Sequence of activities at index time

Re: Sequence of activities at index time

Re: Sequence of activities at index time