topic Re: How to split data into separate sourcetypes with transforms in Getting Data In

How to split data into separate sourcetypes with transforms

the_rains — Fri, 23 Apr 2021 14:18:36 GMT

I am trying to split some data into difference source types using a lookup table.

I am testing this locally.

I have a source type called A and wish to extract fields to source type B.

A snippet of my data is below.

4/23/21 11:30:29.000 AM 23 Fri Apr 23 2021 11:30:29 www1 sshd[4878]: Failed password for invalid user SAMPLE123:ABC11:snmp from 10.0.0.1 port 3118 ssh host = 192.168.1.1 source = /A.log sourcetype = A

props.conf

[a] DATETIME_CONFIG = LINE_BREAKER = ([\r\n]+) NO_BINARY_CHECK = true category = Custom pulldown_type = 1 LOOKUP-alookup = lookuptable snmp_trap AS host OUTPUT host AS host_output TRANSFORMS-changesourcetype = B

Transforms.conf

[lookuptable] batch_index_query = 0 case_sensitive_match = 0 filename = lookuptable.csv max_matches = 1 min_matches = 1

host snmp_poll syslog snmp_trap 10.0.01 SAMPLE123:ABC11:ipfix SAMPLE123:ABC11:snmp_trap SAMPLE123:ABC11:syslog

I have achieved similar in the past using Regex to separate source type but having issues doing this via a lookup table

Any help appreciated.

Re: How to split data into separate sourcetypes with transforms

venkatasri — Mon, 26 Apr 2021 03:07:37 GMT

Hi @the_rains

Lookups are only used during search-time on search head component, what you are trying to achieve before indexing at HF layer. Try using regex and other method which you can think of.

-----------------------------------------------------

An upvote would be appreciated if it helps!

Re: How to split data into separate sourcetypes with transforms

the_rains — Mon, 26 Apr 2021 08:03:24 GMT

@venkatasri

Yes correct this is not an index time extraction .

This method works using regex, but unable to with a lookuptable.