topic Re: Define identification field in Splunk Enterprise

Define identification field

leandromatperei — Fri, 19 Mar 2021 22:41:29 GMT

Guys,

I need support, I need to upload these files and process this data, but I need them to be indexed by the _time field, I was unable to use the standard type of pre-defined data. Can you help me?

Another detail, the Valor and Frete fields have commas, and refer to currency, there is a way to treat this data before indexing, so that this data is treated as "." for example ?

Example file: https://www.dropbox.com/s/exh7g1glumxcetr/log.txt?dl=0

Re: Define identification field

to4kawa — Fri, 19 Mar 2021 23:57:44 GMT

index=_internal | head 1 | fields _raw | eval _raw="IP=189.41.40.129,Produto=\"Test1\",Valor=179,00,Categoria=Banho,Campanha=1,Vendeu=1,MetododeCompra=1,Bandeira=1,Transportadora=2,Frete=18,57,\"_time\"2021/01/25 19:20:37.374\" IP=201.0.205.197,Produto=\"Test2\",Valor=123,98,Categoria=Jogos,Campanha=1,Vendeu=0,MetododeCompra=0,Bandeira=0,Transportadora=5,Frete=12,58,\"_time\"2021/01/25 19:20:38.977\" IP=187.125.147.178,Produto=\"Teste3\",Valor=139,90,Categoria=Cozinha,Campanha=1,Vendeu=1,MetododeCompra=1,Bandeira=1,Transportadora=3,Frete=14,27,\"_time\"2021/01/25 19:20:38.977\" IP=187.115.202.233,Produto=\"Test4\",Valor=139,90,Categoria=Cozinha,Campanha=1,Vendeu=1,MetododeCompra=1,Bandeira=1,Transportadora=2,Frete=14,51,\"_time\"2021/01/25 19:20:39.579\" IP=187.111.15.221,Produto=\"Test5\",Valor=164,00,Categoria=Banho,Campanha=2,Vendeu=1,MetododeCompra=1,Bandeira=1,Transportadora=1,Frete=16,81,\"_time\"2021/01/25 19:20:40.580\"" | multikv noheader=t | fields _raw | rex mode=sed "s/\"_time\"/Time=/ s/=(\d.*?)(?=,[A-Z])/=\"\1\"/g" | rex "(?<time>\d{4}/.*)" | eval _time=strptime(time,"%Y/%m/%d %T.%3N") | kv

Re: Define identification field

leandromatperei — Sat, 20 Mar 2021 00:08:27 GMT

Hi @to4kawa

This is in the query and how can I make this definition directly in sourcetype? I have a thousand line file that follows this same pattern.

Re: Define identification field

to4kawa — Sat, 20 Mar 2021 00:16:50 GMT

How about SEDCMD with props.conf.

Re: Define identification field

leandromatperei — Sat, 20 Mar 2021 00:44:51 GMT

@to4kawa

I defined it that way within the sourcetype and it almost went, I just couldn't get the same match for some fields: Category and Shipping

[ <SOURCETYPE NAME> ] SHOULD_LINEMERGE=true LINE_BREAKER=([\r\n]+) NO_BINARY_CHECK=true CHARSET=AUTO disabled=false TIME_FORMAT=%Y/%m/%d %H.%.M:%.S.%3N TIME_PREFIX=_time SEDCMD=s/=(\d.*?)(?=,[A-Z])/="\1"/g

How can I leave these fields the same way?

Re: Define identification field

to4kawa — Sat, 20 Mar 2021 09:37:47 GMT

If you have SECCMD do what I'm doing and search in smart mode, you're basically good to go.