topic Re: index time fields extraction from source?? in Knowledge Management

index time fields extraction from source??

rajasekhar14 — Wed, 30 Sep 2020 00:36:02 GMT

Hi All,

I'm trying to do index field extractions from source files, here is the my settings
file names are like:

/tmp/test-raj/abc/bcd.log
/tmp/test-raj/xyz/cccc.log

i want to extract the 3rd directory as a fields called raj
raj=abc
raj=xyz
+
+ etc

Transforms.conf
i placed transforms.conf file in UF and HF and indexer
[netscreen-error]
SOURCE_KEY = MetaData:Source

REGEX = \/tmp\/test-raj\/(?\w+[^\/]+)\/\S+ in source
FORMAT = raj::"$1"
WRITE_META = true

Props.conf
i placed props.conf file in UF and HF and indexer
[test-123]
TRANSFORMS-netscreen = netscreen-error

fileds.conf
[raj]
INDEXED = true

let me know your thoughts?

Re: index time fields extraction from source??

koshyk — Sun, 19 May 2019 08:19:31 GMT

Your regex seems incorrect

Please try in transforms.conf

[netscreen-error]
SOURCE_KEY = MetaData:Source 
REGEX = \/tmp\/test-raj\/([^\/]+)\/.+
FORMAT = raj::$1
WRITE_META = true

Regex101 => https://regex101.com/r/OK0pNj/1

Re: index time fields extraction from source??

DavidHourani — Sun, 19 May 2019 12:20:42 GMT

Hi @rajasekhar14,

In addition to fixing your regex and format as @koshyk mentioned it :

 REGEX = \/tmp\/test-raj\/([^\/]+)\/.+
 FORMAT = raj::$1

Make sure you add the fields.conf file to the indexer as even if your indexed field is written in the metadata the indexer will not use it unless defined in the fields.conf file.

Cheers,
David

Re: index time fields extraction from source??

rajasekhar14 — Wed, 30 Sep 2020 00:36:18 GMT

@koshyk thanks for the answer, i changed my Regex but its not working.
now all 3 files are in the only indexers
[splunk@**** local]$ cat transforms.conf
[netscreen-error]
SOURCE_KEY = MetaData:Source
REGEX = \/tmp\/test-raj\/([^\/]+)\/.+
FORMAT = raj::$1
WRITE_META = true

[splunk@**** local]$ cat props.conf
[test-123]
TRANSFORMS-netscreen = netscreen-error

[splunk@*** local]$ cat ../../spl_fields/local/fields.conf
[raj]
INDEXED = true

do i need to change these .conf to HF or UF??

Re: index time fields extraction from source??

rajasekhar14 — Sun, 19 May 2019 15:43:33 GMT

@DavidHourani i placed the fileds.conf file and other files in the indexers only, but look likes it nor working. any thoughts??

Re: index time fields extraction from source??

koshyk — Sun, 19 May 2019 20:08:52 GMT

if you have HF, you need to send to HF & Indexers

Need to restart HF & indexers too if possible

Re: index time fields extraction from source??

DavidHourani — Mon, 20 May 2019 06:30:39 GMT

yeah it depends on where your data is coming from. If its going through a Heavy Forwarder then you need props.conf and transforms.conf on the HF and fields.conf on the indexer. If there is no HF then putting all on the indexer should do the trick.

Re: index time fields extraction from source??

rajasekhar14 — Wed, 22 May 2019 14:16:04 GMT

Thanks @DavidHourani

Re: index time fields extraction from source??

DavidHourani — Wed, 22 May 2019 14:43:07 GMT

Most welcome @rajasekhar14 ! Please accept or upvote my answer and comments ❤️

Re: index time fields extraction from source??

rajasekhar14 — Wed, 17 Jul 2019 18:31:19 GMT

Hi @koshyk ,

i have a small question on this, the above settings will use for source file name right? if i want to extract a index filed extraction in side from source file,?

i changed like this but its not working. can you please take a look.
props.conf
[ms:iis:auto]
TRANSFORMS-raj_namee = test-raj

Transforms.conf
[test-raj]
REGEX = ^(?:[^ \n]* ){2}([^ ]+)
FORMAT = appname::$1
WRITE_META = true

filed.conf
INDEXED=true

and the log format is

2019-07-17 18:21:33 xx-xx.xxx test 10.185.162.2 GET /monitor/monitor.html ----

and i'm using the above regex bold text and it need extract as a appname.