topic Re: How to forward logs with Splunk Universal Forwarder for the files with no header and logs should be in form of key/value in Getting Data In

How to forward logs with Splunk Universal Forwarder for the files with no header and logs should be in form of key/value

anupam491 — Sun, 06 Aug 2017 00:29:22 GMT

I have a Splunk Forwarder setup already on my host.

I have certain files on folder (/tom/mike/). File names are starting with Back*.

The content of file may in one or multiple line. There are multiple fixed position values in each line with no header.

Content (Example: Consider "-" as one space)

Tom---516-----RTYUI------45678
Mik---345-----XYXFF------56789

I need splunk logs for each line.

Like:

Key1= Tom   Key2=516   Key3= RTYUI  Key4= 45678

Key1= Mike  Key2= 345  Key3= XYXFF  Key4= 56789

I know inputs.conf changes would be like below.

[monitor:///tom/mike/Back*]
index=myIndex
blacklist=\.(gz|zip|bkz|arch|etc)$
sourcetype = BackFileData

Please suggest changes which can be done in props.conf. Please keep in mind that delimiter is fixed for each value in line but its not same (like 2 spaces) for all column values. There are no headers as well in these files.

Re: How to forward logs with Splunk Universal Forwarder for the files with no header and logs should be in form of key/value

woodcock — Sun, 06 Aug 2017 12:35:58 GMT

As long as the spacing is consistent, just use traditional RegEx means like this in transforms.conf:

[BackFileData_main_fields]
#Tom---516-----RTYUI------45678
REGEX = ^(\w{3})\s{3}(\d{3})\s{5}(\w{5})\s{6}(\d+)$
FORMAT = Key1::$1 Key2::$2 Key3::$3 Key4::$4

Re: How to forward logs with Splunk Universal Forwarder for the files with no header and logs should be in form of key/value

anupam491 — Tue, 29 Sep 2020 15:16:50 GMT

I assume that _main_fields is a keyword here.

So BackFileData_main_fields means sourcetype_main_fields ?

Re: How to forward logs with Splunk Universal Forwarder for the files with no header and logs should be in form of key/value

jkat54 — Tue, 29 Sep 2020 15:15:14 GMT

You'd need this in props.conf to enable the answer by @woodcock

[BackFileData]
TRANSFORMS-backfiledata=BackFileData_main_fields

Re: How to forward logs with Splunk Universal Forwarder for the files with no header and logs should be in form of key/value

anupam491 — Mon, 07 Aug 2017 07:32:54 GMT

Still logs are coming as a whole without keys.

I have done these changes in props.conf and transforms.conf. Regex is correct as I have checked that.

Re: How to forward logs with Splunk Universal Forwarder for the files with no header and logs should be in form of key/value

anupam491 — Mon, 07 Aug 2017 07:36:15 GMT

I think some more changes are required. Right now both lines are coming together without keys.

Re: How to forward logs with Splunk Universal Forwarder for the files with no header and logs should be in form of key/value

jkat54 — Mon, 07 Aug 2017 12:28:45 GMT

Here's how I would do it:

On the search heads, create an app with a props.conf:

 [BackFileData]
 EXTRACT-fields=^(?<Key1>\S+)\s+(?<Key2>\d+)\s+(?<Key3>\S+)\s+(?<Key4>\d+)

Then I'd make the permissions on the app "global" if I wanted the extraction to work for anyone in any app, "app level" if I only wanted it to work inside the app, or "private" if I only wanted myself to have the extraction (in any app).

Doing it this way will apply the schema at search time which is the best practice versus transforming the data and indexing the fields.

Re: How to forward logs with Splunk Universal Forwarder for the files with no header and logs should be in form of key/value

anupam491 — Mon, 07 Aug 2017 20:13:22 GMT

How to make the permissions on the app "global"? I tried without any permission setup and got the same result.

Re: How to forward logs with Splunk Universal Forwarder for the files with no header and logs should be in form of key/value

jkat54 — Mon, 07 Aug 2017 21:38:56 GMT

You go to the little cog / gear next to apps on the home screen (app manager) and you click permissions next to the app. Don't forget to restart Splunk too.

Re: How to forward logs with Splunk Universal Forwarder for the files with no header and logs should be in form of key/value

anupam491 — Mon, 07 Aug 2017 22:19:19 GMT

I have checked and found this is as "app only" and I think that is correct.

Sorry to bother you but these steps didn't work. Can you please explain steps considering my initial questions.

Re: How to forward logs with Splunk Universal Forwarder for the files with no header and logs should be in form of key/value

jkat54 — Mon, 07 Aug 2017 22:30:04 GMT

If you change it to all apps that makes it global. Otherwise you will have to be in the app you created when you search for the extractions to work.

Re: How to forward logs with Splunk Universal Forwarder for the files with no header and logs should be in form of key/value

woodcock — Mon, 07 Aug 2017 22:36:27 GMT

You need to deploy to all indexers, restart all Splunk instances there, and then test by ONLY looking at post-restart events (old events will stay broken).

Re: How to forward logs with Splunk Universal Forwarder for the files with no header and logs should be in form of key/value

anupam491 — Tue, 29 Sep 2020 15:18:00 GMT

I am trying to go by first suggestion.
Directory

~/splunk/heavyforwarder/etc/apps/Splunk_TA_nix/default

props.conf

[BackFileData]
TRANSFORMS-backfiledata=BackFileData_main_fields

transforms.conf

[BackFileData_main_fields]
REGEX = ^(\w{3})\s{3}(\d{3})\s{5}(\w{5})\s{6}(\d+)$
FORMAT = Key1::$1 Key2::$2 Key3::$3 Key4::$4

inputs.conf

[monitor:///tom/mike/Back*]
index=cs_it
blacklist=.(gz|zip|bkz|arch|etc)$
sourcetype = BackFileData

I am still getting logs like below.

8/9/17
3:58:38.000 PM Tom 516 RTYUI 45678

Can someone please tell that what is wrong in transforms.conf/props.conf ?

Re: How to forward logs with Splunk Universal Forwarder for the files with no header and logs should be in form of key/value

jkat54 — Wed, 09 Aug 2017 23:20:48 GMT

Instead of the \s{x} where x is a number, try \s+. Same for \w and \d.

Re: How to forward logs with Splunk Universal Forwarder for the files with no header and logs should be in form of key/value

jkat54 — Wed, 09 Aug 2017 23:26:38 GMT

BTW, your logs will remain the same... it's the fields Key1-Key4 that you're creating here.

Are you trying to transform your data instead?

If so you will need SEDCMD in props.conf instead:

SEDCMD-keys=s/(\S+)\s+(\d+)\s+(\S+)\s+(\d+)/Key1=\1 Key2=\2 Key3=\3 Key4=\4/g

You won't need the transforms.

You will have to index new data to see the props applied, and the props need to be on the forwarder.

Re: How to forward logs with Splunk Universal Forwarder for the files with no header and logs should be in form of key/value

anupam491 — Thu, 10 Aug 2017 00:14:33 GMT

Thanks. Now keys are coming.

It is coming like

8/9/17
5:11:34.000 PM

Key1=Tom Key2=516 Key3=RTYUI Key4=45678
Key1=Mik Key2=516 Key3=RTYUI Key4=45678

But I want to forward each line separately like:

8/9/17
5:11:34.000 PM

Key1=Tom Key2=516 Key3=RTYUI Key4=45678

8/9/17
5:11:34.000 PM

Key1=Mik Key2=516 Key3=RTYUI Key4=45678

Re: How to forward logs with Splunk Universal Forwarder for the files with no header and logs should be in form of key/value

jkat54 — Thu, 10 Aug 2017 01:11:07 GMT

Add this to the props

 SHOULD_LINEMERGE=false
  LINE_BREAKER=([\n\r]+)