Getting Data In
Highlighted

How to forward logs with Splunk Universal Forwarder for the files with no header and logs should be in form of key/value

Engager

I have a Splunk Forwarder setup already on my host.

I have certain files on folder (/tom/mike/). File names are starting with Back*.

The content of file may in one or multiple line. There are multiple fixed position values in each line with no header.

Content (Example: Consider "-" as one space)

Tom---516-----RTYUI------45678
Mik---345-----XYXFF------56789

I need splunk logs for each line.

Like:

Key1= Tom   Key2=516   Key3= RTYUI  Key4= 45678

Key1= Mike  Key2= 345  Key3= XYXFF  Key4= 56789

I know inputs.conf changes would be like below.

[monitor:///tom/mike/Back*]
index=myIndex
blacklist=\.(gz|zip|bkz|arch|etc)$
sourcetype = BackFileData

Please suggest changes which can be done in props.conf. Please keep in mind that delimiter is fixed for each value in line but its not same (like 2 spaces) for all column values. There are no headers as well in these files.

0 Karma
Highlighted

Re: How to forward logs with Splunk Universal Forwarder for the files with no header and logs should be in form of key/value

Esteemed Legend

As long as the spacing is consistent, just use traditional RegEx means like this in transforms.conf:

[BackFileData_main_fields]
#Tom---516-----RTYUI------45678
REGEX = ^(\w{3})\s{3}(\d{3})\s{5}(\w{5})\s{6}(\d+)$
FORMAT = Key1::$1 Key2::$2 Key3::$3 Key4::$4
0 Karma
Highlighted

Re: How to forward logs with Splunk Universal Forwarder for the files with no header and logs should be in form of key/value

Engager

I assume that mainfields is a keyword here.

So BackFileDatamainfields means sourcetypemainfields ?

0 Karma
Highlighted

Re: How to forward logs with Splunk Universal Forwarder for the files with no header and logs should be in form of key/value

SplunkTrust
SplunkTrust

You'd need this in props.conf to enable the answer by @woodcock

[BackFileData]
TRANSFORMS-backfiledata=BackFileDatamainfields

0 Karma
Highlighted

Re: How to forward logs with Splunk Universal Forwarder for the files with no header and logs should be in form of key/value

Engager

Still logs are coming as a whole without keys.

I have done these changes in props.conf and transforms.conf. Regex is correct as I have checked that.

0 Karma
Highlighted

Re: How to forward logs with Splunk Universal Forwarder for the files with no header and logs should be in form of key/value

Engager

I think some more changes are required. Right now both lines are coming together without keys.

0 Karma
Highlighted

Re: How to forward logs with Splunk Universal Forwarder for the files with no header and logs should be in form of key/value

Esteemed Legend

You need to deploy to all indexers, restart all Splunk instances there, and then test by ONLY looking at post-restart events (old events will stay broken).

0 Karma
Highlighted

Re: How to forward logs with Splunk Universal Forwarder for the files with no header and logs should be in form of key/value

SplunkTrust
SplunkTrust

Here's how I would do it:

On the search heads, create an app with a props.conf:

 [BackFileData]
 EXTRACT-fields=^(?<Key1>\S+)\s+(?<Key2>\d+)\s+(?<Key3>\S+)\s+(?<Key4>\d+)

Then I'd make the permissions on the app "global" if I wanted the extraction to work for anyone in any app, "app level" if I only wanted it to work inside the app, or "private" if I only wanted myself to have the extraction (in any app).

Doing it this way will apply the schema at search time which is the best practice versus transforming the data and indexing the fields.

View solution in original post

Highlighted

Re: How to forward logs with Splunk Universal Forwarder for the files with no header and logs should be in form of key/value

Engager

How to make the permissions on the app "global"? I tried without any permission setup and got the same result.

0 Karma
Highlighted

Re: How to forward logs with Splunk Universal Forwarder for the files with no header and logs should be in form of key/value

SplunkTrust
SplunkTrust

You go to the little cog / gear next to apps on the home screen (app manager) and you click permissions next to the app. Don't forget to restart Splunk too.

0 Karma