I have a Splunk Forwarder setup already on my host.
I have certain files on folder (/tom/mike/). File names are starting with Back*.
The content of file may in one or multiple line. There are multiple fixed position values in each line with no header.
Content (Example: Consider "-" as one space)
Tom---516-----RTYUI------45678
Mik---345-----XYXFF------56789
I need splunk logs for each line.
Like:
Key1= Tom Key2=516 Key3= RTYUI Key4= 45678
Key1= Mike Key2= 345 Key3= XYXFF Key4= 56789
I know inputs.conf changes would be like below.
[monitor:///tom/mike/Back*]
index=myIndex
blacklist=\.(gz|zip|bkz|arch|etc)$
sourcetype = BackFileData
Please suggest changes which can be done in props.conf. Please keep in mind that delimiter is fixed for each value in line but its not same (like 2 spaces) for all column values. There are no headers as well in these files.
Here's how I would do it:
On the search heads, create an app with a props.conf:
[BackFileData]
EXTRACT-fields=^(?<Key1>\S+)\s+(?<Key2>\d+)\s+(?<Key3>\S+)\s+(?<Key4>\d+)
Then I'd make the permissions on the app "global" if I wanted the extraction to work for anyone in any app, "app level" if I only wanted it to work inside the app, or "private" if I only wanted myself to have the extraction (in any app).
Doing it this way will apply the schema at search time which is the best practice versus transforming the data and indexing the fields.
Here's how I would do it:
On the search heads, create an app with a props.conf:
[BackFileData]
EXTRACT-fields=^(?<Key1>\S+)\s+(?<Key2>\d+)\s+(?<Key3>\S+)\s+(?<Key4>\d+)
Then I'd make the permissions on the app "global" if I wanted the extraction to work for anyone in any app, "app level" if I only wanted it to work inside the app, or "private" if I only wanted myself to have the extraction (in any app).
Doing it this way will apply the schema at search time which is the best practice versus transforming the data and indexing the fields.
How to make the permissions on the app "global"? I tried without any permission setup and got the same result.
I am trying to go by first suggestion.
Directory
~/splunk/heavyforwarder/etc/apps/Splunk_TA_nix/default
props.conf
[BackFileData]
TRANSFORMS-backfiledata=BackFileData_main_fields
transforms.conf
[BackFileData_main_fields]
REGEX = ^(\w{3})\s{3}(\d{3})\s{5}(\w{5})\s{6}(\d+)$
FORMAT = Key1::$1 Key2::$2 Key3::$3 Key4::$4
inputs.conf
[monitor:///tom/mike/Back*]
index=cs_it
blacklist=.(gz|zip|bkz|arch|etc)$
sourcetype = BackFileData
I am still getting logs like below.
8/9/17
3:58:38.000 PM Tom 516 RTYUI 45678
Can someone please tell that what is wrong in transforms.conf/props.conf ?
Instead of the \s{x} where x is a number, try \s+. Same for \w and \d.
BTW, your logs will remain the same... it's the fields Key1-Key4 that you're creating here.
Are you trying to transform your data instead?
If so you will need SEDCMD in props.conf instead:
SEDCMD-keys=s/(\S+)\s+(\d+)\s+(\S+)\s+(\d+)/Key1=\1 Key2=\2 Key3=\3 Key4=\4/g
You won't need the transforms.
You will have to index new data to see the props applied, and the props need to be on the forwarder.
Thanks. Now keys are coming.
It is coming like
8/9/17
5:11:34.000 PM
Key1=Tom Key2=516 Key3=RTYUI Key4=45678
Key1=Mik Key2=516 Key3=RTYUI Key4=45678
But I want to forward each line separately like:
8/9/17
5:11:34.000 PM
Key1=Tom Key2=516 Key3=RTYUI Key4=45678
8/9/17
5:11:34.000 PM
Key1=Mik Key2=516 Key3=RTYUI Key4=45678
Add this to the props
SHOULD_LINEMERGE=false
LINE_BREAKER=([\n\r]+)
You go to the little cog / gear next to apps on the home screen (app manager) and you click permissions next to the app. Don't forget to restart Splunk too.
I have checked and found this is as "app only" and I think that is correct.
Sorry to bother you but these steps didn't work. Can you please explain steps considering my initial questions.
If you change it to all apps that makes it global. Otherwise you will have to be in the app you created when you search for the extractions to work.
As long as the spacing is consistent, just use traditional RegEx means like this in transforms.conf:
[BackFileData_main_fields]
#Tom---516-----RTYUI------45678
REGEX = ^(\w{3})\s{3}(\d{3})\s{5}(\w{5})\s{6}(\d+)$
FORMAT = Key1::$1 Key2::$2 Key3::$3 Key4::$4
I assume that _main_fields is a keyword here.
So BackFileData_main_fields means sourcetype_main_fields ?
Still logs are coming as a whole without keys.
I have done these changes in props.conf and transforms.conf. Regex is correct as I have checked that.
I think some more changes are required. Right now both lines are coming together without keys.
You need to deploy to all indexers, restart all Splunk instances there, and then test by ONLY looking at post-restart events (old events will stay broken).
You'd need this in props.conf to enable the answer by @woodcock
[BackFileData]
TRANSFORMS-backfiledata=BackFileData_main_fields