topic Re: Log pre processing in Getting Data In

Log pre processing

faustf — Mon, 13 Feb 2017 16:38:32 GMT

Hi guys,
I defined my source type as follow (in props.conf):

[anomalies]
DATETIME_CONFIG =
FIELD_NAMES = COL1, COL2, TIMESTAMP, COL4, COL5, KPI_ID ,COL7, COL8, COL9, COL10, COL11, COL12, COL13, ALARM
INDEXED_EXTRACTIONS = csv
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
category = AAAA
pulldown_type = 1
disabled = false
FIELD_DELIMITER = ,
TIME_PREFIX = .*?,.*?,
MAX_TIMESTAMP_LOOKAHEAD = 10
TZ = UTC

and my log file is this:

1,2,1411261200000,4,5,6,7,8,9,10,[11],12,13,[ALARM]
1,2,1411261200000,4,5,6,7,8,9,10,[11],12,13,[ALARM]

My problem is that I need to replace all the [ and ] characters with "[ or ]"
I need this pre-processing because in my log file I've also some lines in the following format:

1,2,1411261200000,4,5,6,7,8,9,10,[11,111,1111],12,13,[ALARM]

The field [11,111,1111] is my problem because Splunk split this filed in 3 different fields:
[11
111
1111]

How can I solve this problem?

Thank you!

Re: Log pre processing

jkat54 — Mon, 13 Feb 2017 16:50:07 GMT

Add this to your props.conf on your ingestion side (forwarders or indexers)

[sourceTypeName]
SEDCMD-AAAremoveSquareBrackets = s/\[|\]//g

This is to say... for every [ OR ] replace with nothing and do so globally (g is probably not required in this case)

We put AAA in front of class name because SEDCMD's are processed with ASCII order of precedence and AAA is most likely the highest priority in you environment.

Re: Log pre processing

faustf — Tue, 14 Feb 2017 07:25:54 GMT

Thank for your answer.

I need to convert the list in square brackets in a single element otherwise the number of the fields of the csv depends on the number of the elements in the square list. I modified the props.conf in this way

[anomalies]
....
SEDCMD-squares-open = s/\[/"[/g
SEDCMD-squares-close = s/\]/]"/g

But, even in this way, the list is split into several elements

As you can see in the first line of the image the square brackets have been replaced with "[ and ]" but Splunk still split these elements (COL11,COL12,COL13)

Re: Log pre processing

gvmorley — Tue, 14 Feb 2017 08:32:31 GMT

Hi,

A different approach would be to do the extractions at Search time, using a combination of props.conf and transforms.conf

I did a quick test with this in my props.conf:

[test-csv]
REPORT-fields = sourcetype-test-csv

You'd want to remove all of the FIELD_NAMES, INDEXED_EXTRACTIONS and FIELD_DELIMITER stuff.

And this in my transforms.conf:

[sourcetype-test-csv]
REGEX = ([^,]+),([^,]+),([^,]+),([^,]+),([^,]+),([^,]+),([^,]+),([^,]+),([^,]+),([^,]+),\[([^]]+)\],([^,]+),([^,]+),\[([^]]+)\]
FORMAT = COL1::$1 COL2::$2 COL3::$3 COL4::$4 COL5::$5 COL6::$6 COL7::$7 COL8::$8 COL9::$9 COL10::$10 COL11::$11 COL12::$12 COL13::$13 COL14::$14

This looks like it would give you what you're looking for?

Just make sure that the REGEX you use in transforms.conf is appropriate for the format of your data. It was a rough and ready example, so may need refining!

Hope that helps to get you closer to what you need.
(Oh and change the name of the 14th field to be ALARM if you want it to be the same as your example)

Re: Log pre processing

faustf — Tue, 14 Feb 2017 09:03:00 GMT

Thanks.
This seems what I need but I don't know why it isn't working on my environment.

I've a Splunk Enterprise server where I crated an index called "anomalies".
Also I have a second node where I installed the Splunk universal forwarder

The configuration of the Splunk universal forwarder is the following:

File: /opt/splunkforwarder/etc/apps/search/local/inputs.conf

[monitor:///tmp/my.log]
disabled = false
index = anomalies
sourcetype = test-csv

File: /opt/splunkforwarder/etc/system/local/props.conf

[test-csv]
DATETIME_CONFIG = CURRENT
REPORT-fields = sourcetype-test-csv

File: /opt/splunkforwarder/etc/system/local/transforms.conf

[sourcetype-test-csv]
REGEX = ([^,]+),([^,]+),([^,]+),([^,]+),([^,]+),([^,]+),([^,]+),([^,]+),([^,]+),([^,]+),\[([^]]+)\],([^,]+),([^,]+),\[([^]]+)\]
FORMAT = COL1::$1 COL2::$2 COL3::$3 COL4::$4 COL5::$5 COL6::$6 COL7::$7 COL8::$8 COL9::$9 COL10::$10 COL11::$11 COL12::$12 COL13::$13 COL14::$14

File /tmp/my.log

1,2,1411261200000,4,5,6,7,8,9,10,[11],12,13,[ALARM]
1,2,1411261200000,4,5,6,7,8,9,10,[11],12,13,[ALARM]
1,2,1411261200000,4,5,GCD244,7,8,9,10,[11,12,13,[ALARM]
1,2,1411261200000,4,5,6,7,8,9,10,[11],12,13,[ALARM]
1,2,1411261200000,4,5,6,7,8,9,10,[11,111,1111],12,13,[ALARM]

Splunk Web search:

Re: Log pre processing

gvmorley — Tue, 14 Feb 2017 09:07:10 GMT

I believe that the props.conf and transforms.conf files need to be on your Search Head. Which, if you have just one Splunk Enterprise server, is the same as the Indexer.

My understanding is that the Universal Forwarder doesn't do any 'processing'. You'd need to use a Heavy Forwarder for that.

But as these props and transforms are after indexing, your Search Head (Splunk Server) is the place for them.

Re: Log pre processing

gvmorley — Tue, 14 Feb 2017 09:08:44 GMT

Also,

Always have a go with this stuff on a development system first!

I'd recommend running Splunk (with the Free License) on your laptop or PC, so that it's easy to test with.

Re: Log pre processing

faustf — Tue, 14 Feb 2017 09:17:03 GMT

You are right.
I moved the props.conf and transforms.conf in the Search Head. Perfect!!

But I don't understand why the first version of my props.conf was working (it was in the Universal Splunk forwarder). This confused me.

Thank you again!!!

Re: Log pre processing

faustf — Tue, 14 Feb 2017 09:17:16 GMT

Of course!!!
Thanks

Re: Log pre processing

gvmorley — Tue, 14 Feb 2017 09:25:20 GMT

No problem.

Just to round off the 'why was props.conf working on the forwarder'...

From looking at the manual here:
http://docs.splunk.com/Documentation/Splunk/latest/Admin/Propsconf

If you look at the section on 'Structured Data Header Extraction and configuration', there's this:

"This feature and all of its settings apply at input time, when data is first read by Splunk. The setting is used on a Splunk system that has configured inputs acquiring the data."

So, as they apply at 'input' time, they get used by the forwarder, as it's the one with the configured input.

At least that's my understanding!

Re: Log pre processing

jkat54 — Tue, 14 Feb 2017 12:07:32 GMT

Well that's not what your original question said so my solution won't work for the updated version of your question.

They were broke into different fields because of the comma delimiter. It was my understanding that you just needed to remove square brackets. Happy you found your solution either way!