topic Re: How to modify CSV raw event data before fields extraction stage of INDEXED_EXTRACTIONS=CSV in Getting Data In

How to modify CSV raw event data before fields extraction stage of INDEXED_EXTRACTIONS=CSV

Maite35 — Fri, 27 Mar 2015 09:31:58 GMT

Hello,

I am using FIELD_DELIMITER=; and am working on data that use commas instead of decimals. I want to use a SED to replace those with dots when indexing (s /,/./ g) I tried this in props.conf:

SEDCMD-coma = s/,/./g

I also tried this in props. Conf :

TRANSFORMS-toto = toto

And in transforms.conf :

[toto]
REGEX = s/,/./g

And in all cases the behavior is the same : on my raw events ( _raw ) it works fine:

18/03/2015;23:50:00;XXX;XXX;XXX;16;6.52;41740109;0.03;46987.89;193790;0;12885230;0;25215.5;0;15;87;0;0;40008787;0;37.97;0;667;563.19;47255.63;525.22;369.59

But it never effects the fields that are exracted:

10 premières valeurs,          Nombre,     %
0          3832     6,415 %
0,07        108     0,181 %
0,76        103     0,172 %
0,02        97      0,162 %
0,77        96      0,161 %

Ideas to do this?

Thank you in advance. Best Regards.

Re: How to modify CSV raw event data before fields extraction stage of INDEXED_EXTRACTIONS=CSV

somesoni2 — Fri, 27 Mar 2015 20:16:32 GMT

Give this a try

In props.conf:

    SEDCMD-coma = s/(\d*),(\d*)/\1.\2/g

Re: How to modify CSV raw event data before fields extraction stage of INDEXED_EXTRACTIONS=CSV

Maite35 — Mon, 30 Mar 2015 07:04:33 GMT

hello somesoni2 and thank you for your answer and help.
The behavior with what you offer is the same as quoted above: dot is present in _raw but not passed on to the fields extracted from csv file.

Re: How to modify CSV raw event data before fields extraction stage of INDEXED_EXTRACTIONS=CSV

woodcock — Mon, 10 Aug 2015 13:35:19 GMT

How are you creating your fields? Are you using INDEXED_EXTRACTIONS as described here?

http://docs.splunk.com/Documentation/Splunk/latest/Data/Extractfieldsfromfileheadersatindextime

Re: How to modify CSV raw event data before fields extraction stage of INDEXED_EXTRACTIONS=CSV

Maite35 — Mon, 10 Aug 2015 13:41:45 GMT

Hi woodcock,

Yes I am using INDEXED_EXTRACTIONS=CSV

Re: How to modify CSV raw event data before fields extraction stage of INDEXED_EXTRACTIONS=CSV

woodcock — Mon, 10 Aug 2015 14:19:15 GMT

It looks like you will probably have to pre-process the file outside of Splunk. I wish there was more detail here:

http://docs.splunk.com/Documentation/Splunk/latest/admin/Configurationparametersandthedatapipeline

Re: How to modify CSV raw event data before fields extraction stage of INDEXED_EXTRACTIONS=CSV

Maite35 — Mon, 10 Aug 2015 14:22:05 GMT

Thanks for your help !
finaly, I used Data-model to sed my coma with point ...

Re: How to modify CSV raw event data before fields extraction stage of INDEXED_EXTRACTIONS=CSV

woodcock — Mon, 10 Aug 2015 14:41:12 GMT

OK, post exactly what you did as an Answer and then Accept your answer so that we can all learn.

Re: How to modify CSV raw event data before fields extraction stage of INDEXED_EXTRACTIONS=CSV

Maite35 — Tue, 11 Aug 2015 07:33:17 GMT

Finaly I used Date Model :

rex mode=sed field=FIELD "s/,/./g"

Re: How to modify CSV raw event data before fields extraction stage of INDEXED_EXTRACTIONS=CSV

woodcock — Tue, 11 Aug 2015 13:38:01 GMT

OK, your solution was to post-modify the fields one-by-one at search time. You don't have to use a Data Model, you can just do it like this whenever you need it (search bar, dashboard), like this:

... | rex mode=sed field=<SomeFieldName> "s/,/./g"

Re: How to modify CSV raw event data before fields extraction stage of INDEXED_EXTRACTIONS=CSV

rapmancz — Sat, 09 Apr 2016 21:13:42 GMT

please what did you do exactly?

Re: How to modify CSV raw event data before fields extraction stage of INDEXED_EXTRACTIONS=CSV

woodcock — Sun, 10 Apr 2016 01:20:36 GMT

You should "Accept" the answer from the person who gives you the answer.