How to modify CSV raw event data before fields extraction stage of INDEXED_EXTRACTIONS=CSV

[Maite35]

Hello,

I am using FIELD_DELIMITER=; and am working on data that use commas instead of decimals. I want to use a SED to replace those with dots when indexing (s /,/./ g) I tried this in props.conf:

SEDCMD-coma = s/,/./g

I also tried this in props. Conf :

TRANSFORMS-toto = toto

And in transforms.conf :

[toto]
REGEX = s/,/./g

And in all cases the behavior is the same : on my raw events ( _raw ) it works fine:

18/03/2015;23:50:00;XXX;XXX;XXX;16;6.52;41740109;0.03;46987.89;193790;0;12885230;0;25215.5;0;15;87;0;0;40008787;0;37.97;0;667;563.19;47255.63;525.22;369.59

But it never effects the fields that are exracted:

10 premières valeurs,          Nombre,     %
0          3832     6,415 %
0,07        108     0,181 %
0,76        103     0,172 %
0,02        97      0,162 %
0,77        96      0,161 %

Ideas to do this?

Thank you in advance. Best Regards.

[woodcock]

OK, your solution was to post-modify the fields one-by-one at search time. You don't have to use a Data Model, you can just do it like this whenever you need it (search bar, dashboard), like this:

... | rex mode=sed field=<SomeFieldName> "s/,/./g"

[Maite35 · Accepted solution]

Finaly I used Date Model :

rex mode=sed field=FIELD "s/,/./g"

[woodcock]

You should "Accept" the answer from the person who gives you the answer.

[woodcock]

It looks like you will probably have to pre-process the file outside of Splunk. I wish there was more detail here:

http://docs.splunk.com/Documentation/Splunk/latest/admin/Configurationparametersandthedatapipeline

[Maite35]

Thanks for your help !
finaly, I used Data-model to sed my coma with point ...

[rapmancz]

please what did you do exactly?

[woodcock]

OK, post exactly what you did as an Answer and then Accept your answer so that we can all learn.

[woodcock]

How are you creating your fields? Are you using INDEXED_EXTRACTIONS as described here?

http://docs.splunk.com/Documentation/Splunk/latest/Data/Extractfieldsfromfileheadersatindextime

[Maite35]

Hi woodcock,

Yes I am using INDEXED_EXTRACTIONS=CSV

[Maite35]

hello somesoni2 and thank you for your answer and help.
The behavior with what you offer is the same as quoted above: dot is present in _raw but not passed on to the fields extracted from csv file.

[somesoni2]

Give this a try

In props.conf:

    SEDCMD-coma = s/(\d*),(\d*)/\1.\2/g