Unable to import a file with two spaces as separator, which occur once. How to edit my transform configurations?

[strousseau]

Hello,

I'm trying to import this kind of file :

\#DATE TITRE1 TITRE2 TITRE3
#LINE TO IGNORE
20170101  LIGNE1COL1 LIGNE1COL2 "LIGNE1 COL 3"
20170101  LIGNE2COL1 LIGNE2COL2 "LIGNE2 COL 3"
20170101  LIGNE3COL1 LIGNE3COL2 "LIGNE3 COL 3"
20170101  LIGNE1COL1 LIGNE1COL2 "LIGNE1 COL 3"
20170101  LIGNE2COL1 LIGNE2COL2 "LIGNE2 COL 3"
20170101  LIGNE3COL1 LIGNE3COL2 "LIGNE3 COL 3"
20170101  LIGNE1COL1 LIGNE1COL2 "LIGNE1 COL 3"
20170101  LIGNE2COL1 LIGNE2COL2 "LIGNE2 COL 3"
20170101  LIGNE3COL1 LIGNE3COL2 "LIGNE3 COL 3"
20170101  LIGNE1COL1 LIGNE1COL2 "LIGNE1 COL 3"
20170101  LIGNE2COL1 LIGNE2COL2 "LIGNE2 COL 3"

Please take take of two space between the date and the second column in the "values" oart and only one space in the "header" part.
Titles are in the first line.
I don't want the second line to be read.

So, I defined a new source type with spaces as separator. I ignore lines starting with character # and I say fields names are on the first line.
The problem is that Splunk defines a new field named "EXTRA_FIELD_5" with values from the last column ("LIGNE1 COL 3").
The field TITRE2 has values of the second column (LIGNE1COL1) and the field TITRE3 has values like LIGNE1COL2.

If I replace the two spaces into only one space, the values are loaded in the right way except the second line which is read whatever i set up.

So I tried to use transformation to transform two spaces in one space but this doesn't work. Here is the rules :

[source::\*mycompany\*.log]
#SEDCMD-single-whitespace = s/  / /g
#SEDCMD-remove-line = s/#LINE TO IGNORE\n//

I am not sure I'm using the right props.conf (Splunk\etc\users\admin\search\local\props.conf, Splunk\etc\apps\search\default\props.conf or Splunk\etc\system\default\props.conf).

I don't understand why the second line is not ignored.
I googled the problem and saw some answer talking about HEADER_FIELD_LINE_NUMBER = 1 and PREAMBLE_REGEX = ^#.*
This doesn't work.

I think my syntax is correct because this search works : source="\*mycompany\*.log" | rex field=DATE mode=sed "s/2017/2018/g"

I'm using Splunk Light Free Version 6.4.1

So please help me to understand what is wrong and how to solve my problem.

Regards

Stephane

[strousseau]

Hello,

Some good news.
When I defined my own sourcetype, it was assigned to the app search and not to the app system.
If I assign my sourcetype to app system it works !!! But only for the transformation of the two spaces !! The third line is used as fields names.

If I try to manage the first line as fields names and to ignore the second line, the transformation of two spaces is no more executed !!

[hunters_splunk]

Hi strousseau,

First, you should edit your props.conf located in Splunk\etc\apps\search\local\, which will override the .conf file under ...\default.
Secondly, the "sed-like" syntax in incorrect in your props.conf, which is not the same as the sedcmd in the rex syntax.
The sedcmd syntax is used for simplified data modifications and provides “search and replace” using regular expressions and substitutions.
Example: Hide the first 5 digits of an account number in the vendor_sales.log source

[22/Oct/2014:00:46:27] VendorID=9112 Code=B AcctID=4902636948
[22/Oct/2014:00:48:40] VendorID=1004 Code=J AcctID=4236256056
[22/Oct/2014:00:50:02] VendorID=5034 Code=H AcctID=8462999288

In props.conf:

[source::.../vendor_sales.log]
SEDCMD-1acct = s/AcctID=\d{5}(\d{5})/AcctID=xxxxx\1/g

Note that \1 indicates the captured group.
In your case, if you want to replace double spaces with a single space, you should use \s in REGEX to represent spaces.

 [source::\*mycompany\*.log]
 #SEDCMD-single-whitespace = s/(^\d{8})\s\s/\1 /g

Hope this helps. Thanks!
Hunter

[strousseau]

Hello,

Depending on setup for horodating, I have this message in the import wizard : CSV StreamId: 0 has extra incorrect columns in certain fields.
I have written the SEDCMD line to convert two spaces in one space but it seems not to be executed.
Another rule to convert 2017 to 2018 is not executed.

Regards

[strousseau]

Hello,

Is there any log I can read to see if the rule is called ?
Is SEDCMD is functionnal on Windows ?

I don't understand what I'm doing wrong. Even replace 2017 by 2018 is ko.

Please help me...

strousseau

[strousseau]

Hello,

I made some another tries with vendor_sales.log from the tutorial and put it on d:.
I updated C:\Program Files\Splunk\etc\apps\search\local\props.conf

I never succeeded to make the replacement.
Even with the wizard, no replacement.

Regards,

Stéphane

[DalJeanis]

have you been restarting the indexer for each change?

According to this post - https://answers.splunk.com/answers/210096/how-to-configure-sedcmd-in-propsconf.html - the SED scripts work at indexing time, so indexer restart is required.

[strousseau]

Yes,
I restart splunk server at each try.
I have the same problem with Splunk Enterprise 6.5.1 on linux

When I import datas in Splunk, at the second step (somthing like setup source type) and put my own sourcetype, I see my SEDCMD rule but nothing happens. I try to replace 2017 by 2018, a basic rule, and I doesn't work.

[strousseau]

Hello,

I made so many tries that I am not sure there is no typo in my code.
I even tried to insert the rule in two différents sections of props.conf. First one is the section for my new source type and second one is the section named [source::*mycompany*.log]. None works.

So I will try the sample from hunters [Splunk] with replacement in AcctId field.

Regards

Stéphane

[strousseau]

Hello,

Thanks a lot for your answer,
I made many tries but unfortunately, no one worked.

I modified the right props.cof (C:\Program Files\Splunk\etc\apps\search\local\props.conf) and restarted Splunk each time.
Even at import time, I tried to modify the search and replace string in Advanced tab and it didn't work.

Here are the different syntax I used.
SEDCMD-single-whitespace = s/2017/2018/g
SEDCMD-single-whitespace = s/(^\d{8})\s\s/\1\s/g
SEDCMD-single-whitespace = s/(^\d{8})\s\s/\1 /g
SEDCMD-single-whitespace = s/(^\d{8})\s\s/\1/g
SEDCMD-single-whitespace = s/(^\d{8})\s\s(*)/\1 \2/g

The first one is very simple but it didn't work !

Di I need to make a tranformation and modify traform.conf ?

Regards

Stephane

[DalJeanis]

Stephane -

Your sample codes look right,

According to this post - https://answers.splunk.com/answers/210096/how-to-configure-sedcmd-in-propsconf.html - SED scripts work at indexing time, so indexer restart is required.