Getting Data In

When import TSV file, a field end with '\' will be combined with the following field wrongly

ryu_kahou
Explorer

I'm importing tab-delimited files formatted as the following. The space is tab.

 "field1    field2    field3    field4"

The files are imported with command "splunk add oneshot " and parsed with the following definition in transforms.conf.

DELIMS = "\t"
FIELDS = "field_name1","field_name2","field_name3","field_name4"

It works fine but for some line the value goes into wrong field name. It took me a lot of time to find the reason. I found that if a field ends with '\', it will be combined with its next field.

For example, "field1 field2\ field3 field4" will be parsed as

field_name1=field1
field_name2=field2\ field3
field_name3=field4
field_name4=

If I remove all ending '\' it will work fine. But I have to import the original data as is. Is there any way to import field end with '\'?

0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

The backslash is escaping the tab so it is treated as an ordinary character rather than as a separator. One solution is to use a regex. Replace the DELIMS and FIELDS lines in your transforms.conf with this line:

REGEX=(?<field_name1>.*?[\\]?)\t(?<field_name2>.*?[\\]?)\t(?<field_name3>.*?[\\]?)\t(?<field_name4>.*?)

If the backslash is not part of the field value then use this REGEX string:

REGEX=(?<field_name1>.*?)[\\]?\t(?<field_name2>.*?)[\\]?\t(?<field_name3>.*?)[\\]?\t(?<field_name4>.*?)
---
If this reply helps you, an upvote would be appreciated.

View solution in original post

richgalloway
SplunkTrust
SplunkTrust

The backslash is escaping the tab so it is treated as an ordinary character rather than as a separator. One solution is to use a regex. Replace the DELIMS and FIELDS lines in your transforms.conf with this line:

REGEX=(?<field_name1>.*?[\\]?)\t(?<field_name2>.*?[\\]?)\t(?<field_name3>.*?[\\]?)\t(?<field_name4>.*?)

If the backslash is not part of the field value then use this REGEX string:

REGEX=(?<field_name1>.*?)[\\]?\t(?<field_name2>.*?)[\\]?\t(?<field_name3>.*?)[\\]?\t(?<field_name4>.*?)
---
If this reply helps you, an upvote would be appreciated.

View solution in original post

ryu_kahou
Explorer

Thank you for your answer.

I think it's a bug of DELIMS. Even simple split function doesn't have the problem.

0 Karma
Take the 2021 Splunk Career Survey

Help us learn about how Splunk has
impacted your career by taking the 2021 Splunk Career Survey.

Earn $50 in Amazon cash!