Splunk Search

Splunk not displaying log data

jangid
Builder

My log file contain a long line (35000 chars) with continuous spaces [more then 60 spaces] multiple times inside the log - I can't see this log information in Splunk.

I can't change the log because its coming from 3rd party tool

Any Idea?

Tags (2)
1 Solution

kristian_kolb
Ultra Champion

Have you looked at the TRUNCATE parameter for props.conf?

By default Splunk will cut lines that are longer than 10000 chars. Changing this to 50000 might improve your situation.

Other things to check - if you are not seeing the data at all - are the simple things;

Look for interesting error messages in splunkd.log
Are timestamps parsed correctly? Try to search for 'All Time'
Are you looking in the right index?
Do you have permissions to read that index? Check in the Manager -> Account controls -> roles -> your role

Hope this helps,

Kristian

View solution in original post

kristian_kolb
Ultra Champion

Have you looked at the TRUNCATE parameter for props.conf?

By default Splunk will cut lines that are longer than 10000 chars. Changing this to 50000 might improve your situation.

Other things to check - if you are not seeing the data at all - are the simple things;

Look for interesting error messages in splunkd.log
Are timestamps parsed correctly? Try to search for 'All Time'
Are you looking in the right index?
Do you have permissions to read that index? Check in the Manager -> Account controls -> roles -> your role

Hope this helps,

Kristian

jangid
Builder

I can't see this error anymore

0 Karma

jangid
Builder

Hi Drainy, can you please tell me how to remove TAB and space char before indexing?

0 Karma

Drainy
Champion

You can remove them before indexing, but its unlikely that Splunk is struggling with them. It could be the Web UI struggling to render them perhaps but I am dubious that it isn't dealing with the tab or spaces as at a very basic level they are just another character.

0 Karma

jangid
Builder

Thanks for your quick response.
in splunk is there any option to remove tab char or spaces between > < char before forwarding?

0 Karma

Drainy
Champion

Sorry, I've been out and abooot. Well Splunk shouldn't have any issue with that really, if anything it could be a line-breaking issue with how Splunk is splitting the data. Perhaps you could paste it to pastebin.com, then we can grab a decent copy to test on our own systems

0 Karma

jangid
Builder

Hi Drainy any update regarding my query?

0 Karma

jangid
Builder

I0705 13:59:35.496698 19354 test.cpp:500] test::call: setParam(i=0, 'TEST.XML,,,,,,<<?xml version="1.0" encoding="UTF-8"?>###.APPLICATION162588014750291.03:5 tab1 US0010001 ###.APPLICATION #####.######## , I VALIDATE ID003350000000F3 ########FTI_1341493112469 <####S##een>#########.##############_801475029102</####S##een> fieldName:###########.#### fieldName:#####.####.## fieldName:#####.######## fieldName:#####.###### fieldName:######.####.## <###ordRead>1</###ordRead> ##.###.###.## #########.#### ####.####.#####.### ####.####.##### 1 ####.####.##.#### ####.####.## 1 ############## ############### ############## ############### ############# ########## ############ ########## ############# ################# ############### ########### ###.##### 1 ###.######## 1 ###.###.####### ###########.## ##.#########.#### ##.##.##.#### 1 ##.##.##.### 1 1 ####.###### ####.###.###### 1 ######.### 1 ######.#### ######.COM.DISPLAY ######.### 1 ######.#### 1 ########.######## ######.####.## ####.#### ######.##### ######.###### ###.######.### ###.###.###.### ##.#### COLL.###.## ##########.### 1 ##########.#### ##########.### 1 ##########.#### 1 #####.###### ##.######.###D.Y.N ######.####.## IBMUSD ######.###### ######.####.#### ######.######## ######.######## ######.#####.### ######.#####.#### ####.## ########.###.## 1 ########.###.DR 1 ####.GROUP.LEVEL ########.#### ########.SPREAD DATE.TIME 1 DEAL.MARKET DEALER.DESK #####.####.## DELLUSD #####.###### 1000 #####.####.#### #####.######## USD #####.######## #####.#####.### #####.#####.#### ########.##### ########.### ########.###### 1 ####.#### DR.######.###D.Y.N DRAWN.###OUNT EXPECTED.#####.## EXPECTED.###S.## EXPOSURE.#### EXTEND.###MAT EXTEND.#### FED.##### FREE.TEXT.###TO 1 ##.###.#####.### ##.###.#####.## 1 ##.####.#####.### ##.###.####.## ##.###.##### 1 ##.###.######## 1 ##.###.###.####### ##.##.##.## 1 ##.#.###.###.## 1 ##.#.##.#.##.## 1 ##.#.##TMED.## 1 ##.#.###.## 1 ##.#####.####.## ##.#####.VDATE ##.EXCH.#### ##.##STR.#### 1 ##.##TERMED.### ##.##TERMED.## 1 ##.##TMED.##### 1 ##.###.####.#### ##.ORDERING.## 1 ##.ORDERING.CUS 1 ##.###.######S 1 ##.PROCESS.ERR 1 ##.REASON.OVE 1 ##.###.### ##.###.####.### ##.###.####.## 1 ##.####.### 1 ##.####.####.## 1 ##.#####.### 1 ##.TIME.### 1 INP.###.### ######## 1 INSTRCTN.#### 1 ##########.### #######.#####.### #######.##### 1 ###.####.### ######.###.#### ######.######.#### ###.###.######## ###.###.######## ###.####.### ###.###.###S.### ###.##T.###.### #####.#.######.### #####.#.##### MAILING MESSAGE.#### 1 MESSAGE 1 ####.####.###### ###.#### 1 ###.#### 1 #####.#### ###.######.###NO #######.###### ##.##.#####.##### ###.####.#### ###.####.#### ORDERING.##### 1 ORDERING.#### 1 OVERRIDE 1 ######.###.## ###.###### #####.#### #######.######S 1 ########.#### ##########.### ##########.#### ######.######.#### ######.######.#### ####.###### ####.######.### ####.#####.### ###.####.#####.### ###.####.##### 1 ######.###.#### ########.##### ######.###### ########.### 1 ########.### ########.1 ########.10 ########.11 ########.12 ########.13 ########.14 ########.15 ########.16 ########.17 ########.18 ########.2 ########.3 ########.4 ########.5 ########.6 ########.7 ########.8 ########.9 ######.###### ######.##.#### ########.### #####.#### #######.### ####.#######.Y.N ####.##.##### 1 ######.## ######### 1 ###### ####.### 1 #####.###### #####.######## ###.### 1 ###.#### 1 #####.####.#### ###.######### ####.### 1 1 ###.###.### ###.###.###.##### ###.###.###.### ###.###.#### ###.###.####.### ###.###.###.##### #####.######.### #####.###.###### ###########.#### AC ########.#### ###.###.### ')

0 Karma

jangid
Builder
0 Karma

Drainy
Champion

Could you paste the data in pastebin? This will keep its original formatting and we can test it on our systems. I doubt its lost any of the data, its probably just parsing it or rendering it oddly 🙂

0 Karma

jangid
Builder

After removing tab spaces I am still loosing my data in splunk 😞

0 Karma

jangid
Builder

I think this is bug in Splunk - this same behavior for similar log with multiple tab or spaces.

can you guys please confirm?

0 Karma

jangid
Builder

Does anyone get a chance to figure out?

0 Karma

jangid
Builder

I am not sure whether this editor will truncate some TAB char or not, if you give me your email address I'll send you text format.

0 Karma

Drainy
Champion

Could you post some example data with the secret parts hashed out or obscured?

0 Karma

jangid
Builder

is this bug in Splunk?

0 Karma

jangid
Builder

I can see the data after this line in splunk search

0 Karma

Drainy
Champion

How did you verify that Splunk has read the file correctly?

0 Karma

jangid
Builder

Yes I verified - my log line contains 15 tab char continuously 9-10 times.

due to some privacy reason I can't post entire line here.

0 Karma

Drainy
Champion

Have you verified that it has read the file correctly or checked for any errors in the internal logs?

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...