Splunk Search

PROPS Configuration for text file with header

SplunkDash
Motivator

Hello,

I have a text source file with header. Some sample events (first line is a header) and props that I wrote given below.

My props is working ok, except it breaks the events at TEST\2qw123|Employee, TEST\3eraa2|Employee, TEST\87xaqw|Employee, at Obj.BasePage.Page, TEST\m69xcb, at Obj.BasePage.Page, and TEST\7yxccd|Employee  instead of breaking events at  TEST\2qw123|Employee, TEST\3eraa2|Employee, TEST\87xaqw|Employee, TEST\m69xcb, and TEST\7yxccd|Employee .  So from following sample events, I should have 5 events , but getting 7 events. Any help will be highly appreciated. Thank you. 

UserID|UserType|System|EventType|EventID|Subject|SessionID|SrcAddr|EventStatus|TimeStamp|AdditionalData|DeviceID|DestSrcAddr
TEST\2qw123|Employee|COM|TESTUSER|NTINCheckKCase|089524234|ybzjlie3d4ayr1i2|10.212.48.121|00|20220217122935|Case Information request: (Case-170) - 201612-30|mct0ma01ma4352855|10.219.174.222
TEST\3eraa2|Employee|COM|TESTUSER|NTINCheckKCase|046453942|ybzjlie3d4ayr1i2|10.212.48.121|00|20220217123142|Case Information request: (Case -85) - 201912-30|mct0ma01ma4352855|10.219.174.222
TEST\87xaqw|Employee|COM|SYSTEM|SystemMsg||zsod0mvomcelp3hvln5smm1u|10.216.22.17|01|20220217124743|Type:'error'; Ref:'Case/CaseInventory.aspx?Query=true&Scope=ServiceWide'; Msg: experienced <br>Source: App_Web_pc<br>Message: Object reference not set to an instance of an object.<br> /Case/CaseInventory.aspx<br>Trace: at Case.CaseInventory()
at Obj.BasePage.Page_Load(Object sender, EventArgs e)<br><br>Please try to login again.|mct0ma01ma4382154|10.210.174.221
TEST\m69xcb|Employee|COM|SYSTEM|SystemMsg||z0ae3c25zggbzx5p|10.215.173.231|01|20220217130933|Type:'error'; Ref:'Case/CaseInventory.aspx?Query=true&Scope=ServiceWide'; Msg: experienced a error:<br><br>Source: App_Web_pcf3kniw<br>Message: Object reference not set to an instance of an object.<br> /Case/CaseInventory.aspx<br>Trace: at Case.CaseInventory.page_load3()
at Obj.BasePage.Page_Load(Object sender, EventArgs e)<br><br>Please try to login again.|mct0ma01ma4353159|10.210.174.221
TEST\7yxccd|Employee|COM|TESTUSER|NTINCheckKCase|008422123|zggbzx5pzgnw1nih|10.215.173.231|00|20220217131108|Case Information request: (Case -24) - 202112-30|mct0ma1ma4353159|10.210.174.221

 

[sourcename]

SHOULD_LINEMERGE=false

LINE_BREAKER=([\r\n]+)

INDEXED_EXTRACTIONS=psv

MAX_TIMESTAMP_LOOKAHEAD=14

HEADER_FIELD_LINE_NUMBER=1

TIME_FORMAT=%Y%m%d%H%M%S

TIMESTAMP_FIELDS=TimeStamp

TRUNCATE=2000

Labels (1)
Tags (1)
0 Karma
1 Solution

VatsalJagani
SplunkTrust
SplunkTrust

No, it (changing LINE_BREAKER) shouldn't make any difference as you are using INDEXED_EXTRACTION.

 

------
Upvote would be appreciated!!!

View solution in original post

0 Karma

PickleRick
SplunkTrust
SplunkTrust

Your data is inconsistent with the definition. You have header specifying some fields and then you have two events with not enough data to fill those fields,

VatsalJagani
SplunkTrust
SplunkTrust

Try using search-time field extraction instead of Index time (INDEXED_EXTRACTIONS) with below configurations:

 

props.conf

[sourcename]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)
TRUNCATE = 2000
TRANSFORMS-filter_events = data_filter_headers
TIME_PREFIX = [^|]*|[^|]*|[^|]*|[^|]*|[^|]*|[^|]*|[^|]*|[^|]*|[^|]*|
MAX_TIMESTAMP_LOOKAHEAD = 20
TIME_FORMAT = %Y%m%d%H%M%S
REPORT-headers = data_headers

 

transforms.conf

[data_headers]
CLEAN_KEYS = 0
DELIMS = "|"
FIELDS = UserID,UserType,System,EventType,EventID,Subject,SessionID,SrcAddr,EventStatus,TimeStamp,AdditionalData,DeviceID,DestSrcAddr

[data_filter_headers]
REGEX = ^UserID|UserType|System|EventType|EventID|Subject|SessionID|SrcAddr|EventStatus|TimeStamp|AdditionalData|DeviceID|DestSrcAddr
DEST_KEY = queue
FORMAT = nullQueue

 

I hope this helps!!!

SplunkDash
Motivator

Hello,

Thank you so much for your quick response. Are there any ways we can fix it using indexed time field extraction or without using Transform.conf file?

0 Karma

VatsalJagani
SplunkTrust
SplunkTrust

Your configuration for that seems correct. Try checking the splunkd error and warning logs.
If that doesn't help open a case with Splunk and see if they can help!!

0 Karma

SplunkDash
Motivator

Hello,

Thank you so much you all.  Just wonder, is it possible to use the pattern of like TEST\3eraa2|Employee|  as an event breaking clause? Thank you again.

0 Karma

VatsalJagani
SplunkTrust
SplunkTrust

You can, but you don't need it.

Each of your events is in the new line, so you can just use simply: 

SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)

This is easier and better. 

SplunkDash
Motivator

Hello,

Thank you again. Agree and I used that way as you mentioned. But, thought, if I use like TEST\3eraa2|Employee, then it may give be 5 events instead of 7.

 

0 Karma

VatsalJagani
SplunkTrust
SplunkTrust

No, it (changing LINE_BREAKER) shouldn't make any difference as you are using INDEXED_EXTRACTION.

 

------
Upvote would be appreciated!!!

0 Karma

SplunkDash
Motivator

Hello,

Do you think following props is a good approach, as I am getting exactly 5 events using this props. Any feedback on it will be highly appreciated. Thank you

 

[sourcetype]

SHOULD_LINEMERGE=false

LINE_BREAKER=([\r\n]+)DS\\

CHARSET=UTF-8

TIME_PREFIX=\|\d{2}\|

TIME_FORMAT=%Y%m%d%H%M%S

MAXIMUM_TIMESTAMP_LOOKAHEAD=14

HEADER_FIELD_LINE_NUMBER=1

TRUNCATE=2000

 

0 Karma

VatsalJagani
SplunkTrust
SplunkTrust

LINE_BREAKER=([\r\n]+)DS\\

  • Why DS?
  • Are you sure all lines will start with DS?
0 Karma

SplunkDash
Motivator

Oh Sorry, you are right, it's TEST\ ....thank you and should be  ...start of each event, is it now makes sense to use this props instead.

 

LINE_BREAKER=([\r\n]+)TEST\\

0 Karma
Get Updates on the Splunk Community!

How to Monitor Google Kubernetes Engine (GKE)

We’ve looked at how to integrate Kubernetes environments with Splunk Observability Cloud, but what about ...

Index This | How can you make 45 using only 4?

October 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this ...

Splunk Education Goes to Washington | Splunk GovSummit 2024

If you’re in the Washington, D.C. area, this is your opportunity to take your career and Splunk skills to the ...