PROPS Configuration for text file with header

[SplunkDash]

Hello,

I have a text source file with header. Some sample events (first line is a header) and props that I wrote given below.

My props is working ok, except it breaks the events at TEST\2qw123|Employee, TEST\3eraa2|Employee, TEST\87xaqw|Employee, at Obj.BasePage.Page, TEST\m69xcb, at Obj.BasePage.Page, and TEST\7yxccd|Employee  instead of breaking events at  TEST\2qw123|Employee, TEST\3eraa2|Employee, TEST\87xaqw|Employee, TEST\m69xcb, and TEST\7yxccd|Employee .  So from following sample events, I should have 5 events , but getting 7 events. Any help will be highly appreciated. Thank you. 

UserID|UserType|System|EventType|EventID|Subject|SessionID|SrcAddr|EventStatus|TimeStamp|AdditionalData|DeviceID|DestSrcAddr
TEST\2qw123|Employee|COM|TESTUSER|NTINCheckKCase|089524234|ybzjlie3d4ayr1i2|10.212.48.121|00|20220217122935|Case Information request: (Case-170) - 201612-30|mct0ma01ma4352855|10.219.174.222
TEST\3eraa2|Employee|COM|TESTUSER|NTINCheckKCase|046453942|ybzjlie3d4ayr1i2|10.212.48.121|00|20220217123142|Case Information request: (Case -85) - 201912-30|mct0ma01ma4352855|10.219.174.222
TEST\87xaqw|Employee|COM|SYSTEM|SystemMsg||zsod0mvomcelp3hvln5smm1u|10.216.22.17|01|20220217124743|Type:'error'; Ref:'Case/CaseInventory.aspx?Query=true&Scope=ServiceWide'; Msg: experienced <br>Source: App_Web_pc<br>Message: Object reference not set to an instance of an object.<br> /Case/CaseInventory.aspx<br>Trace: at Case.CaseInventory()
at Obj.BasePage.Page_Load(Object sender, EventArgs e)<br><br>Please try to login again.|mct0ma01ma4382154|10.210.174.221
TEST\m69xcb|Employee|COM|SYSTEM|SystemMsg||z0ae3c25zggbzx5p|10.215.173.231|01|20220217130933|Type:'error'; Ref:'Case/CaseInventory.aspx?Query=true&Scope=ServiceWide'; Msg: experienced a error:<br><br>Source: App_Web_pcf3kniw<br>Message: Object reference not set to an instance of an object.<br> /Case/CaseInventory.aspx<br>Trace: at Case.CaseInventory.page_load3()
at Obj.BasePage.Page_Load(Object sender, EventArgs e)<br><br>Please try to login again.|mct0ma01ma4353159|10.210.174.221
TEST\7yxccd|Employee|COM|TESTUSER|NTINCheckKCase|008422123|zggbzx5pzgnw1nih|10.215.173.231|00|20220217131108|Case Information request: (Case -24) - 202112-30|mct0ma1ma4353159|10.210.174.221

[sourcename]

SHOULD_LINEMERGE=false

LINE_BREAKER=([\r\n]+)

INDEXED_EXTRACTIONS=psv

MAX_TIMESTAMP_LOOKAHEAD=14

HEADER_FIELD_LINE_NUMBER=1

TIME_FORMAT=%Y%m%d%H%M%S

TIMESTAMP_FIELDS=TimeStamp

TRUNCATE=2000

[PickleRick]

Your data is inconsistent with the definition. You have header specifying some fields and then you have two events with not enough data to fill those fields,

[VatsalJagani]

Try using search-time field extraction instead of Index time (INDEXED_EXTRACTIONS) with below configurations:

props.conf

[sourcename]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)
TRUNCATE = 2000
TRANSFORMS-filter_events = data_filter_headers
TIME_PREFIX = [^|]*|[^|]*|[^|]*|[^|]*|[^|]*|[^|]*|[^|]*|[^|]*|[^|]*|
MAX_TIMESTAMP_LOOKAHEAD = 20
TIME_FORMAT = %Y%m%d%H%M%S
REPORT-headers = data_headers

transforms.conf

[data_headers]
CLEAN_KEYS = 0
DELIMS = "|"
FIELDS = UserID,UserType,System,EventType,EventID,Subject,SessionID,SrcAddr,EventStatus,TimeStamp,AdditionalData,DeviceID,DestSrcAddr

[data_filter_headers]
REGEX = ^UserID|UserType|System|EventType|EventID|Subject|SessionID|SrcAddr|EventStatus|TimeStamp|AdditionalData|DeviceID|DestSrcAddr
DEST_KEY = queue
FORMAT = nullQueue

I hope this helps!!!

[SplunkDash]

Hello,

Thank you so much for your quick response. Are there any ways we can fix it using indexed time field extraction or without using Transform.conf file?

[VatsalJagani]

Your configuration for that seems correct. Try checking the splunkd error and warning logs.
If that doesn't help open a case with Splunk and see if they can help!!

[SplunkDash]

Hello,

Thank you so much you all.  Just wonder, is it possible to use the pattern of like TEST\3eraa2|Employee|  as an event breaking clause? Thank you again.

[VatsalJagani]

You can, but you don't need it.

Each of your events is in the new line, so you can just use simply: 

SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)

This is easier and better. 

[SplunkDash]

Hello,

Thank you again. Agree and I used that way as you mentioned. But, thought, if I use like TEST\3eraa2|Employee, then it may give be 5 events instead of 7.

[VatsalJagani · Accepted solution]

No, it (changing LINE_BREAKER) shouldn't make any difference as you are using INDEXED_EXTRACTION.

------
Upvote would be appreciated!!!

[SplunkDash]

Hello,

Do you think following props is a good approach, as I am getting exactly 5 events using this props. Any feedback on it will be highly appreciated. Thank you

[sourcetype]

SHOULD_LINEMERGE=false

LINE_BREAKER=([\r\n]+)DS\\

CHARSET=UTF-8

TIME_PREFIX=\|\d{2}\|

TIME_FORMAT=%Y%m%d%H%M%S

MAXIMUM_TIMESTAMP_LOOKAHEAD=14

HEADER_FIELD_LINE_NUMBER=1

TRUNCATE=2000

[VatsalJagani]

LINE_BREAKER=([\r\n]+)DS\\

• Why DS?
• Are you sure all lines will start with DS?

[SplunkDash]

Oh Sorry, you are right, it's TEST\ ....thank you,  and should be  ...start of each event, is it now makes sense to use this props instead.

LINE_BREAKER=([\r\n]+)TEST\\