- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello,
I have a text source file with header. Some sample events (first line is a header) and props that I wrote given below.
My props is working ok, except it breaks the events at TEST\2qw123|Employee, TEST\3eraa2|Employee, TEST\87xaqw|Employee, at Obj.BasePage.Page, TEST\m69xcb, at Obj.BasePage.Page, and TEST\7yxccd|Employee instead of breaking events at TEST\2qw123|Employee, TEST\3eraa2|Employee, TEST\87xaqw|Employee, TEST\m69xcb, and TEST\7yxccd|Employee . So from following sample events, I should have 5 events , but getting 7 events. Any help will be highly appreciated. Thank you.
UserID|UserType|System|EventType|EventID|Subject|SessionID|SrcAddr|EventStatus|TimeStamp|AdditionalData|DeviceID|DestSrcAddr
TEST\2qw123|Employee|COM|TESTUSER|NTINCheckKCase|089524234|ybzjlie3d4ayr1i2|10.212.48.121|00|20220217122935|Case Information request: (Case-170) - 201612-30|mct0ma01ma4352855|10.219.174.222
TEST\3eraa2|Employee|COM|TESTUSER|NTINCheckKCase|046453942|ybzjlie3d4ayr1i2|10.212.48.121|00|20220217123142|Case Information request: (Case -85) - 201912-30|mct0ma01ma4352855|10.219.174.222
TEST\87xaqw|Employee|COM|SYSTEM|SystemMsg||zsod0mvomcelp3hvln5smm1u|10.216.22.17|01|20220217124743|Type:'error'; Ref:'Case/CaseInventory.aspx?Query=true&Scope=ServiceWide'; Msg: experienced <br>Source: App_Web_pc<br>Message: Object reference not set to an instance of an object.<br> /Case/CaseInventory.aspx<br>Trace: at Case.CaseInventory()
at Obj.BasePage.Page_Load(Object sender, EventArgs e)<br><br>Please try to login again.|mct0ma01ma4382154|10.210.174.221
TEST\m69xcb|Employee|COM|SYSTEM|SystemMsg||z0ae3c25zggbzx5p|10.215.173.231|01|20220217130933|Type:'error'; Ref:'Case/CaseInventory.aspx?Query=true&Scope=ServiceWide'; Msg: experienced a error:<br><br>Source: App_Web_pcf3kniw<br>Message: Object reference not set to an instance of an object.<br> /Case/CaseInventory.aspx<br>Trace: at Case.CaseInventory.page_load3()
at Obj.BasePage.Page_Load(Object sender, EventArgs e)<br><br>Please try to login again.|mct0ma01ma4353159|10.210.174.221
TEST\7yxccd|Employee|COM|TESTUSER|NTINCheckKCase|008422123|zggbzx5pzgnw1nih|10.215.173.231|00|20220217131108|Case Information request: (Case -24) - 202112-30|mct0ma1ma4353159|10.210.174.221
[sourcename]
SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n]+)
INDEXED_EXTRACTIONS=psv
MAX_TIMESTAMP_LOOKAHEAD=14
HEADER_FIELD_LINE_NUMBER=1
TIME_FORMAT=%Y%m%d%H%M%S
TIMESTAMP_FIELDS=TimeStamp
TRUNCATE=2000
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

No, it (changing LINE_BREAKER) shouldn't make any difference as you are using INDEXED_EXTRACTION.
------
Upvote would be appreciated!!!
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

Your data is inconsistent with the definition. You have header specifying some fields and then you have two events with not enough data to fill those fields,
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

Try using search-time field extraction instead of Index time (INDEXED_EXTRACTIONS) with below configurations:
props.conf
[sourcename]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)
TRUNCATE = 2000
TRANSFORMS-filter_events = data_filter_headers
TIME_PREFIX = [^|]*|[^|]*|[^|]*|[^|]*|[^|]*|[^|]*|[^|]*|[^|]*|[^|]*|
MAX_TIMESTAMP_LOOKAHEAD = 20
TIME_FORMAT = %Y%m%d%H%M%S
REPORT-headers = data_headers
transforms.conf
[data_headers]
CLEAN_KEYS = 0
DELIMS = "|"
FIELDS = UserID,UserType,System,EventType,EventID,Subject,SessionID,SrcAddr,EventStatus,TimeStamp,AdditionalData,DeviceID,DestSrcAddr
[data_filter_headers]
REGEX = ^UserID|UserType|System|EventType|EventID|Subject|SessionID|SrcAddr|EventStatus|TimeStamp|AdditionalData|DeviceID|DestSrcAddr
DEST_KEY = queue
FORMAT = nullQueue
I hope this helps!!!
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello,
Thank you so much for your quick response. Are there any ways we can fix it using indexed time field extraction or without using Transform.conf file?
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

Your configuration for that seems correct. Try checking the splunkd error and warning logs.
If that doesn't help open a case with Splunk and see if they can help!!
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello,
Thank you so much you all. Just wonder, is it possible to use the pattern of like TEST\3eraa2|Employee| as an event breaking clause? Thank you again.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

You can, but you don't need it.
Each of your events is in the new line, so you can just use simply:
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)
This is easier and better.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello,
Thank you again. Agree and I used that way as you mentioned. But, thought, if I use like TEST\3eraa2|Employee, then it may give be 5 events instead of 7.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

No, it (changing LINE_BREAKER) shouldn't make any difference as you are using INDEXED_EXTRACTION.
------
Upvote would be appreciated!!!
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello,
Do you think following props is a good approach, as I am getting exactly 5 events using this props. Any feedback on it will be highly appreciated. Thank you
[sourcetype]
SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n]+)DS\\
CHARSET=UTF-8
TIME_PREFIX=\|\d{2}\|
TIME_FORMAT=%Y%m%d%H%M%S
MAXIMUM_TIMESTAMP_LOOKAHEAD=14
HEADER_FIELD_LINE_NUMBER=1
TRUNCATE=2000
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

LINE_BREAKER=([\r\n]+)DS\\
- Why DS?
- Are you sure all lines will start with DS?
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Oh Sorry, you are right, it's TEST\ ....thank you, and should be ...start of each event, is it now makes sense to use this props instead.
LINE_BREAKER=([\r\n]+)TEST\\
