Splunk Search

In Windows Custom Events, why are the bigger JSON events while using REGEX getting truncated?

Venkat_16
Contributor

In our environment, the application writes logs into Windows Events in JSON format under Message section.
We need to segregate these application logs and remove the default windows metadata/envelope around it.
Please see my config below:

inputs.conf

[WinEventLog://Application]
disabled = 0
start_from = oldest
current_only = 0
checkpointInterval = 5
sourcetype = my_temp_windows_sourcetype
index=my_index

props.conf

[my_temp_windows_sourcetype]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n](?=\d{2}/\d{2}/\d{2,4} \d{2}:\d{2}:\d{2} [aApPmM]{2}))
TRANSFORMS-sourcetype_raw = my_windows_event_default,my_windows_event_sourcetype,my_windows_event_raw

transforms.conf

[my_windows_event_default]
REGEX = .
FORMAT = sourcetype::WinEventLog:Application
DEST_KEY = MetaData:Sourcetype

[my_windows_event_sourcetype]
REGEX = ImportantKeyWord
FORMAT = sourcetype::my_new_sourcetype
DEST_KEY = MetaData:Sourcetype

[my_windows_event_raw]
REGEX = Message=(.*ImportantKeyWord.*)$
FORMAT = $1
DEST_KEY = _raw

This works fine when the length of the JSON Message is small (<3000 characters).
However, for bigger JSON, events are getting truncated.
We also see a pattern here, events are truncated at same length (approx 3800-3900).
I doubt if the REGEX = Message=(.*ImportantKeyWord.*)$ here might be causing the truncation?
Because, if we try with SED in props.conf, events are not getting truncated, however, that is not I want.

SEDCMD-drop = s/(?ims)[0-9][0-9]\/[0-9][0-9]\/[0-9][0-9][0-9][0-9] [0-9][0-9]\:[0-9][0-9]\:[0-9][0-9].*[\r\n].*Message\=//g

I want only events with ImportantKeyWord in the Message to be re-written as _raw

Any suggestions welcome.

0 Karma
1 Solution

Venkat_16
Contributor

We raised Splunk Ticket and got this resolved by adding "LOOKAHEAD"

[my_windows_event_raw]
LOOKAHEAD = 10000
REGEX = Message=(.*ImportantKeyWord.*)$
FORMAT = $1
DEST_KEY = _raw

https://docs.splunk.com/Documentation/Splunk/latest/Admin/Transformsconf#GLOBAL_SETTINGS

LOOKAHEAD = <integer>
* NOTE: This option is valid for all index time transforms, such as
  index-time field creation, or DEST_KEY modifications.
* Optional. Specifies how many characters to search into an event.
* Defaults to 4096.
* You may want to increase this value if you have event line lengths that
  exceed 4096 characters (before linebreaking).

View solution in original post

0 Karma

Venkat_16
Contributor

We raised Splunk Ticket and got this resolved by adding "LOOKAHEAD"

[my_windows_event_raw]
LOOKAHEAD = 10000
REGEX = Message=(.*ImportantKeyWord.*)$
FORMAT = $1
DEST_KEY = _raw

https://docs.splunk.com/Documentation/Splunk/latest/Admin/Transformsconf#GLOBAL_SETTINGS

LOOKAHEAD = <integer>
* NOTE: This option is valid for all index time transforms, such as
  index-time field creation, or DEST_KEY modifications.
* Optional. Specifies how many characters to search into an event.
* Defaults to 4096.
* You may want to increase this value if you have event line lengths that
  exceed 4096 characters (before linebreaking).
0 Karma

Rob2520
Communicator

Hi @Venkat_16. I am facing similar kind of issue and adding LOOKAHEAD setting didn't fix. In my case props and transforms are on heavy forwarder. Does these conf settings needs to go on Indexers as well?

0 Karma

Venkat_16
Contributor

@cpetterborg - these json are not multiline, however will try and keep you posted.

0 Karma

deepashri_123
Motivator

Hey@Venkat_16,

You can try adding this parameter in props.conf
TRUNCATE = 0

Let me know if this helps!!

0 Karma

Venkat_16
Contributor

No luck 😞

0 Karma

Venkat_16
Contributor

Default Truncate limit is 10k, our max length is 7k. Anyways will try and keep you posted.

0 Karma

cpetterborg
SplunkTrust
SplunkTrust

Have you tried setting MAX_EVENTS in props.conf?

0 Karma

Venkat_16
Contributor

No Luck 😞

0 Karma
Get Updates on the Splunk Community!

Database Performance Sidebar Panel Now on APM Database Query Performance & Service ...

We’ve streamlined the troubleshooting experience for database-related service issues by adding a database ...

IM Landing Page Filter - Now Available

We’ve added the capability for you to filter across the summary details on the main Infrastructure Monitoring ...

Dynamic Links from Alerts to IM Navigators - New in Observability Cloud

Splunk continues to improve the troubleshooting experience in Observability Cloud with this latest enhancement ...