Getting Data In

Two different TIME_PREFIX, one includes json formatted events

johnward4
Communicator

Hello,

I'm trying to create a working props/transforms to separate standard events from json formatted logs (by filtering/resetting the json logs to their own sourcetype).  Here's what I've tried so far and I am able to do most of what I want with the exception of timestamp recognition of the json events..  The below trims my json event headers only and filters/resets the json events to their own separate sourcetype.  Since the header is trimmed splunk is doing a great job auto extracting my json field value pairs.  I'm looking for help on getting the timestamp or _time value to match my json field "log_time".  

PROPS.conf

 

 

 

[mainlog]
MAX_TIMESTAMP_LOOKAHEAD = 25
TIME_PREFIX = (?=[20])|log_time:
SEDCMD-remove-jsonheader = s/^[0-9T\:Z]*.*?\s*{/{/g
TRANSFORMS-set_sourcetype = example_json

[mainlog:json]
TIME_PREFIX = log_time:
#TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3NZ
MAX_TIMESTAMP_LOOKAHEAD = 3
INDEXED_EXTRACTIONS = json

 

 

 


TRANSFORMS.conf

 

 

 

[example_json]
REGEX = \{\"json\"\:
FORMAT = sourcetype::mainlog:json
DEST_KEY = MetaData:Sourcetype

 

 

 



sample log:

 

 

 

2023-08-21 11:59:10 TRACE [pool-12-thread-1] c.a.l.m.e.AbstractElasticSearchBatch$ElasticSearchBatch [Slf4jLogging.scala:13] Deadline time left is 302ms and record count is 72
2023-08-21 11:11:41 TRACE [pool-11-thread-1] c.a.l.m.e.AbstractElasticSearchBatch$ElasticSearchBatch [Slf4jLogging.scala:13] Indexing {"json":"s3://example/logs/2023/08/21/0111111a-2222-33ff-9e4e-c1a01dfdf448.gz","phase":"ingest","log_time":"2023-08-21T15:11:31.073Z","tick":"7777777777","id":"0111111a-2222-33ff-9e4e-c1a01dfdf448","source_time":"2023-08-21T11:11:25Z","status":"submitted","client":"555555","environment":"test","category":"changestream","account":"9","level":7}

 

 

 

 

Labels (2)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

The regular expression used in TIME_PREFIX must match the data, which is not the case in the example.  Try these settings.

[mainlog:json]
TIME_PREFIX = log_time":"
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N%Z
MAX_TIMESTAMP_LOOKAHEAD = 24
INDEXED_EXTRACTIONS = json

Also, the "Z" in the timestamp is a time zone abbreviation which should be represented as "%Z" in TIME_FORMAT.

---
If this reply helps you, Karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

IM Landing Page Filter - Now Available

We’ve added the capability for you to filter across the summary details on the main Infrastructure Monitoring ...

Dynamic Links from Alerts to IM Navigators - New in Observability Cloud

Splunk continues to improve the troubleshooting experience in Observability Cloud with this latest enhancement ...

Detecting Remote Code Executions With the Splunk Threat Research Team

WATCH NOWRemote code execution (RCE) vulnerabilities pose a significant risk to organizations. If exploited, ...