Getting Data In

Can I fix timestamping while routing to new sourcetypes?

tkw03
Communicator

Hello

I built an app that routes data to specific sourcetypes using transforms and regex while also trying to get the timestamping correct. Pretty basic setup:

props.conf

[ncipher]
SHOULD_LINEMERGE=false
NO_BINARY_CHECK=true
TRANSFORMS-sourcetye_routing = mySourcetype_ncipher_hardserver, mySourcetype_ncipher_hsglue

[ncipher:hardserver]
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
TIME_FORMAT = %Y-%m-%d %H:%M:%S
TIME_PREFIX = ]:\s
category = Custom
description = nCipher Timestamped Logs
disabled = false
pulldown_type = true

[ncipher:hsglue]
DATETIME_CONFIG = CURRENT
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
category = Custom
description = nCipher Bad timestamped logs get ingestion timestamp
disabled = false
pulldown_type = true

transforms.conf

[mySourcetype_ncipher_hardserver]
DEST_KEY = MetaData:Sourcetype
REGEX = \shardserver\[
FORMAT = sourcetype::ncipher:hardserver

[mySourcetype_ncipher_hsglue]
DEST_KEY = MetaData:Sourcetype
REGEX = \shsglue\:
FORMAT = sourcetype::ncipher:hsglue

 

data sample

Feb 24 02:07:36 nethsm hardserver[1516]: 2021-02-24 02:07:36: nFast server: Notice: CreateClient (v1) pid: 17267, process name: /opt/nfast/bin/nfcp
Feb 24 02:37:36 nethsm hardserver[1516]: 2021-02-24 02:37:36: nFast server: Notice: CreateClient (v1) pid: 18393, process name: /opt/nfast/bin/nfcp
Feb 24 02:38:03 nethsm hsglue: warrant DC11-1AB2-3456 loaded
Feb 24 02:39:30 nethsm hsglue: nohup: ignoring input
Feb 24 02:40:37 nethsm hardserver[1516]: 2021-02-24 02:40:37: nFast server: Notice: CreateClient (v1) pid: 18394, process name: /opt/nfast/bin/nfcp
Feb 24 02:41:30 nethsm hsglue: Started hardserver at pid 1516

 

What Im trying to accomplish is to send all of the records with "hardserver" which has well formatted timestamps in the records to go to ncipher:hardserver and the "hsmglue" records to go to ncipher:hsglue and get the CURRENT time as timestamp.

On test ingestion the recordfs split into the correct sourcetype HOWEVER timestamping didnt work for either which Im trying to solve.

 

Any ideas what might be happening?

Thanks for the thoughts!

0 Karma

venkatasri
SplunkTrust
SplunkTrust

Hi,

Reason being the order of Splunk pipelines execution (Typing) transforms.conf works only after timestamp extraction (Merging) completes. See Diagram 3 to understand the order of stanzas in each pipeline.

See the order here - Community:HowIndexingWorks - Splunk Wiki

You can use _indextime for hsglue sourcetype during search time assuming there are no major delays in indexing the data from forwarding layer.

Query: index=main sourcetype="ncipher:hsglue" | rename _indextime as _time

========================

Upvote if it helps!

0 Karma
Get Updates on the Splunk Community!

Security Highlights: September 2022 Newsletter

 September 2022 The Splunk App for Fraud Analytics (SFA) is now Splunk SupportedUse your existing Splunk ...

Platform Highlights | September 2022 Newsletter

 September 2022 What’s New in 9.0 and How to UpgradeGet a walk through of what is new Splunk Enterprise 9.0 ...

Observability Highlights | September 2022 Newsletter

 September 2022 Splunk Observability SuiteAccess to "Classic" SignalFx Interface Will be Removed on Sept 30, ...