Can I fix timestamping while routing to new sourcetypes?

[tkw03]

Hello

I built an app that routes data to specific sourcetypes using transforms and regex while also trying to get the timestamping correct. Pretty basic setup:

props.conf

[ncipher]
SHOULD_LINEMERGE=false
NO_BINARY_CHECK=true
TRANSFORMS-sourcetye_routing = mySourcetype_ncipher_hardserver, mySourcetype_ncipher_hsglue

[ncipher:hardserver]
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
TIME_FORMAT = %Y-%m-%d %H:%M:%S
TIME_PREFIX = ]:\s
category = Custom
description = nCipher Timestamped Logs
disabled = false
pulldown_type = true

[ncipher:hsglue]
DATETIME_CONFIG = CURRENT
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
category = Custom
description = nCipher Bad timestamped logs get ingestion timestamp
disabled = false
pulldown_type = true

transforms.conf

[mySourcetype_ncipher_hardserver]
DEST_KEY = MetaData:Sourcetype
REGEX = \shardserver\[
FORMAT = sourcetype::ncipher:hardserver

[mySourcetype_ncipher_hsglue]
DEST_KEY = MetaData:Sourcetype
REGEX = \shsglue\:
FORMAT = sourcetype::ncipher:hsglue

data sample

Feb 24 02:07:36 nethsm hardserver[1516]: 2021-02-24 02:07:36: nFast server: Notice: CreateClient (v1) pid: 17267, process name: /opt/nfast/bin/nfcp
Feb 24 02:37:36 nethsm hardserver[1516]: 2021-02-24 02:37:36: nFast server: Notice: CreateClient (v1) pid: 18393, process name: /opt/nfast/bin/nfcp
Feb 24 02:38:03 nethsm hsglue: warrant DC11-1AB2-3456 loaded
Feb 24 02:39:30 nethsm hsglue: nohup: ignoring input
Feb 24 02:40:37 nethsm hardserver[1516]: 2021-02-24 02:40:37: nFast server: Notice: CreateClient (v1) pid: 18394, process name: /opt/nfast/bin/nfcp
Feb 24 02:41:30 nethsm hsglue: Started hardserver at pid 1516

What Im trying to accomplish is to send all of the records with "hardserver" which has well formatted timestamps in the records to go to ncipher:hardserver and the "hsmglue" records to go to ncipher:hsglue and get the CURRENT time as timestamp.

On test ingestion the recordfs split into the correct sourcetype HOWEVER timestamping didnt work for either which Im trying to solve.

Any ideas what might be happening?

Thanks for the thoughts!

[venkatasri]

Hi,

Reason being the order of Splunk pipelines execution (Typing) transforms.conf works only after timestamp extraction (Merging) completes. See Diagram 3 to understand the order of stanzas in each pipeline.

See the order here - Community:HowIndexingWorks - Splunk Wiki

You can use _indextime for hsglue sourcetype during search time assuming there are no major delays in indexing the data from forwarding layer.

Query: index=main sourcetype="ncipher:hsglue" | rename _indextime as _time

========================

Upvote if it helps!