Getting Data In

Can I fix timestamping while routing to new sourcetypes?

tkw03
Communicator

Hello

I built an app that routes data to specific sourcetypes using transforms and regex while also trying to get the timestamping correct. Pretty basic setup:

props.conf

[ncipher]
SHOULD_LINEMERGE=false
NO_BINARY_CHECK=true
TRANSFORMS-sourcetye_routing = mySourcetype_ncipher_hardserver, mySourcetype_ncipher_hsglue

[ncipher:hardserver]
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
TIME_FORMAT = %Y-%m-%d %H:%M:%S
TIME_PREFIX = ]:\s
category = Custom
description = nCipher Timestamped Logs
disabled = false
pulldown_type = true

[ncipher:hsglue]
DATETIME_CONFIG = CURRENT
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
category = Custom
description = nCipher Bad timestamped logs get ingestion timestamp
disabled = false
pulldown_type = true

transforms.conf

[mySourcetype_ncipher_hardserver]
DEST_KEY = MetaData:Sourcetype
REGEX = \shardserver\[
FORMAT = sourcetype::ncipher:hardserver

[mySourcetype_ncipher_hsglue]
DEST_KEY = MetaData:Sourcetype
REGEX = \shsglue\:
FORMAT = sourcetype::ncipher:hsglue

 

data sample

Feb 24 02:07:36 nethsm hardserver[1516]: 2021-02-24 02:07:36: nFast server: Notice: CreateClient (v1) pid: 17267, process name: /opt/nfast/bin/nfcp
Feb 24 02:37:36 nethsm hardserver[1516]: 2021-02-24 02:37:36: nFast server: Notice: CreateClient (v1) pid: 18393, process name: /opt/nfast/bin/nfcp
Feb 24 02:38:03 nethsm hsglue: warrant DC11-1AB2-3456 loaded
Feb 24 02:39:30 nethsm hsglue: nohup: ignoring input
Feb 24 02:40:37 nethsm hardserver[1516]: 2021-02-24 02:40:37: nFast server: Notice: CreateClient (v1) pid: 18394, process name: /opt/nfast/bin/nfcp
Feb 24 02:41:30 nethsm hsglue: Started hardserver at pid 1516

 

What Im trying to accomplish is to send all of the records with "hardserver" which has well formatted timestamps in the records to go to ncipher:hardserver and the "hsmglue" records to go to ncipher:hsglue and get the CURRENT time as timestamp.

On test ingestion the recordfs split into the correct sourcetype HOWEVER timestamping didnt work for either which Im trying to solve.

 

Any ideas what might be happening?

Thanks for the thoughts!

0 Karma

venkatasri
SplunkTrust
SplunkTrust

Hi,

Reason being the order of Splunk pipelines execution (Typing) transforms.conf works only after timestamp extraction (Merging) completes. See Diagram 3 to understand the order of stanzas in each pipeline.

See the order here - Community:HowIndexingWorks - Splunk Wiki

You can use _indextime for hsglue sourcetype during search time assuming there are no major delays in indexing the data from forwarding layer.

Query: index=main sourcetype="ncipher:hsglue" | rename _indextime as _time

========================

Upvote if it helps!

0 Karma
Get Updates on the Splunk Community!

Dashboards: Hiding charts while search is being executed and other uses for tokens

There are a couple of features of SimpleXML / Classic dashboards that can be used to enhance the user ...

Splunk Observability Cloud's AI Assistant in Action Series: Explaining Metrics and ...

This is the fourth post in the Splunk Observability Cloud’s AI Assistant in Action series that digs into how ...

Brains, Bytes, and Boston: Learn from the Best at .conf25

When you think of Boston, you might picture colonial charm, world-class universities, or even the crack of a ...