All Apps and Add-ons

Why are milliseconds not being parsed in cluster environment - Puppet Enterprise 7.0.1?

freedomson
Explorer

Splunk setup
- Cluster environment
- Splunk Enterprise 7.0.1
- Centos 7

Set up props.conf at index master (./etc/master-apps/_cluster/local/props.conf):

[my_source_type]
CHARSET=UTF-8
SHOULD_LINEMERGE=false
TIME_FORMAT=%s%3N
TIME_PREFIX=start_time"*.:*."
TRUNCATE=60000
disabled=false
MAX_TIMESTAMP_LOOKAHEAD=13

Raw event example:

{"metadata":{"start_time":"150479000","delivered_by":"CDN"}}

But events never get assigned a millisecond value:

alt text

When using the upload utility Splunk is able to parse timestamp correctly with the same setup:

alt text

Any ideas?

Thank you.

Related:

https://answers.splunk.com/answers/683842/please-help-me-identify-why-splunk-is-omitting-ext.html#an...

0 Karma

freedomson
Explorer

UPDATE:
After further debugging the reason for milliseconds not appear was very simple:

I was using a transforms.conf in the indexer with the format: DEST_KEY = _meta
Changed it to WRITE_META = true and all fine no need to force local processing anymore.

You may still read below for awareness...


I had the same issue on a non structured source also:

Splunk Enterprise
Version: 7.1.0
Build: 2e75b3406c5b
Source file: source::/var/log/springboot/dailyLogFile.log
Mask: [INFO ] 2019-02-01 11:02:13.178 ...

After trying for a couple of hours to get miliseconds parsed correctly the solution I found was to set on the Splunk Universal Forwarders property force_local_processing to true on

/opt/splunkforward/system/local/props.conf

[source::/var/log/springboot/dailyLogFile.log]
# 2019-02-01 11:02:13.178
# TIME_FORMAT=%Y-%m-%d %H:%M:%S.%3N -> It was proven not needed as datetime.xml seems to cover it ???
# TIME_PREFIX=^\[\w*\s*\]\s -> It was proven not needed as datetime.xml seems to cover it ???
 =true

You need to restart Splunk Universal Forwarder to changes to take place:
/opt/splunkforwarder/bin/splunk restart

I have tried to configure the setting at the indexers using multiple configuration without sucess.
Ultimatly I found the answer in https://docs.splunk.com/Documentation/Splunk/7.2.3/Admin/Propsconf

Maybe Splunk should open the source code it would make much easier for us to debug this.

IMPORTANT: After forcing local processing I had also to localy add my trasforms.conf setup as they would no longer be parsed at the indexers.

0 Karma

freedomson
Explorer

I manage to fix this by not setting up props.conf at index master but at each individual splunk universal forwarder of the cluster.

/opt/splunkforwarder/etc/system/local/props.con

[example_sourcetype]
INDEXED_EXTRACTIONS = json
CHARSET=UTF-8
SHOULD_LINEMERGE=false
TIME_FORMAT=%s%3N
TIME_PREFIX=start_time"*.:*."
TRUNCATE=60000
disabled=false
MAX_TIMESTAMP_LOOKAHEAD=13

I bumped into this when reading https://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Routeandfilterdatad
On the above page there is a reference for caveats for routing and filtering structured data namely:

The forwarded data must arrive at the indexer already parsed.

This was the trigger to set this at the forwarder and not on the indexer.

0 Karma
Get Updates on the Splunk Community!

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

The Splunk Success Framework: Your Guide to Successful Splunk Implementations

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...