Splunk Enterprise

Why are events in the futures coming from a UF (File Monitor)?

Hutch
Path Finder

Hey Everyone,

We are currently running into an issue with one of our sourcetypes coming in roughly five hours in the future. We have reviewed the appliance and verified that the server is set to the correct time but we are unable to locate any issues on the appliance. We are attempting to modify inputs.conf and props.conf to manually adjust the time. We need the time to come in as central time.

Example of timestamp in log

 

 

{ "Timestamp": "7/1/2022 4:15:28 PM", "MessageTime": "7/1/2022 4:15:31 PM" ..........

 

 

UF local/inputs.conf

 

 

[monitor://D:\loglog\*.log]
disabled=false
followTail=0
index=logindex

 

 

UF local/props.conf

 

 

[source::....log]
sourcetype=logsourcetype
TZ=America/Chicago
Time_PREFIX=Timestamp":

 

 

 

I have attempted to modify local/inputs.conf but reverted back to original one above.

 

 

[monitor://D:\loglog\*.log]
disabled=false
followTail=0
TZ=America/Chicago
index=logindex

 

 

I have also review system/local to ensure that there is nothing in there as well. Any recommendations? We have been troubleshooting this timing issue for a few month now and I would like to finally find a way to resolve it on Splunk side as we appear to be unable to resolve it server side.

Labels (1)
0 Karma

Hutch
Path Finder

We are not having an issue with the timestamp only with timing. The data is being parsed correctly and the time format is correct within the parsed data. The data is just coming in 5 hours in the future. Adding the TZ to inputs.conf did not resolve the issue.

0 Karma

PickleRick
SplunkTrust
SplunkTrust

Adding TZ to inputs.conf couldn't have helped because that setting doesn't apply there. It's just an empty statement ignored by splunk.

Hutch
Path Finder

When you were referring to setting TZ at the input level. We’re you referring to the host time or a splunk .conf file?

0 Karma

PickleRick
SplunkTrust
SplunkTrust

I mean that putting "TZ" setting into inputs.conf will not do anything because the time parsing is not performed at that stage.

Hutch
Path Finder

Correct, it should go in props.conf how I have it currently. I also attempted to change it to a different TZ to see if I get different results. I also am encountering a similar issue on a HF so I believe it is a user error on my part, but I just can’t locate what I am doing wrong.

0 Karma

PickleRick
SplunkTrust
SplunkTrust

OK. What is your architecture and where are you putting those props? If you change the TZ setting and there is no change in parsed timestamp it means that it's not working, is it?

Most of the settings in props.conf (including time parsing - that's what interests us here) for a given kind of events should be set on the first "heavy" (indexer or heavy forwarder) component in event's path.

Hutch
Path Finder

For the UF I am modifying props.conf that is located in $splunk_home/apps/app_name/local/props.conf. In defaults the APP defined TZ=UTC.

 

[source::....log]
sourcetype = logsourcetype
TZ = America/Chicago
TIME_FORMAT = %-m/%e/%Y %l:%M:%S %p
Time_PREFIX = Timestamp":

[host::hostname01]
sourcetype = mysourcetype
TZ = America/Chicago
TIME_FORMAT = %-m/%e/%Y %l:%M:%S %p
TIME_PREFIX = Timstamp":

 

So you think I need to change the TIME_PREFIX to what is seen below?

TIME_PREFIX=  "Timestamp":

 

The app has  in the default props.conf

TIME_PREFIX = Timstamp":
0 Karma

PickleRick
SplunkTrust
SplunkTrust

As I said before - UF does not perform timestamp parsing so any settings regarding this aspect that you put on UF will have no effect.

You should put those settings on the splunk component you're sending your events to from the UF (indexers of HFs)

Hutch
Path Finder

After looking into our environment. I have located that the UF is sending events to a HF which is sending the data to the indexers. I have had created a new stanza in $splunk_home/etc/system/local/props.conf on the HF and the indexers, performing restarts and verifying permissions. The new stanza that was created is shown below. 

 

[host::hostname]
TZ = America/Chicago

 

Do I need to add the additional props.conf info that we have discussed prior?

0 Karma

PickleRick
SplunkTrust
SplunkTrust

Yes, If Splunk doesn't recognize time format and placement within the event, it will not parse it. And with a non-standard setup it will most probably not be able to find it. So you need the TIME_PREFIX and TIME_FORMAT on your HF.

Hutch
Path Finder

Located the issue! It appears that there was an additional app that was also attempting to change the time. I was also using the wrong TZ. I ended up removed the TZ=America/Chicago from system/local/props.conf and putting TZ=UTC in the app instead which resolved the issue. 

0 Karma

PickleRick
SplunkTrust
SplunkTrust

Your timestamp prefix seems wrong (there is another space and quotation mark before the timestamp itself). That's one. Two - it's usually good to define timestamp format. Don't waste splunk time trying to guess format. Three - you don't specify TZ on input level. It won't work.

Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

ICYMI - Check out the latest releases of Splunk Edge Processor

Splunk is pleased to announce the latest enhancements to Splunk Edge Processor.  HEC Receiver authorization ...

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...