Help with the Timezone conversion?

[varunesh]

Hi All,

Good Day.

Need help in Splunk data receiving.

We have Avamar backup node which is sending the data to splunk is in EST time zone.

The splunk server is configured with the UTC time zone.

The data is being received by splunk which shows the correct time but when the index server parsing the data, thinking it was 4 hours old data and ignoring it. So backup failures are not captured and ticket is not generating for backup failures.

Upon verifiying in splunk side, some of the received data between _time & indextime difference of 4 hours and some of them receving correctly. When the difference time is 4 hours splunk is ignoring the data and not generating ticket for failures.

Note: The search is running for every 15 minutes if we increase the search duration to see last 4 hours then we will receive lot of duplicates.

We have contacted Dell support but they are updating me that backup application just send the data with the MIB file as it when receives the data. It would be the splunk to process the data correctly.

Please help to solve the issue and any recommendation is appreciated.

[varunesh]

Thanks for the reply, can you please let me know how to access the props.conf file? Do I need to capture the configuration file from Avamar or it will be in splunk server? I have access to splunk cloud where we access our jobs. 

Please let me know the procedure to access the file.

[richgalloway]

Since this is Splunk Cloud, you should be able to view the props via the UI (if you have permissions).  Go to Settings->Source types and select the sourcetype used for Avamar data.  Click on the Advanced tab to see all of the settings in one place.

These settings should be in an app that is stored somewhere on-prem, ideally in a source code management system.

[varunesh]

Also please check the below image.

[richgalloway]

This often is caused by an incorrect TIME_PREFIX setting or an incorrect/missing TZ setting in props.conf.  Without a specified time zone, Splunk will assume the event occurred in the system time zone, resulting in events being off by hours.

Please share the props.conf settings for the sourcetype.

[varunesh]

Hi,

I checked my access and dont have permission to view the prop.conf will reach out to my splunk admins.

I have couple of questions

1. If I change the TIME_PREFIX setting in the prop.conf, it applies only to our Avamar configuration or it applies to the entire splunk configuration?

2.  If the setting applies to the entire configuration is there a option to change it only for our Avamar reporting only?

[richgalloway]

It depends on where TIME_PREFIX is placed in the props.conf file.  If it's in the [default] stanza then it will apply to all sourcetypes.  This is not recommended.

The settings should be in (and would only apply to) a specific stanza, either for a sourcetype, source, or host.

[varunesh]

I have received the content of props.conf and I searched for both the keywords TIME_PREFIX & TZ but I found some entries for TIME_PREFIX only.

Please check the attached images and update back.

Even I searched for Avamar keyword but dont find any entries in the conf file and also for logging using syslog I have attached those contents too from the conf file.

Thanks 

[richgalloway]

It's difficult to know if a TIME_PREFIX setting is correct without seeing sample data.  The props.conf stanzas that do not have a TIME_FORMAT setting probably need one.

If you can't find a stanza for Avamar (did you try btool?) then you probably need one.  Every sourcetype should have one.  The search results in your OP should say what sourcetype was used with the Avamar events.  Make sure there are props.conf settings for that sourcetype.

[varunesh]

I found the stanza name "avamar " by providing the sourcetype in the table query. I verified the prop.conf but couldn't find any stanza named avamar.

In the other defined sanza in prop.conf they have used the below for the TIME_PREFIX setting

TIME_PREFIX = \[

As we need to update the TIME_PREFIX setting specific only to our Avamar host, can you please let me know the syntax for the TIME_PREFIX setting to set specific host or IP to the UTC time zone?

Also please share any technote to understand the syntax and other options in TIME_PREFIX setting.

Thanks in advance.

[richgalloway]

The Avamar props may not be in a stanza called "avamar".  It could be in a source-specific or host-specific stanza.

I think it may be a good idea to specify TIME_PREFIX and TZ separately, since TZ is specific to the machine and TIME_PREFIX is specific to the data.

[host::<<host name>>]
TZ = EST

[source::<<Avamar file path>>]
# This setting is an assumption.  Examine sample data to verify it is correct
TIME_PREFIX = \[

Replace words in <<>> with actual values (omitting the brackets).

The tech notes for all config files are in $SPLUNK_HOME/etc/system/README/*.spec

[varunesh]

I have prepared based on your input please confirm.

[avamar]
[host::usxxxx*]
TZ = US/Eastern
TIME_PREFIX = \[

I dont understand the below line
[source::<<Avamar file path>>]

Avamar bkp failure data is being sent from Avamar server to syslog server as when the failures occur , which means in Avamar we configured to send the data to syslog server and splunk avamar data is being indexed from the syslog server, so what is the source I need to mention here?

[richgalloway]

I forgot you're using syslog.  Sorry about that.  In that case, have the syslog server write Avamar data to a different location (how to do this depends on your syslog server).  When Splunk reads from that location, it can associate the appropriate props settings.

Do NOT do this

[host::usxxxx*]
TZ = US/Eastern
TIME_PREFIX = \[

This will tell Splunk to look for a timestamp after a left bracket in ALL data that comes from the usxxxx host.  Unless that host provides a single type of data, that likely to be an incorrect setting.  The TIME_PREFIX setting needs to be associated with a specific source or sourcetype rather than a specific host.