Universal Forwarder failing to forward logs to Splunk Enterprise.

[ucorral]

Hello guys!,

I have a month trying to forward my logs from iMacs using the UF with the following format:

Resources,line,data,process
2023-09-30T06:35:02,"Scanned disks....... "
2023-09-30T06:35:02,User: ......
2023-09-30T06:35:02,...........
2023-09-30T06:35:02,............
2023-09-30T06:35:02,Time of completion: ..........

but when the log get into Splunk it only shows the first row:

Resources,line,data,process

and the rest of the log reaches splunk 6 hours later.

I've added the following rule in props.conf but it still failling.

path: /Applications/SplunkForwarder/etc/system/local/props.conf 

[name_of_my_sourcetype]

CHARSET=UTF-8
TIME_FORMAT=%Y-%m-%dT%H:%M:%S,
TIME_PREFIX=^
LINE_BREAKER=([\r\n]+)
NO_BINARY_CHECK=true
SHOULD_LINEMERGE=true
TZ=America/Mexico_City
disabled=false

Every change I made I always restart the splunk forwarder using ./splunk restart

I have no access to the Splunk server (SSH) but if needed I could try to make some configurations but I do not where.

[PickleRick]

It looks as if you were getting your events from the file but getting the date parsed in the wrong timezone. Most of the settings you showed belong on the receiving Splunk instance, not the forwarder.

And don't set SHOULD_LINEMERGE to true. This one should almost never be set to true.

Also indexed extractions should not be overused. And this case doesn't seem to be one justifying use of indexed extractions (here I disagree with @gcusello )

[ucorral]

Hello @PickleRick 

Thanks for the heads up, I'll delete them from the props.conf, however the information is still reaching 6 hours late, What could be the best recommendation?

Thanks,

[PickleRick]

See the https://docs.splunk.com/Documentation/Splunk/latest/Data/Applytimezoneoffsetstotimestamps article to understand how Splunk applies timezone information.

It could be done in several different places, most probably you'd want to set the TZ on the forwarder so that it doesn't interfere with other components' settings.

[gcusello]

Hi @ucorral,

use INDEXED_EXTRACTIONS = csv.

You could tray to manually upload your file using the GUI that guides you in the best sourcetype definition.

Ciao.

Giuseppe

[ucorral]

@gcusello I added the INDEXED_EXTRACTIONS=csv, then I restarted the splunk daemon.

[my_custom_sourcetype]
CHARSET=UTF-8
INDEXED_EXTRACTIONS=csv
TIME_FORMAT=%Y-%m-%dT%H:%M:%S,
TIME_PREFIX=^
LINE_BREAKER=([\r\n]+)
NO_BINARY_CHECK=true
SHOULD_LINEMERGE=true
TZ=America/Mexico_City
disabled=false

But I continue receiving logs from 6 hours ago. 

Copying the last log received in Splunk

9/30/23
6:35:02.000 AM	
2023-09-30T06:35:02,Time of completion: 00:35:02  ***** 0 sec (00:00:00)
host = ******* source = /var/log/****/*****log.****.txtsourcetype = my_custom_sourcetype

as you can see the last log have received at 06:35:02am -> but was created at 00:35:02 of my current time in Mexico City. At the moment no more logs showed in splunk 😞 

But now I realized the logs come split for some reason.

[gcusello]

Hi @ucorral,

could you share some sample of your logs?

Ciao.

Giuseppe

[ucorral]

Hi @gcusello 

I can explain with some screenshots the problem:

The logs are related with an Antivirus (policies, detected viruses and so on), in the first image you can see the file was created at 00:35:00 , this is an Antivirus Scan

This is the content of the file:

....but as you can see timestamp shows 06:35 (That's why I added the TZ option in the props.conf)

Finally this is an image of the Splunk search, the _time column is aligned with the timestamp with the log content 

The register was supposed to arrive at 00:35, but was entered at 06:35. (6 hours after the scan)

The hour is set at GMT-6. I tried to look the AV settings to set the time at GMT-6 but it does not have that option.

[gcusello]

Hi @ucorral,

let me understand: Splunk read the correct timestamp from the log, did you configure the Timezone in the props.conf (https://docs.splunk.com/Documentation/Splunk/9.1.1/Data/Applytimezoneoffsetstotimestamps)?

Then, did you configure the timezone for the user in GUI [<your_user_name> > Preferences]?

Ciao.

Giuseppe

[ucorral]

Hi @gcusello , That's correct I used the GUI data extraction feature to obtain the parameters, and the output(showed above) was applied in my props.conf file 😞

[gcusello]

Hi @ucorral,

it's really strange, because this seems to be a csv file and you don't have INDEXED_EXTRACTIOS = csv in props.conf of the Universal Forwarder.

Ciao.

Giuseppe

[ucorral]

Hello @gcusello ,

I've already added to the stanza but still failling 😞

[gcusello]

Hy @ucorral,

did you tried to create the sourcetype by GUI, manually uploading the file usng the Add Data feature?

Ciao.

Giuseppe