Getting Data In

how can I use log filename to set "host" "date"

mehrdad_2000
Path Finder

Hi I have log files like this:

log.machine03.20200310.bz2
log.machine04.20200310.bz2
log.machine05.20200310.bz2

these files copy each day to /opt , and splunk continuously monitor /opt.
now how can I use log filename to set "host" "date" of logs in splunk?

FYI: in every line of log file only store time NOT date! that's why I need to use date that exist in file name.
e.g.
file name = log.machine05.20200310.bz2
01:00:00 info logmessage
02:00:00 info logmessage
03:00:00 info logmessage
...

Thanks,

0 Karma

manjunathmeti
Champion

You can set host from file name in inputs.conf on forwarder.
inputs.conf

[monitor://<path>]
sourcetype = sourcetype_name
host_regex = [a-zA-Z]+\d+(?!$)

For setting _time from file name use INGET-EVAL. Add below configurations on indexer server.

props.conf

[sourcetype_name]
TRANSFORMS = timestamp_eval

trsansforms.conf

[timestamp_eval]
INGEST_EVAL = _time=strptime(replace(source, "/opt/log\.[\w]+\.", ""), "%Y%m%d.bz2")

Note: INGEST_EVAL is supported in splunk version >= 7.2.0.

0 Karma

mehrdad_2000
Path Finder

Unfortunately I can’t use forwarders on Operation and logs copy manually to logserver.

0 Karma

manjunathmeti
Champion

You wrote splunk continuously monitor /opt. Which server is this? You need to put inputs.conf here.

0 Karma

mehrdad_2000
Path Finder

On Splunk server I add /opt to index continuously.
Logs coming from other machines without Forwarders.

0 Karma
.conf21 CFS Extended through 5/20!

Don't miss your chance
to share your Splunk
wisdom in-person or
virtually at .conf21!

Call for Speakers has
been extended through
Thursday, 5/20!