Getting Data In

how can I use log filename to set "host" "date"

indeed_2000
Motivator

Hi I have log files like this:

log.machine03.20200310.bz2
log.machine04.20200310.bz2
log.machine05.20200310.bz2

these files copy each day to /opt , and splunk continuously monitor /opt.
now how can I use log filename to set "host" "date" of logs in splunk?

FYI: in every line of log file only store time NOT date! that's why I need to use date that exist in file name.
e.g.
file name = log.machine05.20200310.bz2
01:00:00 info logmessage
02:00:00 info logmessage
03:00:00 info logmessage
...

Thanks,

0 Karma

manjunathmeti
Champion

You can set host from file name in inputs.conf on forwarder.
inputs.conf

[monitor://<path>]
sourcetype = sourcetype_name
host_regex = [a-zA-Z]+\d+(?!$)

For setting _time from file name use INGET-EVAL. Add below configurations on indexer server.

props.conf

[sourcetype_name]
TRANSFORMS = timestamp_eval

trsansforms.conf

[timestamp_eval]
INGEST_EVAL = _time=strptime(replace(source, "/opt/log\.[\w]+\.", ""), "%Y%m%d.bz2")

Note: INGEST_EVAL is supported in splunk version >= 7.2.0.

0 Karma

indeed_2000
Motivator

Unfortunately I can’t use forwarders on Operation and logs copy manually to logserver.

0 Karma

manjunathmeti
Champion

You wrote splunk continuously monitor /opt. Which server is this? You need to put inputs.conf here.

0 Karma

indeed_2000
Motivator

On Splunk server I add /opt to index continuously.
Logs coming from other machines without Forwarders.

0 Karma
Get Updates on the Splunk Community!

Splunk Observability as Code: From Zero to Dashboard

For the details on what Self-Service Observability and Observability as Code is, we have some awesome content ...

[Puzzles] Solve, Learn, Repeat: Character substitutions with Regular Expressions

This challenge was first posted on Slack #puzzles channelFor BORE at .conf23, we had a puzzle question which ...

Shape the Future of Splunk: Join the Product Research Lab!

Join the Splunk Product Research Lab and connect with us in the Slack channel #product-research-lab to get ...