Getting Data In

Inputs defined sourcetype overwrite a specific source using props and transforms

svemurilv
Path Finder

Hi ,
all my /var/log file are are input configured to redirect to sourcetype=unixlogs and now i would like to redirect all /var/log/smart* to different sourcetypes based on the folder structure its having.

`[monitor:///var/log]
 disabled = 0
index=inux
sourcetype = unixlog`

Props
`[ source::.../var/log/smart... ]
TRANSFORMS-setSourceType = smart_source`

    Transforms:

     [ smart_source ]
     REGEX ="(\/var\/log\/)(?<![^\/\n])(\w..+)\/(\w..+)"
      DEST_KEY = MetaData:Sourcetype
    FORMAT = sourcetype::$1
    SOURCE_KEY = MetaData:Source

regx also not working

    /var/log/smart-agent/smart-agent.log
    /var/log/smart-agent/smart-agent.out
    /var/log/smart-infra-solr/solr.log
    /var/log/smart-infra-solr/solr_gc.log
    /var/log/smart-metrics-collector/smart-metrics-collector.log
    /var/log/smart-metrics-collector/collector-gc.log-201804231606
    /var/log/smart-metrics-collector/collector-gc.log-201805100755
    /var/log/smart-metrics-collector/gc.log-201804231606
    /var/log/smart-metrics-collector/gc.log-201805100755
    /var/log/smart-metrics-collector/hbase-ams-master-xo1ph140.log
    /var/log/smart-metrics-grafana/grafana.log
    /var/log/smart-metrics-grafana/grafana.out
    /var/log/smart-metrics-monitor/smart-metrics-monitor.out
    /var/log/smart-server/smart-alerts.log
    /var/log/smart-server/smart-audit.log
    /var/log/smart-server/smart-config-changes.log
    /var/log/smart-server/smart-eclipselink.log
    /var/log/smart-server/smart-server.log
    /var/log/smart-server/smart-server-command.log
    /var/log/smart-server/capshed-view/capshed-view.log
0 Karma

somesoni2
Revered Legend

If you're monitoring the path [monitor:///var/log], then to override sourcetype of all sources/files with path starting with /var/log/smart, try this on your indexer/heavy forwarder

Props (no spaces in stanza name)

[source::/var/log/smart*]
TRANSFORMS-setSourceType = smart_source

Transforms:(no spaces in stanza name)

[smart_source]
# Do not need doublequotes. Simple regex to capture folder name after /var/log
REGEX = \/var\/log\/([^\/\n]+)
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::$1
SOURCE_KEY = MetaData:Source
0 Karma

svemurilv
Path Finder

Do i need to get those props and transforms on indexers also ?

0 Karma

somesoni2
Revered Legend

If youre using a heavy forwarder (or intermediate forwader) which is a full Enterprise Splunk instance then it should (just go there), not on indexers. If your data is coming from univeraal forwarder to indexers directly this should be done on Indexers.

0 Karma

FrankVl
Ultra Champion
  1. Why the ... at the start of your source:: stanza in props?
  2. Your regex in transforms.conf doesn’t make much sense to me. Can you explain based on the sample folder structure which bit you want to capture into the source type field? You’re currently assigning $1 which is the first capture group and based on your regex that is the /var/log/ bit...
0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Announcing Modern Navigation: A New Era of Splunk User Experience

We are excited to introduce the Modern Navigation feature in the Splunk Platform, available to both cloud and ...

Kubernetes Alerts That Work Out of the Box with AutoDetect detectors in Splunk ...

A deployment rolls out slowly, one pod sits not ready for a few minutes, and whoever is on call gets an alert ...

Observability at Your Fingertips Directly From Splunk Platform—for Free

At .conf, we announced our roadmap to bring observability foundations to Splunk Platform. Bringing metrics and ...