Deployment Architecture
Highlighted

sourcetype override on indexer cluster

Explorer

Hi guys,
We have a clustered 6.5.3 deployment and following this doc to re-assign a new sourcetype to squid access logs but to no success.
The squid log is in the default log format as per https://wiki.squid-cache.org/Features/LogFormat

The sourcetype is currently access-too_small.

I don't have control of the UFs to update its inputs.conf so taking this approach for now.

Re-assigning all events so regex is .*. I have seen . as well .*. Which is the correct syntax to match all events?

Here are the props and transforms on the indexers:

/opt/splunk/etc/slave-apps/Splunk_TA_squid/local/transforms.conf       [squid_override_sourcetype]
/opt/splunk/etc/system/default/transforms.conf                         CAN_OPTIMIZE = True
/opt/splunk/etc/system/default/transforms.conf                         CLEAN_KEYS = True
/opt/splunk/etc/system/default/transforms.conf                         DEFAULT_VALUE = 
/opt/splunk/etc/slave-apps/Splunk_TA_squid/local/transforms.conf       DEST_KEY = MetaData:Sourcetype
/opt/splunk/etc/slave-apps/Splunk_TA_squid/local/transforms.conf       FORMAT = sourcetype::squid
/opt/splunk/etc/system/default/transforms.conf                         KEEP_EMPTY_VALS = False
/opt/splunk/etc/system/default/transforms.conf                         LOOKAHEAD = 4096
/opt/splunk/etc/system/default/transforms.conf                         MATCH_LIMIT = 100000
/opt/splunk/etc/system/default/transforms.conf                         MV_ADD = False
/opt/splunk/etc/slave-apps/Splunk_TA_squid/local/transforms.conf       REGEX = .*
/opt/splunk/etc/system/default/transforms.conf                         SOURCE_KEY = _raw
/opt/splunk/etc/system/default/transforms.conf                         WRITE_META = False


/opt/splunk/etc/slave-apps/Splunk_TA_squid/local/props.conf       [source::...squid/access.log]
/opt/splunk/etc/system/default/props.conf                         ANNOTATE_PUNCT = True
/opt/splunk/etc/system/default/props.conf                         AUTO_KV_JSON = true
/opt/splunk/etc/system/default/props.conf                         BREAK_ONLY_BEFORE = 
/opt/splunk/etc/system/default/props.conf                         BREAK_ONLY_BEFORE_DATE = True
/opt/splunk/etc/system/default/props.conf                         CHARSET = UTF-8
/opt/splunk/etc/system/default/props.conf                         DATETIME_CONFIG = /etc/datetime.xml
/opt/splunk/etc/system/default/props.conf                         HEADER_MODE = 
/opt/splunk/etc/system/default/props.conf                         LEARN_MODEL = true
/opt/splunk/etc/system/default/props.conf                         LEARN_SOURCETYPE = true
/opt/splunk/etc/system/default/props.conf                         LINE_BREAKER_LOOKBEHIND = 100
/opt/splunk/etc/system/default/props.conf                         MATCH_LIMIT = 100000
/opt/splunk/etc/system/default/props.conf                         MAX_DAYS_AGO = 2000
/opt/splunk/etc/system/default/props.conf                         MAX_DAYS_HENCE = 2
/opt/splunk/etc/system/default/props.conf                         MAX_DIFF_SECS_AGO = 3600
/opt/splunk/etc/system/default/props.conf                         MAX_DIFF_SECS_HENCE = 604800
/opt/splunk/etc/system/default/props.conf                         MAX_EVENTS = 256
/opt/splunk/etc/system/default/props.conf                         MAX_TIMESTAMP_LOOKAHEAD = 128
/opt/splunk/etc/system/default/props.conf                         MUST_BREAK_AFTER = 
/opt/splunk/etc/system/default/props.conf                         MUST_NOT_BREAK_AFTER = 
/opt/splunk/etc/system/default/props.conf                         MUST_NOT_BREAK_BEFORE = 
/opt/splunk/etc/system/default/props.conf                         SEGMENTATION = indexing
/opt/splunk/etc/system/default/props.conf                         SEGMENTATION-all = full
/opt/splunk/etc/system/default/props.conf                         SEGMENTATION-inner = inner
/opt/splunk/etc/system/default/props.conf                         SEGMENTATION-outer = outer
/opt/splunk/etc/system/default/props.conf                         SEGMENTATION-raw = none
/opt/splunk/etc/system/default/props.conf                         SEGMENTATION-standard = standard
/opt/splunk/etc/system/default/props.conf                         SHOULD_LINEMERGE = True
/opt/splunk/etc/system/default/props.conf                         TRANSFORMS = 
/opt/splunk/etc/slave-apps/Splunk_TA_squid/local/props.conf       TRANSFORMS-sourcetype = squid_override_sourcetype
/opt/splunk/etc/system/default/props.conf                         TRUNCATE = 10000
/opt/splunk/etc/system/default/props.conf                         detect_trailing_nulls = false
/opt/splunk/etc/system/default/props.conf                         maxDist = 100
/opt/splunk/etc/system/default/props.conf                         priority = 
/opt/splunk/etc/system/default/props.conf                         sourcetype = 

I have also tried using the [host::redacted_hostname] and [access-too_small] stanzas in props but to no avail.

Cluster automatically did a rolling restart on initial app push, but subsequent updates to props/transforms didn't. Did a rolling restart just in case as well.

Having read a lot of related posts here I see people having success so I know its an oversight on my side so any pointers would be much appreciated.

The settings shown above are on the first indexers receiving the events from the UF where logs are collected.
The instance collecting the logs is a true UF, not a full splunk enterprise install

0 Karma
Highlighted

Re: sourcetype override on indexer cluster

Builder

Hi @eddiet,

Try something like below,

in local folder create props.conf and use host/source to identify events & override sourcetype as mentioned below,

props.conf
[host::redacted_hostname]
sourcetype = squid_override_sourcetype
0 Karma
Highlighted

Re: sourcetype override on indexer cluster

Explorer

Thanks but I did try this, didn't work then realised that it only takes effect on the forwarding instance, not on the receiver.

0 Karma
Highlighted

Re: sourcetype override on indexer cluster

Builder

props.conf and transforms.conf can't be configured in Universal forwarder. so it should take effect on either Heavy forwarder/Indexer.

0 Karma
Highlighted

Re: sourcetype override on indexer cluster

Esteemed Legend

To do a true sourcetype override like you think you need, you need to deploy the changes to the first Heavy Forwarder or Indexers that receives the events. If you installed full Splunk Enterprise on your "UF", then it is not a UF, it is HF and you need to deploy there. But maybe you do not need to do this at all. Have you considered a sourcetype rename (happens on the Search Heads at search time)?

https://docs.splunk.com/Documentation/Splunk/latest/Data/Renamesourcetypes

0 Karma
Highlighted

Re: sourcetype override on indexer cluster

Builder

Hi,

Thanks for the link. just got to know that there is an option to rename sourcetype at search time.

0 Karma
Highlighted

Re: sourcetype override on indexer cluster

Explorer

The settings shown in my first post are on the first indexers receiving the events.
The instance collecting the logs is a true UF, not a full splunk enterprise install.

0 Karma
Speak Up for Splunk Careers!

We want to better understand the impact Splunk experience and expertise has has on individuals' careers, and help highlight the growing demand for Splunk skills.