Hi guys,
We have a clustered 6.5.3 deployment and following this doc to re-assign a new sourcetype to squid access logs but to no success.
The squid log is in the default log format as per https://wiki.squid-cache.org/Features/LogFormat
The sourcetype is currently access-too_small.
I don't have control of the UFs to update its inputs.conf so taking this approach for now.
Re-assigning all events so regex is .*
. I have seen .
as well .*
. Which is the correct syntax to match all events?
Here are the props and transforms on the indexers:
/opt/splunk/etc/slave-apps/Splunk_TA_squid/local/transforms.conf [squid_override_sourcetype]
/opt/splunk/etc/system/default/transforms.conf CAN_OPTIMIZE = True
/opt/splunk/etc/system/default/transforms.conf CLEAN_KEYS = True
/opt/splunk/etc/system/default/transforms.conf DEFAULT_VALUE =
/opt/splunk/etc/slave-apps/Splunk_TA_squid/local/transforms.conf DEST_KEY = MetaData:Sourcetype
/opt/splunk/etc/slave-apps/Splunk_TA_squid/local/transforms.conf FORMAT = sourcetype::squid
/opt/splunk/etc/system/default/transforms.conf KEEP_EMPTY_VALS = False
/opt/splunk/etc/system/default/transforms.conf LOOKAHEAD = 4096
/opt/splunk/etc/system/default/transforms.conf MATCH_LIMIT = 100000
/opt/splunk/etc/system/default/transforms.conf MV_ADD = False
/opt/splunk/etc/slave-apps/Splunk_TA_squid/local/transforms.conf REGEX = .*
/opt/splunk/etc/system/default/transforms.conf SOURCE_KEY = _raw
/opt/splunk/etc/system/default/transforms.conf WRITE_META = False
/opt/splunk/etc/slave-apps/Splunk_TA_squid/local/props.conf [source::...squid/access.log]
/opt/splunk/etc/system/default/props.conf ANNOTATE_PUNCT = True
/opt/splunk/etc/system/default/props.conf AUTO_KV_JSON = true
/opt/splunk/etc/system/default/props.conf BREAK_ONLY_BEFORE =
/opt/splunk/etc/system/default/props.conf BREAK_ONLY_BEFORE_DATE = True
/opt/splunk/etc/system/default/props.conf CHARSET = UTF-8
/opt/splunk/etc/system/default/props.conf DATETIME_CONFIG = /etc/datetime.xml
/opt/splunk/etc/system/default/props.conf HEADER_MODE =
/opt/splunk/etc/system/default/props.conf LEARN_MODEL = true
/opt/splunk/etc/system/default/props.conf LEARN_SOURCETYPE = true
/opt/splunk/etc/system/default/props.conf LINE_BREAKER_LOOKBEHIND = 100
/opt/splunk/etc/system/default/props.conf MATCH_LIMIT = 100000
/opt/splunk/etc/system/default/props.conf MAX_DAYS_AGO = 2000
/opt/splunk/etc/system/default/props.conf MAX_DAYS_HENCE = 2
/opt/splunk/etc/system/default/props.conf MAX_DIFF_SECS_AGO = 3600
/opt/splunk/etc/system/default/props.conf MAX_DIFF_SECS_HENCE = 604800
/opt/splunk/etc/system/default/props.conf MAX_EVENTS = 256
/opt/splunk/etc/system/default/props.conf MAX_TIMESTAMP_LOOKAHEAD = 128
/opt/splunk/etc/system/default/props.conf MUST_BREAK_AFTER =
/opt/splunk/etc/system/default/props.conf MUST_NOT_BREAK_AFTER =
/opt/splunk/etc/system/default/props.conf MUST_NOT_BREAK_BEFORE =
/opt/splunk/etc/system/default/props.conf SEGMENTATION = indexing
/opt/splunk/etc/system/default/props.conf SEGMENTATION-all = full
/opt/splunk/etc/system/default/props.conf SEGMENTATION-inner = inner
/opt/splunk/etc/system/default/props.conf SEGMENTATION-outer = outer
/opt/splunk/etc/system/default/props.conf SEGMENTATION-raw = none
/opt/splunk/etc/system/default/props.conf SEGMENTATION-standard = standard
/opt/splunk/etc/system/default/props.conf SHOULD_LINEMERGE = True
/opt/splunk/etc/system/default/props.conf TRANSFORMS =
/opt/splunk/etc/slave-apps/Splunk_TA_squid/local/props.conf TRANSFORMS-sourcetype = squid_override_sourcetype
/opt/splunk/etc/system/default/props.conf TRUNCATE = 10000
/opt/splunk/etc/system/default/props.conf detect_trailing_nulls = false
/opt/splunk/etc/system/default/props.conf maxDist = 100
/opt/splunk/etc/system/default/props.conf priority =
/opt/splunk/etc/system/default/props.conf sourcetype =
I have also tried using the [host::redacted_hostname]
and [access-too_small]
stanzas in props but to no avail.
Cluster automatically did a rolling restart on initial app push, but subsequent updates to props/transforms didn't. Did a rolling restart just in case as well.
Having read a lot of related posts here I see people having success so I know its an oversight on my side so any pointers would be much appreciated.
The settings shown above are on the first indexers receiving the events from the UF where logs are collected.
The instance collecting the logs is a true UF, not a full splunk enterprise install
To do a true sourcetype override like you think you need, you need to deploy the changes to the first Heavy Forwarder or Indexers that receives the events. If you installed full Splunk Enterprise on your "UF", then it is not a UF, it is HF and you need to deploy there. But maybe you do not need to do this at all. Have you considered a sourcetype rename (happens on the Search Heads at search time)?
https://docs.splunk.com/Documentation/Splunk/latest/Data/Renamesourcetypes
The settings shown in my first post are on the first indexers receiving the events.
The instance collecting the logs is a true UF, not a full splunk enterprise install.
Hi,
Thanks for the link. just got to know that there is an option to rename sourcetype at search time.
Hi @eddiet,
Try something like below,
in local folder create props.conf and use host/source to identify events & override sourcetype as mentioned below,
props.conf
[host::redacted_hostname]
sourcetype = squid_override_sourcetype
Thanks but I did try this, didn't work then realised that it only takes effect on the forwarding instance, not on the receiver.
props.conf and transforms.conf can't be configured in Universal forwarder. so it should take effect on either Heavy forwarder/Indexer.