I installed splunk standalone (9.0.4) with ansible https://github.com/splunk/splunk-ansible/ on Ubuntu jammy. That has worked well. Data is ingested from port 9997 and for now, everything goes to main index. I want to split things between multiple indexes aka windows, linux and other source types.   I think this would be through transforms as per https://docs.splunk.com/Documentation/Splunk/9.0.4/Forwarding/Routeandfilterdatad but this seems to be only valid for heavy forwarder role. Or cluster master as per https://github.com/splunk/splunk-ansible/blob/develop/roles/splunk_cluster_master/tasks/configure_indexes.yml In role variable, only found smartstore with an index array but I believe it is different. I tried * forwarding working with transform in /opt/splunk/etc/system/local/props.conf and /opt/splunk/etc/system/local/transforms.conf but nok     $ sudo cat /opt/splunk/etc/system/local/props.conf # https://docs.splunk.com/Documentation/Splunk/9.0.4/Indexer/Setupmultipleindexes [SOURCE1] TRANSFORMS-index = SOURCE1Redirect $ sudo cat /opt/splunk/etc/system/local/transforms.conf [SOURCE1Redirect] #REGEX = ,"file":{"path":"\/var\/log\/SOURCE1\/SOURCE1.log"}},"message": REGEX = ^{.*SOURCE1.*}$ DEST_KEY = _MetaData:Index FORMAT = SOURCE1     * get tcp data input losing all the json fields extract and only raw unusable data. Similar to https://community.splunk.com/t5/Getting-Data-In/Splunk-is-adding-weird-strings-like-quot-linebreaker-x00-x00/m-p/21598 * set data receiver in forwarding section and setting index in inputs.conf but not getting data ingested even if data received from tcpdump. And not found how to associate a specific receiver port to an index. tried     $ sudo more /opt/splunk/etc/system/local/inputs.conf [splunktcp://9997] disabled = 0 [splunktcp://9525] disabled = 0 index = sourcetype1       Any advices?   Thanks ... View more

I installed Splunk standalone with https://splunk.github.io/splunk-ansible/ Version 9.0.4 on Ubuntu jammy 22.04.2 Instance is up and seems running fine but after configuring data ingestion, I get nothing available in search I tested 4 different inputs and none of them worked. I suspect something related to indexing but did not identify where this could be tracked down (log file, splunk table...) Any advices? Thanks * Data receiving on port 9997 * Data input: TCP for syslog data * Local Log file: /var/log/dpkg.log * Local systemd-journald I can find ingestion activities in /opt/splunk/var/log/splunk/metrics.log but not in `index=_* component=metrics | stats count BY index,component,group,host` (no group=tcpin_connections). It hald systemd-journal permission access issue until I fixed it by adding splunk user to systemd-journal group. Network ports are listeing from `ss -tunap` and tested successfully with `nc` `curl -vk -u user:pass https://localhost:8089/services/admin/inputstatus/TailingProcessor:FileStatus > TailingProcessor-FileStatus.xml` confirms ingestion for local log file only `index=_internal source=*metrics.log tcpin_connections` = no results `| tstats count WHERE index=* OR index=_* BY host` = have data but only splunk internal index (aka _*) and localhost/splunk server `index=_internal component!="Metrics" | stats count BY index,component,group,host` = only _internal/LMStack*/Trial/splunkhost Settings > Licensing: "Trial license group": current = No licensing alerts/violations, 0% of quota Settings > Monitoring console: returns nothing on indexing (0 KB/s), historic data only shows internal sources spike at install time. Splunkd system status is green. Tried to restart a few times but did not help I checked the following resources but did not find the issue https://docs.splunk.com/Documentation/Splunk/9.0.4/Forwarding/Receiverconnection https://docs.splunk.com/Documentation/Splunk/9.0.4/Troubleshooting/Cantfinddata https://community.splunk.com/t5/Getting-Data-In/What-are-the-basic-troubleshooting-steps-in-case-of-universal/td-p/456364   The only warning that I have from Web console is on Resource usage IOWait as this is a lab system without production specs. To me, that should only slow down things but not block.   Extract from config     # more /opt/splunk/etc/apps/search/local/inputs.conf [monitor:///var/log/dpkg.log] disabled = false host = mlsplunk002 index = dpkg sourcetype = dpkg [tcp://9514] connection_host = dns host = mlsplunk002 index = syslog sourcetype = syslog [journald://journald] interval = 30 journalctl-exclude-fields = __MONOTONIC_TIMESTAMP,__SOURCE_REALTIME_TIMESTAMP journalctl-include-fields = PRIORITY,_SYSTEMD_UNIT,_SYSTEMD_CGROUP,_TRANSPORT,_PID,_UID,_MACHINE_ID,_GID,_COMM,_EXE journalctl-quiet = true [tcp://6514] connection_host = dns host = mlsplunk002 index = syslog sourcetype = syslog # more /opt/splunk/etc/apps/search/local/props.conf [dpkg] LINE_BREAKER = ([\r\n]+) NO_BINARY_CHECK = true category = Custom description = /var/log/dpkg.log disabled = false pulldown_type = true # more /opt/splunk/etc/apps/search/local/indexes.conf [dpkg] coldPath = $SPLUNK_DB/dpkg/colddb enableDataIntegrityControl = 0 enableTsidxReduction = 0 homePath = $SPLUNK_DB/dpkg/db maxTotalDataSizeMB = 512000 thawedPath = $SPLUNK_DB/dpkg/thaweddb [syslog] coldPath = $SPLUNK_DB/syslog/colddb enableDataIntegrityControl = 0 enableTsidxReduction = 0 homePath = $SPLUNK_DB/syslog/db maxTotalDataSizeMB = 512000 thawedPath = $SPLUNK_DB/syslog/thaweddb       ... View more