Deployment Architecture

Lost all indexes

satiex
Explorer

Hi,

I created an App on my Splunk indexer which had a props.conf, outputs.conf and transforms.conf with the following data:

[syslog]
TRANSFORMS-routing = routeAll, routeSubset

Edit $SPLUNK_HOME/etc/system/local/transforms.conf and add the following text:

[routeAll]
REGEX=(.)
DEST_KEY=_TCP_ROUTING
FORMAT=Everything

[routeSubset]
REGEX=(SYSTEM|CONFIG|THREAT)
DEST_KEY=_TCP_ROUTING
FORMAT=Subsidiary,Everything

Edit $SPLUNK_HOME/etc/system/local/outputs.conf and add the following text:

[tcpout]
defaultGroup=nothing

[tcpout:Everything]
disabled=false
server=10.1.12.1:9997

[tcpout:Subsidiary]
disabled=false
sendCookedData=false
server=10.1.12.2:1234

I then restarted my Splunk service and it froze while restarting. It then appeared that my server had frozen, so I restarted the server on the hypervisor level after waiting some time with splunk not coming back online.

Once it was back online, I realised that all Indexes were completely wiped from the mount point, and the indexes reset and started populating with new data coming from the heavy forwarder.

Is there any way that changes to conf files could have done this? Or could this have happened if the server crashed while Splunk was restarting?

Thank you.

Tags (2)
0 Karma
1 Solution

adonio
Ultra Champion

outputs.conf on the indexers is a really bad idea ...
you literally tell the indexer to ship its data out

View solution in original post

0 Karma

adonio
Ultra Champion

outputs.conf on the indexers is a really bad idea ...
you literally tell the indexer to ship its data out

0 Karma

satiex
Explorer

I see. It sounds like that is what happened.

What I'm trying to accomplish is "Replicate a subset of data to a third-party system" from this page: http://docs.splunk.com/Documentation/Splunk/7.1.1/Forwarding/Routeandfilterdatad

HF recieves data via syslog from a source. I want to filter the syslog from that source and send a subset of that to another syslog server.

I will create a new ticket to ask about how to achieve this, as the example given is for forwarding raw data only from the HF. It says that if you want to forward syslog data it needs to be output from an indexer.

0 Karma
Get Updates on the Splunk Community!

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...

Introducing the 2024 Splunk MVPs!

We are excited to announce the 2024 cohort of the Splunk MVP program. Splunk MVPs are passionate members of ...

Splunk Custom Visualizations App End of Life

The Splunk Custom Visualizations apps End of Life for SimpleXML will reach end of support on Dec 21, 2024, ...