I am considering to implement two separate indexes containing
non-anonymized data and anonyimized on the other.
input data looks like following,
I have configured props.conf and transforms.conf as,
REPORT-1 = searchext
TRANSFORMS-1 = setnull
TRANSFORMS-2 = indexrouting1
TRANSFORMS-3 = anonymize
TRANSFORMS-4 = indexrouting2
DELIMS = ","
FIELDS = "a","b","c"
REGEX = a
DEST_KEY = queue
FORMAT = nullQueue
REGEX = (\d),(\d),(\d)
FORMAT = $1,###,$2
DEST_KEY = _raw
REGEX = ###
DEST_KEY = _MetaData:Index
FORMAT = indexB
It seems that data is only going into index B.
However , I want them to go indexA and indexB
I appreciate if someone can verify this.
Normally, you'd write your transforms like;
TRANSFORMS-blah = transform1, transform2, transform3
This means that each event to be transformed will go through all three transforms before returning to the pipeline for further processing (i.e. indexing). However, that will not let you create multiple copies of the events into different indexes.
One way of doing it is to index the events normally into indexA, and then have a scheduled search that changes the events and populates another index (indexB) ;
index=indexA earliest=-1h@h latest=@h | replace "SecretStuff" with #### in _raw | collect index=indexB
index=indexA earliest=-1h@h latest=@h | rex field=_raw mode=sed "s/<some_regex>/###/" | collect index=indexB
Setting this to run 5 minutes past every hour will ensure that all events are collected only once into indexB. Restrict the access to indexA. Allow general access to indexB. Adjust timeranges and scheduling to your needs.
Hope this helps,
I got it working but it seems that field extraction of stash sourcetype extracts the raw data itself to field.
I would like to disable the stash sourcetype extraction but do you know any way to do this?
collect command allows you to specify an alternate location of where to write the file. Thus you should be able to set up a
[monitor] stanza in inputs.conf with the correct source/sourcetype (so that the field extractions are automatically applied). I'm sorry, but I haven't played with this feature a lot. You'll need to do a bit of your own testing.
...| collect index=dummy file=apa spool=f
Then set up the [monitor] to watch this file in SPLUNK_HOME/var/run/splunk
Set the source and sourcetype like the original data in inputs.conf
At the head of the file that is created there will be a line
*** SPLUNK *** and some info regarding the index you set in the the
collect This line will indexed as a separate event, but you can probably remove it through a nullQueue transform.