why are my configurations not working even after reboot?

[julienoud]

The log files I'm working with are using the log4j syntax, and I'm loading them into splunk through the GUI (not real-time monitoring)
So that I don't need to update the inputs.conf file.
I have customized the following configuration files :

indexes.conf :
[index_infodebug]
homePath=$SPLUNK_DB/$_index_infodebug/db
coldPath= $SPLUNK_DB/$_index_infodebug /colddb
thawedPath=$SPLUNK_DB/$_index_infodebug /thaweddb
frozenTimePeriodInSecs = 2628000 #1month

logs to be erased

[index_testconf]
homePath=$SPLUNK_DB/$_index_testconf /db
coldPath= $SPLUNK_DB/$_index_testconf /colddb
thawedPath=$SPLUNK_DB/$_index_testconf /thaweddb
frozenTimePeriodInSecs = 2628000 #1 month
coldToFrozenDir = my/archive/directory

logs to be retained

transforms.conf:

[infodebug_logs]
REGEX = \d{3}\s*(INFO|DEBUG)\s*[[]
DEST_KEY = _MetaData:Index
FORMAT = index_infodebug

[short_source]
SOURCE_KEY = Metadata:Source
REGEX = Windchill_\d{4}-\d\d-\d\d_\d+\d+.tgz:.\/Windchill\d{4}-\d\d-\d\d_\d+\d+\/(?[0-9a-zA-Z.-]+log) (forget the caracters in italic)
DEST_KEY = MetaData:Source

props.conf:

[testconf_sourcetype]
ADD_EXTRA_TIME_FIELDS = True
ANNOTATE_PUNCT = True
AUTO_KV_JSON = true
BREAK_ONLY_BEFORE = \d\d?d\d:\d\d
BREAK_ONLY_BEFORE_DATE = True
CHARSET = UTF-8
DATETIME_CONFIG = /etc/datetime.xml
DEPTH_LIMIT = 1000
LEARN_MODEL = true
LEARN_SOURCETYPE = true
LINE_BREAKER_LOOKBEHIND = 100
MATCH_LIMIT = 100000
MAX_DAYS_AGO = 2000
MAX_DAYS_HENCE = 2
MAX_DIFF_SECS_AGO = 3600
MAX_DIFF_SECS_HENCE = 604800
MAX_EVENTS = 256
MAX_TIMESTAMP_LOOKAHEAD = 128
SEGMENTATION = indexing
SEGMENTATION-all = full
SEGMENTATION-inner = inner
SEGMENTATION-outer = outer
SEGMENTATION-raw = none
SEGMENTATION-standard = standard
SHOULD_LINEMERGE = True
TRANSFORMS =
TRUNCATE = 10000
category = Application
description = Output produced by any Java 2 Enterprise Edition (J2EE) application server using log4j
detect_trailing_nulls = false
maxDist = 75
pulldown_type = true
TRANSFORMS-index = infodebug_logs
TRANSFORMS-source = short_source

Both regex are working :
-the first aims at routing INFO and DEBUG events to the appropriate index, which is configured to erase them after 1 month. (while other logs are archived)
- the second one is for the extraction of more readable source names.
I've tested them with the REGEX command, so i know they fit my data.

After the restart of the splunk server, i've put my data into splunk.
My problem is that NEITHER both transforms NOR the archiving part are working. I've tried with 60 seconds for the test and nothing happened. The events are only parsed the right way, as I specified in props.conf.

I would be glad if someone could help me with that issues, thanks!

[solarboyz1 · Accepted solution]

[index_infodebug]
homePath=$SPLUNK_DB/$_index_infodebug/db
coldPath= $SPLUNK_DB/$_index_infodebug /colddb

Seems to be incorrect, unless you have $_index_infodebug defined?

[index_infodebug]
homePath=$SPLUNK_DB/index_infodebug/db
coldPath= $SPLUNK_DB/index_infodebug /colddb

Was the infodebug index created?

Also, you don't seem to define the value of the sourcetype you want set:

[short_source]
SOURCE_KEY = Metadata:Source
REGEX = Windchill_\d{4}-\d\d-\d\d_\d+\d+.tgz:.\/Windchill\d{4}-\d\d-\d\d_\d+\d+\/(?[0-9a-zA-Z.-]+log) (forget the caracters in italic)
DEST_KEY = MetaData:Source
Format = VALUE_FOR_SOURCETYPE

Those are the issues I see off-hand

[julienoud]

Thanks for your answer solarboyz1.
Actually i had already solved my issue, but you're right on these 2 points. I had an error of synthax while defining my index, and i forgot the format for the "source" field, it is indeed mandatory for this kind of index-time field!
It works perfectly right now 🙂

[julienoud]

this is the (right) regex i have in my conf file :

REGEX = Windchill_\d{4}-\d\d-\d\d_\d+_\d+\.tgz:\.\/Windchill_\d{4}-\d\d-\d\d_\d+_\d+\/(?<source>[0-9a-zA-Z._-]+log)