monitor stanza in inputs.conf not working, data is not getting indexed

[krishnarajapant]

Hi Experts,

Below is my inputs.conf stanza which we have in forwarders server in the below path. SplnkFwdrinputs is the app name which we have created in deployment server and pushed to the forwarder by defining the serverclass

/opt/splunkforwarder/etc/apps/SplnkFwdrinputs/local

[monitor:///opt/softwareag/profiles/IS/workspace/temp/splunkStorage]
disabled = false
sourcetype = TMF_XML
index = WMTMF
crcSalt = <SOURCE>

We have created the index as well in the indexer server and we could see the index in the indexer server with name WMTMF.

we have all .xml files in the /opt/softwareag/profiles/IS/workspace/temp/splunkStorage path but we are not able to see any data in the indexer

we have defined the stanza in props.conf of indexer server to handle the xml files which is working as expected, same has been tested with manual upload.

we could see all the xml files when we give the command ./splunk monitor list inputs

Not sure what am I missing here?

-Krishna Rajapantula

[tskinnerivsec]

Sorry, meant, I saw no issues with props, not new.

[krishnarajapant]

Hi,

Yes, event i have noticed that and changed the index name in inputs.conf with lower case, still no luck.

can you please see the above props.conf , is there any issue with that [TMF_XML] stanza?

-Krishna Rajapantula

[tskinnerivsec]

I also just noticed, your index name is upper case, which could be causing issues. Per splunkdocs:

1. To create a new index, enter:

A name for the index. User-defined index names must consist of only numbers, lowercase letters, underscores, and hyphens. They cannot begin with an underscore or hyphen, or contain the word "kvstore".

http://docs.splunk.com/Documentation/Splunk/6.2.5/Indexer/Setupmultipleindexes

you may want to change that, edit your file monitor, and see if that helps.

[tskinnerivsec]

You need to see what props and transforms have been applied to that sourcetype on the indexers. If the data is showing up in main, then the file monitor itself is working and communication to the indexers is ok. A configuration on the indexers may be getting applied to the sourcetype and overriding the index it is being directed to.

on the indexers try this from the /splunk/bin directory

./splunk btool list props > props_list.txt

then, look through that file for a line that says
[TMF_XML]

The stanzas below that line will indicate what is being applied to that sourcetype. A clue to your issue may rest
in one of those configuration stanzas.

[krishnarajapant]

Hi,

I have below for [TMF_XML] stanza in props.conf in indexer server

[TMF_XML]
ANNOTATE_PUNCT = True
AUTO_KV_JSON = true
BREAK_ONLY_BEFORE = \
BREAK_ONLY_BEFORE_DATE = false
CHARSET = AUTO
DATETIME_CONFIG = \etc\datetime.xml
HEADER_MODE =
KV_MODE = xml
LEARN_SOURCETYPE = true
LINE_BREAKER_LOOKBEHIND = 100
MAX_DAYS_AGO = 2000
MAX_DAYS_HENCE = 2
MAX_DIFF_SECS_AGO = 3600
MAX_DIFF_SECS_HENCE = 604800
MAX_EVENTS = 256
MAX_TIMESTAMP_LOOKAHEAD = 128
MUST_BREAK_AFTER =
MUST_NOT_BREAK_AFTER =
MUST_NOT_BREAK_BEFORE =
NO_BINARY_CHECK = true
REPORT-xmlext = xml-extr
SEGMENTATION = indexing
SEGMENTATION-all = full
SEGMENTATION-inner = inner
SEGMENTATION-outer = outer
SEGMENTATION-raw = none
SEGMENTATION-standard = standard
SHOULD_LINEMERGE = True
TRANSFORMS =
TRUNCATE = 10000
category = Custom
detect_trailing_nulls = auto
disabled = false
maxDist = 100
priority =
pulldown_type = true
sourcetype =

-Krishna Rajapantula

[tskinnerivsec]

I saw new issues with the props.conf. The inputs.conf file is not the only thing that needed to get changed. Did you recreate the index in lower case? If not, create the new index.

[krishnarajapant]

Hi,
I have created new index with lowercase with name wmtmf1 and updated the inputs.conf, still it is same.

can you please tell me the issues with props.conf

Above result is the output of ./splunk btool list props > props_list.txt

-Krishna Rajapantula.

[krishnarajapant]

Hi tskinnerivsec,

Thanks for your responses.

I have tried all the steps mentioned, still no luck.

surprisingly if i'm not mentioning then it is getting indexed in default index

So I believe it is only problem with the indexes other than default "main" index.

I have created those index manually before pushing this data , not sure where is the issue?

-Krishna Rajapantula.

[tskinnerivsec]

Have you performed a tcp dump from the source host to see if the forwarder is attempting to send the data to your indexer on tcp port 9997 ? Also, is any eventdata at all being indexed from this universal forwarder?

[tlelle_splunk]

You can take out

[tcpout-server://indexerip:9997]

from your outputs.conf, assuming that you have an actual IP listed in the line

[tcpout:default-autolb-group]
server = indexerip:9997

also try the command ./splunk list forward-server to ensure that the connection to your indexer is active.

If you make the changes and the server is still showing inactive, ensure that port 9997 is open on your indexer.

[krishnarajapant]

Hi tlelle,

Yes correct, I have mentioned actual ip in that stanza instead of "indexerip"

I have removed the [tcpout-server://indexerip:9997] stanza from outputs.conf

I have one forwarder indexerip:9997 with active list in my forwarder servr.

Still i'm not able to see any data in my indexer

-Krishna Rajapantula

[tlelle_splunk]

Have you previously indexed these files into another index (say, maybe a test index) and have since switched the index that you want to contain the data? If so, have the source files that you are trying to index changed?

It's a stretch, but maybe try changing your crcsalt?

Have you had any success testing out your data with a oneshot command? http://docs.splunk.com/Documentation/Splunk/6.2.5/Data/MonitorfilesanddirectoriesusingtheCLI#CLI_com...

[krishnarajapant]

Hi tlelle,

Thanks for your responses.

I have tried all the steps mentioned, still no luck.

surprisingly if i'm not mentioning then it is getting indexed in default index

So I believe it is only problem with the indexes other than default "main" index.

I have created those index manually before pushing this data , not sure where is the issue?

-Krishna Rajapantula.

[jensonthottian]

Did you try restarting splunkd?

[tlelle_splunk]

What does your outputs.conf look like? Were you already able to onboard data previously from this machine?

[krishnarajapant]

Hi tlelle,

Thanks for your response.

outputs.conf is in /opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/local

[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
server = indexerip:9997

[tcpout-server://indexerip:9997]

No first time i'm trying to push data from this machine.

even I have tried the same in another environment as well, same issue, not sure what i'm missing.

-Krishna Rajapantula

[krishnarajapant]

Thanks jensonthottian for your response.

Yes I did that..Still no luck.

-Krishna Rajapantula