How do I apply props.conf to create a sourcetype for my xml?

[damonmanni]

Problem:

I am stuck trying to apply props.conf to map my xml file. Read many docs + many attempts but can't figure out how to use the conf file. I placed my props.conf on my indexer. I want each iwuser entry/element. Also, do I need transforms.conf to make this work? Lastly, when I run my query I don't get any results in the table. A solid response/example would be much appreciated.

xml file

<nextid value="316">                                
</nextid>
<iwuser name="usernamehere">                        
    <email value="emailaddresshere">                
    </email>                                        
    <preferredui value="ccstd">                     
    </preferredui>
    <ismaster value="no">                           
    </ismaster>                                     
</iwuser>                                           
<iwuser name="iwov">                                
    <email value="sssss@gigi.com">            
    </email>
    <preferredui value="ccpro">                     
    </preferredui>                                  
    <ismaster value="yes">                          
    </ismaster>                                     
</iwuser>
<iwuser name="jojo" userdbid="ldap" tsid="33">
    <display-name value="the dolphin">             
    </display-name>                                
    <email value="jojo.dolphin@gigi.com">          
    </email>                                       
    <preferredui value="ccproonly">               
    </preferredui>                                 
    <ismaster value="no">                          
    </ismaster>                                    
</iwuser>

props.conf

[teamsite_xml]
DATETIME_CONFIG = CURRENT
KV_MODE = xml
LINE_BREAKER = ()
MUST_BREAK_AFTER = \
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = false
TRUNCATE = 0
pulldown_type = 1

FIELDALIAS-nextID = nextid{@value} as nextID
FIELDALIAS-userName = iwuser{@name} as userName
FIELDALIAS-userEmail = iwuser{@email} as userEmail
FIELDALIAS-userUI = iwuser{@preferredui} as userUI
FIELDALIAS-userMaster = iwuser{@ismaster} as userMaster

Query
index=dolphin sourcetype=teamsite_xml | table nextID userName userEmail userUI userMaster

[maciep]

For the parse time settings, i think this should work.

[teamsite_xml]
DATETIME_CONFIG = CURRENT
LINE_BREAKER = ([\r\n])+(?=\s*\<iwuser)
SHOULD_LINEMERGE = false

Splunk uses the line breaker to determine where events are broken. It will essentially replace the first capture group of the regex with its line breaker. So in this case we're capturing a crlf, but specifically the crlf before the opening iwuser tag. That captured crlf is where splunk will break events. We use this format with a lot of line breakers as most logs are broken with a crlf followed by some specific thing like a timestamp.

And of course, since we're telling splunk exactly where to do it, we don't want Splunk to then try to merge lines back together. I believe that's what all of the " must break *" settings are used for.

Again, those are your parse time settings - things done before splunk indexes the data. The KV_MODE and FIELDALIAS settings happen at search time - after the data has been indexed. So if you have a separate search head, then you want to put your props.conf out there as well. You can have one conf with both parse/search time settings. Or you could create one with just parse time for your indexers and one with just search time for your search heads. In any case, the search heads need to know about your field aliases and kv mode.

My laptop is both my indexer and search head, and i do have nextID, userName, etc. with your settings..so they are correct.

And one last note, the NO_BINARY_CHECK setting happens at input time, when the xml file is read initially, probably on a UF? I always recommend this wiki page to understand which settings happen where and what that might mean in your env.

https://wiki.splunk.com/Where_do_I_configure_my_Splunk_settings%3F

[damonmanni]

Thank you for your response and advice. I tried it but seem to be more tangled up. Can't get anything to work now. Here is exactly what I did in flow, including your suggestions:

[damonmanni]

1) XML test data I used

<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<iwusers>
    <nextid value="316">
    </nextid>
    <iwuser name="usernamehere">
        <email value="emailaddresshere">
        </email>
        <preferredui value="ccstd">
        </preferredui>
        <ismaster value="no">
        </ismaster>
    </iwuser>
    <iwuser name="iwov">
        <email value="jojo@dolphin.com">
        </email>
        <preferredui value="ccpro">
        </preferredui>
        <ismaster value="yes">
        </ismaster>
    </iwuser>
    <iwuser name="kahuna">
        <email value="mahalo@water.com">
        </email>
        <display-name value="kahuna">
        </display-name>
        <preferredui value="ccpro">
        </preferredui>
        <ismaster value="no">
        </ismaster>
    </iwuser>
<iwusers>

2) Index my test data
splunk add oneshot "/opt/splunkforwarder/teamsite.xml" -sourcetype teamsite_xml
-index lola

3) On splunk client containing xml file to monitor:
/opt/splunkforwarder/etc/apps/search/local/inputs.conf

[monitor:///somePathHere/tsusers.xml]
disabled = false
index = lola
sourcetype = teamsite_xml
ignoreOlderThan = 60d

4) On SHCD (search head cluster deployer)
/opt/splunk/etc/shcluster/apps/search/local/props.conf

[teamsite_xml]
DATETIME_CONFIG = CURRENT
LINE_BREAKER = ([\r\n])+(?=\s*\<iwuser)
SHOULD_LINEMERGE = false
FIELDALIAS-nextID       = nextid{@value} as nextID
FIELDALIAS-userName     = iwuser{@name} as userName
FIELDALIAS-userEmail    = iwuser{@email} as userEmail
FIELDALIAS-userUI       = iwuser{@preferredui} as userUI
FIELDALIAS-userMaster   = iwuser{@ismaster} as userMaster

5) Deployed/pushed bundle out to my SH cluster members
splunk apply shcluster-bundle -target https://:8089
waited for completion

6) Verified that new props.conf has reached each search head member
cat /opt/splunk/etc/apps/search/default/props.conf
passed

7) Cleaned up an lingering xml indexed data b/f doing new test
index=lola source="/somePathHere/tsusers.xml" | delete

**8) New test to see if props.conf is parsing properly
index=lola sourcetype=teamsite_xml source="/somePathHere/tsusers.xml"

RESULTs
I only get 2 events back instead of each iwuser being its own event. What am I doing wrong?

1   3/11/19
4:18:09.000 PM  
    <nextid value="316">
    </nextid>
    <iwuser name="usernamehere">
        <email value="emailaddresshere">
        </email>
        <preferredui value="ccstd">
        </preferredui>
        <ismaster value="no">
        </ismaster>
    </iwuser>
    <iwuser name="iwov">
        <email value="jojo@dolphin.com">
        </email>
        <preferredui value="ccpro">
        </preferredui>
        <ismaster value="yes">
        </ismaster>
    </iwuser>
    <iwuser name="kahuna">
        <email value="mahalo@water.com">
Show all 28 lines
eventtype = astra cutover  cutover host = mdc1vrs30b92a  astra  cutover  mcom  ui index = astra name = usernamehere source = /opt/splunkforwarder/astra_teamsite.xml sourcetype = teamsite_xml splunk_server = mdc1brc0210  jc_indexer tag = astra  tag = cutover  tag = jc_indexer  tag = mcom  tag = ui
2   3/11/19
4:18:09.000 PM  
<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
eventtype = astra cutover  cutover host = mdc1vrs30b92a  astra  cutover  mcom  ui index = astra source = /opt/splunkforwarder/astra_teamsite.xml sourcetype = teamsite_xml splunk_server = mdc1brc0210  jc_indexer tag = astra  tag = cutover  tag = jc_indexer  tag = mcom  tag = ui

[damonmanni]

Where am I going wrong?

[maciep]

The DATETIME_CONFIG, LINE_BREAKER, SHOULD_LINEMERGE settings need to be on your indexer(s). Otherwise, the events won't be broken correctly as you've noticed.

The FIELDALIAS settings should remain on the SHC, because that's when they'll be applied.

For now, i would just copy that props.conf to your indexers as well and try again.

[damonmanni]

Thanks for this help also. I did as u suggested but still no luck. This is not easy. 🙂 Here is my recent attempt:

1) Setup props on SHCD then deployed to SHs:

[teamsite_xml]                                                 
KV_MODE = xml                                                  
pulldown_type = 1                                              
MUST_BREAK_AFTER = \                                
NO_BINARY_CHECK = 1                                            
FIELDALIAS-nextID       = nextid{@value} as nextID             
FIELDALIAS-userName     = iwuser{@name} as userName            
FIELDALIAS-userEmail    = iwuser.email{@value} as userEmail    
FIELDALIAS-userUI       = iwuser.preferredui{@value} as userUI 
FIELDALIAS-userMaster   = iwuser.ismaster{@value} as userMaster

*2) Setup props.con on Master Node then deployed to Indexer cluster *

[teamsite_xml]
DATETIME_CONFIG = CURRENT
LINE_BREAKER = ([\r\n])+(?=\s*\

** RESULTS **

1 3/12/19
2:34:23.000 PM

Show all 28 lines

eventtype = astra cutover  cutover host = mdc1vrs30b92a  astra  cutover  mcom  ui index = astra source = /opt/splunkforwarder/astra_teamsite.xml sourcetype = teamsite_xml splunk_server = mdc1brc0210  jc_indexer tag = astra  tag = cutover  tag = jc_indexer  tag = mcom  tag = ui

2 3/12/19
2:34:23.000 PM

<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
eventtype = astra cutover cutover host = mdc1vrs30b92a astra cutover mcom ui index = astra source = /opt/splunkforwarder/astra_teamsite.xml sourcetype = teamsite_xml splunk_server = mdc1brc0210 jc_indexer tag = astra tag = cutover tag = jc_indexer tag = mcom tag = u

[maciep]

I wish i just had access to your env 🙂 What i have in my original answer is what you need on your indexers. Either you had a copy/paste problem on your last post or it's just wrong - the line_breaker looks incomplete and should_linemerge isn't there at all?

[damonmanni]

Me too! 🙂

Definitely a C&P issue. I tried to put it into code block which munged it.

Here is exactly whats on indexers:

[teamsite_xml]
DATETIME_CONFIG = CURRENT
LINE_BREAKER = ([\r\n])+(?=\s*\

NO_BINARY_CHECK = 1

FIELDALIAS-nextid = next_id as nextid

FIELDALIAS-username = iwuser_name as username

FIELDALIAS-useremail = iwuser_email as useremail

FIELDALIAS-userui = iwuser_preferredui as userui
FIELDALIAS-usermaster = iwuser_ismaster as usermaster

[damonmanni]

scratch that last garbage. this is frustrating. It'l like the box on this forum munges my paste no matter if i surround it with code blocks or not. trying again:

On the SH site:

[teamsite_xml]
KV_MODE = xml                                           
pulldown_type = 1                                       
MUST_BREAK_AFTER = \</iwusers\>                         
NO_BINARY_CHECK = 1                                     
FIELDALIAS-nextid       = next_id as nextid             
FIELDALIAS-username     = iwuser_name as username       
FIELDALIAS-useremail    = iwuser_email as useremail     
FIELDALIAS-userui       = iwuser_preferredui as userui
FIELDALIAS-usermaster   = iwuser_ismaster as usermaster 

On indexer side:

[teamsite_xml]
DATETIME_CONFIG = CURRENT
LINE_BREAKER = ([\r\n])+(?=\s*\<iwuser)
SHOULD_LINEMERGE = false

[maciep]

Yeah i typically type all the code, highlight it and then click the code button.

And just to check, where are you ingesting this file from? I've been assuming a universal forwarder so I wanted to be sure it's not an indexer or search head or heavy forwarder etc. Because that could change things.

Or is it possible you have any other teamsite_xml stanzas in a props.conf somewhere on your indexer that might be in conflict?

splunk btool props list teamsite_xml --debug

[damonmanni]

• The xml file resides on a client forwarder and sends directly to indexer for ingestion.
• Also, ran you btool on forwarder, SH, and indexer. all normal

We seem to be telling it to 'break' on each 'iwuser' occurrence, and yet it doesn't. why doesn't this work?

I should be able to make up any test file with a similar tag scenario and steer it to break on the tag of my choice and it should do so. It's been days for me trying to get this file to work. not sure why so tricky.

[maciep]

I feel like it's something silly. I have the same config on my laptop and any time i upload your test file, it's broken correctly. Not sure if you are capable of trying to upload it manually on your indexer via the web? You don't need to ingest it, but at least see if the data preview looks right when you choose the teamsite_xml sourcetype....

[damonmanni]

Apologies as my copy/paste got jammed up. Here is the proper post i want to show:

teamsite_xml]
DATETIME_CONFIG = CURRENT
KV_MODE = xml
LINE_BREAKER = (<iwusers>)
MUST_BREAK_AFTER = \</iwusers\>
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = false
TRUNCATE = 0
pulldown_type = 1
###
FIELDALIAS-nextID       = nextid{@value} as nextID
FIELDALIAS-userName     = iwuser{@name} as userName
FIELDALIAS-userEmail    = iwuser{@email} as userEmail
FIELDALIAS-userUI       = iwuser{@preferredui} as userUI
FIELDALIAS-userMaster   = iwuser{@ismaster} as userMaster

[nickhills]

Should the FIELDALIAS's not be:

 FIELDALIAS-nextID       = nextid{@value} as nextID
 FIELDALIAS-userName     = iwuser{@name} as userName
 FIELDALIAS-userEmail    = iwuser.email{@value} as userEmail
 FIELDALIAS-userUI       = iwuser.preferredui {@value} as userUI
 FIELDALIAS-userMaster   = iwuser.ismaster{@value} as userMaster

[damonmanni]

Thanks for your suggestion big Nick. I implemented your suggestions but still trying to get the other part of the props.conf to work. So, probably won't get your help working until the other is parsing properly. I'll post back when i know more. cheers