This led me down an interesting little rabbit hole! So spath did actually break the XML the way I'd hoped, but it doesn't appear to know how to deal with wildcards in the field name. This led me to xpath, but when I tried to use xpath it broke in the exact same fashion xmlkv did! Here's an example block I'm trying to parse: <GET_EMBEDDED_HEALTH_DATA> <HEALTH_AT_A_GLANCE> <BIOS_HARDWARE STATUS= "Failed"/> <FANS STATUS= "OK"/> <TEMPERATURE STATUS= "OK"/> <POWER_SUPPLIES STATUS= "OK"/> <PROCESSOR STATUS= "OK"/> <MEMORY STATUS= "OK"/> <NETWORK STATUS= "OK"/> <STORAGE STATUS= "OK"/> </HEALTH_AT_A_GLANCE> </GET_EMBEDDED_HEALTH_DATA> So I was trying to use a search string like this: sourcetype=ilo:systemhealth | spath | search "GET_EMBEDDED_HEALTH_DATA.HEALTH_AT_A_GLANCE.*" NOT "OK" the whole event is in the neighborhood of 500 lines of nested xml, because below this it captures more detail about the hardware component tests. The idea is to filter within this xml block in the search and table out the offending hardware component and specific details later. ... View more

running splunk 6.3.1 on search head and on all indexers. ... View more

Worked like a champ!! Thanks so much! ... View more

I really Appreciate all of your help! Our environment is rather large. We have eight different indexers across two physical locations. I am setting the configuration entries on the search head but verifying that the bundle replicated to the indexer and that the stanza reflects what is on the search head. I was doing the delete command so based on that perhaps I was just unhiding the events when they were reindexed. I will try again soon issuing the clean command on the index itself and see if I get different results. Thanks again for everything and I will let you know ... View more

It is in the application which I am doing the searching from, in this case /etc/apps/HPCI/local/props.conf ... View more

😞 no such luck.. This is a very odd issue. Especially since using the 6.2 add-data tool, it breaks the event perfectly, evn when the event I used to build the sourcetype interactively is the very same event that got indexed! ... View more

So I tried both approaches you suggested: the first with linebreaker to a string that would never ever show up in this data, and the last with the regex you described above. Rinsed and repeated the same effort: 1) purged index 2) cleaned forwarder event data/fishbucket 3) | extract reload=t and /debug/refresh 4) waited for data to be resent the events still come back in breaking at line 18, , for every event going back on logs with mod times over a year old which definitely aren't being written to any longer. This almost feels like a bug, because I am definitely breaking other xml with more complexity and haven't experienced this issue. ... View more

Sorry, I forgot a step. 🙂 In my situation, I was able to add the can_delete role temporarily to my ID, so I purged the index, and the indexer event data/fishbucket and reindexed it all. ... View more

Thanks for this. Trying it now. with regard to timestamping, I am going on modtime because the directory I'm pulling from is an archival directory, so the logs will have definitely been completed and moved by the application in order for me to read them. I opted to do this because I was worried about the exact issue you were talking about. 🙂 ... View more

First, thank you for helping! I put in the suggestions you made here but unfortunately I get the exact same result after | extract reload=t and doing a /debug/refresh. ... View more

<Job> <Tag>J8675309</Tag> <AppJobID>269</AppJobID> <Name>MEH</Name> <ApplicationName>UNKNOWN</ApplicationName> <ApplicationVersion>0.0</ApplicationVersion> <Site>OUR_SITE</Site> <Project> <P-Survey>FEUD0101</P-Survey> <P-Stage>PROJECT_T</P-Stage> <P-Name>PROJECT001</P-Name> </Project> <UserID>USERID</UserID> <Master>MASTERNODE</Master> <PrintoutFilePath> /PATH/</PrintoutFilePath> <Status>Abend</Status> <PercentDone>0.0</PercentDone> <CollectionDate>2015-08-21 14:59</CollectionDate> <StartDate>2015-08-21 14:57</StartDate> <EndDate>2015-08-21 14:59</EndDate> <ArchiveDate>2015-08-24 23:55</ArchiveDate> <AbortDate>2015-08-21 14:59</AbortDate> <SystemMessage>Job Failed - JOB_FAILED file found</SystemMessage> <UserComments/> <ProjectPriority>8</ProjectPriority> <SequenceGroupName>NOT_USED</SequenceGroupName> <SequenceGroupNumber>0</SequenceGroupNumber> <JobPriority/> <FailedNodeCount>0</FailedNodeCount> <BaseLine>2014.1ext</BaseLine> <Installation>2014.1ext</Installation> <Parameters> <JobServer>COMPUTED</JobServer> <TargetDirectory>/PATH/</TargetDirectory> <PoolOverQuota>NO</PoolOverQuota> <ResourceLoad>light</ResourceLoad> </Parameters> </Job> ... View more

Also, because of the log format, happens to be line 18 every time, if that sheds any light. ... View more

Bump! (and reanimation) so our splunk search head has literally fallen to its knees over the last month or so and it turns out its when splunk-optimize is running against these tsidxstats. we have over 12,000 of them! ... View more

The one thing I am doing that might be a little unique is in the JVM options I am specifying a -Doracle.net.tns_admin path, otherwise I will have instance name resolution issues. However, this works fine with doing a dbxquery, its just in the dbinput wizard. ... View more

./java -version java version "1.8.0_20" Java(TM) SE Runtime Environment (build 1.8.0_20-b26) Java HotSpot(TM) 64-Bit Server VM (build 25.20-b23, mixed mode) ... View more

Also getting this on every drop down in 6.2.3 using the same xml syntax ... View more

Thanks for the help. The primary reason for doing this is because this same group also shares our VirtualCenter, therefore our instance is already pulling in all of their vmware cluster data with the exception of their actual syslogs. There doesn't appear to be a way in the vmware app to segregate multi-tenant vcenter clusters so this was the next best thing we could come up with. ... View more

This is getting me super close, but not quite there Your first example worked for one subset, but I think perhaps I did a bad job of explaining the whole problem: so I have one sourcetype that contains 2 types of events. One event looks like this: Job Id: 78172.queue01 Job_Name = bob Job_Owner = bob@queue01 resources_used.cpupercent = 0 resources_used.cput = 00:00:24 resources_used.mem = 12356kb resources_used.ncpus = 384 resources_used.vmem = 408076kb resources_used.walltime = 00:00:21 job_state = R queue = parallel Resource_List.jg_n24_256_none_FEX_rd_a = 384 And another event will look like this resources_available.jg_n24_256_kepler_FEX_rd_a = 648 resources_available.jg_n24_256_kepler_FEX_rd_b = 0 resources_available.jg_n24_256_kepler_FEX_rd_c = 0 resources_available.jg_n24_256_none_FEX_rd_a = 6384 resources_available.jg_n24_256_none_FEX_rd_b = 9472 resources_available.jg_n24_256_none_FEX_rd_c = 1536 resources_available.jg_n24_256_none_FEX_rd_d = 0 resources_available.jg_n24_256_none_FEX_rd_e = 0 The resource_list value is what is requested, the resources_available number is what is available and the stuff in between are sets of queues. What I'm attempting to do is determine what percentage of the request event A is being used of whats available in event B. ... View more