I could use some expert assistance with a regex for breaking down a custom user-agent field in an IIS log into component fields while avoiding a conflict with other fields.  We run software that uses IIS as a file server, and the software injects a custom user-agent value into the IIS log with every request.  Here is a sample of the user agent:   JTDI+(JDMS+1.0.11.2.20200807;+Win10+10.0;+229.0/62/-1;Branch|UnitType|System|City|ST|SiteIDOverride|SvrType|2.5;C8F7504F064E;UTA-AVD)   The IIS log is space delimited, so all of that lands in the cs_user_agent field just fine.  I made a sort of running mess of extracting the subfields.  Within the string are subfields delimited by semicolon, and sub-subfields delimited by / and |.  Here are my separate extractions, in order as the fields appear in the string:   ^[^\(\n]*JTDI\+\((?P<jkversion>[^;]+) ^[^;\n]*;(?P<os>[^;]+) ^(?:[^;\n]*;){2}(?P<freespace>[^/]+) ^(?:[^;\n]*;){2}\+\d+\.\d+/(?P<pending>\d+) ^(?:[^;\n]*;){3}(?P<SiteDescription>[^;]+) ^(?:[^;\n]*;){4}(?P<MAC>[^;]+) ^(?:[^;\n]*;){5}(?P<cs_hostname>[^\)]+)   Technically after the 'pending' field there should be a 'hits' field (represented by the -1 above), but we don't use it, so I didn't bother extracting it. So my problem is the parentheses.  If a filename shows up in the cs_uri_stem field that includes them, like filename(copy1).txt, the () throw off my jkversion and cs_hostname extractions, because I don't know how to accommodate the possible existence of parentheses outside the cs_user_agent field. So I guess my question is two-fold. 1)  I know my overall user-agent extraction should be a single transform instead of all separate field extractions, but I'm not sure how to tie them all together because I couldn't see a way to extract strings like that in the field extractor interface in Splunk. 2) How can I fix my regex so that parentheses appearing in other fields don't break my jkversion and cs_hostname extractions? Help? ... View more

Since recently completing the upgrade of our search head to 8.0.0, a schedule search that emails and attached csv is now adding an extra line between every event. Some of our scheduled csv's are used by other automation, so the extra lines are breaking something. Is this a known issue that was fixed in 8.0.1? I see in the 8.0.1 "Fixed Issues" this: SPL-176009, SPL-166728 - Alert email spacing issue, but I don't know how to look up what the specific issues are to know if that relates to my problem or not. ... View more

I work with a file delivery system that relies on an xml "index" file that acts as a sort of manifest of files available for download in a given data set. I need to index these xml files so we can search and report on them in Splunk. While the files are fairly simple in construction, I am having a problem when trying to get them indexed cleanly. Here is a sample of an xml file: <?xml version="1.0" encoding="UTF-8"?> <DSIF> <Heading> <GenDate>20191014T201231</GenDate> <Root>\\UMECHUJX\cbm_navy\E-2D\</Root> </Heading> <Record> <Path>HFP/IAVAS</Path> <Size>14394144</Size> <Hash>438b87856704c7c3bb5b3927d6d5959aaf997e8777e1ede40d35ff425eb45116</Hash> <Date>20190825T195424</Date> <Name>windows10.0-kb4500641.exe</Name> </Record> <Record> <Path>ENG/ALL</Path> <Size>1458278573</Size> <Hash>b214cf5847fdd3244fae60036c6c9aad056e5114bd8320dd460cd902bd44a456</Hash> <Date>20190906T143311</Date> <Name>E2D_ALE_System_V091_Point_Release_20190822.zip</Name> </Record> <Record> <Path>HFP/Files</Path> <Size>569985537</Size> <Hash>dc51058df85791c0786e6f41eba315fd044f0fc911a70763748f3ec9ee95d272</Hash> <Date>20190919T013754</Date> <Name>1.02.02_version_update.zip</Name> </Record> Here is my props.conf stanza: [jkcsindex] DATETIME_CONFIG = NONE KV_MODE = xml category = Structured description = JKCS Index file disabled = false pulldown_type = true SHOULD_LINEMERGE = false NO_BINARY_CHECK = true LINE_BREAKER = (\s*)</Record>(\s*)<Record>(\s*) REPORT-jkcsxml = jkcsxml TRANSFORMS-nullIndexHeader = nullIndexHeader From the transforms.conf, here is the nullIndexHeader stanza to remove the header and extra tags: [nullIndexHeader] REGEX = (?m)^(<\?xml)|(\<DSIF\>)|(\<Heading\>)|(\<\/GenDate\>)|(\<\/Root\>)|(\<\/Heading\>)|(\<\/Record\>)|(\<\/DSIF\>) DEST_KEY = queue FORMAT = nullQueue And here is the transforms.conf stanza to break out the xml tags: [jkcsxml] REGEX = <([^>]+)>([^<]*)</\1\> FORMAT = $1::$2 MV_ADD = true REPEAT_MATCH = false So my main problem is that after all of this, when I try to output a simple table, all of the results get doubled, like this: Why is that? Where is this duplication coming from? When I do a simple search to show the raw events, only one of each record is listed. I lot of this is cobbled together from other answers that have been posted here, and some of it I don't entirely understand. I've been fighting regular expressions all last week just to get the fields extracted (because what works for me at regex101.com doesn't seem to apply in the LINE BREAKER in the Add Data GUI), but I can't figure out this doubling of the results in the table. Help greatly appreciated! ... View more

In the past, I have used SEDCMD statements in my props.conf to remove text and whole lines from events so they would not clog our indexes and clutter the search results wiht extraneous text. However, today, I tried to set up an SEDCMD string to remove part of a line in our IIS logs, and it appeared to work, but somehow the actual effect is that it just removed the text from the regular search result, but if you expand the actual event, where you can see the 'Event Actions' button and all the fields, the string we were trying to remove is still available and searchable (still shows in the Interesting Fields). Here is my script: SEDCMD-Auth = s/Basic+.*// I am trying to remove the content of the Authorization field that has been added to our IIS logs, along with everything else to the end of the line after that field. This is a script I used earlier to remove a line from a Windows Event Log, and it is working fine: SEDCMD-EventType = s/EventType=4\r\n// What is the difference (aside from not removing a CR/LF)? What did I do wrong in the first one to cause Splunk to not actually prevent the data from being indexed? These are both in the etc/system/local/props.conf on our indexer, and we only use light forwarders. ... View more