About JDukeSplunk

JDukeSplunk · ‎04-04-2019

Maybe something like? | tstats latest(_time) as last where index=_internal host=splunkservername sourcetype=splunk* by host sourcetype | convert ctime(last)

JDukeSplunk · ‎04-04-2019

I took over a Splunk environment where many of the windows hosts did not report to the deployer and as such they ended up with whatever app set was copied/cloned to them. I use the metrics log to get at least a list of hosts that talk to the indexers with this search. Which I think I got from Answers somewhere. Forwarder list search. index=_internal source=*metrics.log* group=tcpin_connections | regex hostname!="\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}" | eval sos_server=hostname | stats latest(sourceIp) AS IP latest(arch) AS cpu_arch latest(fwdType) AS forwarder_type latest(os) AS os_name latest(version) AS version by sos_server Anyway, I also came up with a dashboard that breaks down sourcetypes by host by index and source. It's not pretty but I've found it useful <form> <label>Splunk Sourcetypes and Hosts</label> <description></description> <fieldset autoRun="false" submitButton="true"> <input type="multiselect" token="hostgroup"> <label>Host</label> <choice value="*">All</choice> <prefix>(</prefix> <suffix>)</suffix> <valuePrefix>host=</valuePrefix> <delimiter> OR </delimiter> <fieldForLabel>host</fieldForLabel> <fieldForValue>host</fieldForValue> <search> <query>|tstats count WHERE index=* by host |table host | eval _sortfield=lower(host) | table host _sortfield | sort _sortfield</query> <earliest>-24h@h</earliest> <latest>now</latest> </search> <default>*</default> <initialValue>*</initialValue> </input> <input type="multiselect" token="idxs" searchWhenChanged="false"> <label>Indexes</label> <search> <query>| eventcount summarize=false index=* | dedup index | fields index</query> <earliest>-15m</earliest> <latest>now</latest> </search> <fieldForLabel>index</fieldForLabel> <fieldForValue>index</fieldForValue> <choice value="*">All</choice> <prefix>(</prefix> <suffix>)</suffix> <delimiter> OR </delimiter> <valuePrefix>index=</valuePrefix> <default>application</default> <initialValue>application</initialValue> </input> <input type="multiselect" token="sourcetypes" searchWhenChanged="false"> <label>Sourcetypes</label> <search> <query>|tstats count WHERE index=* by sourcetype |table sourcetype | eval _sortfield=lower(sourcetype) | table sourcetype _sortfield | sort _sortfield</query> <earliest>@d</earliest> <latest>now</latest> </search> <fieldForLabel>sourcetype</fieldForLabel> <fieldForValue>sourcetype</fieldForValue> <choice value="*">All</choice> <prefix>(</prefix> <suffix>)</suffix> <delimiter> OR </delimiter> <valuePrefix>sourcetype=</valuePrefix> <default>*</default> </input> </fieldset> <row> <panel> <table> <title>Hosts by Sourcetype</title> <search> <query>| tstats count WHERE $idxs$ by index host sourcetype |search $sourcetypes$ AND ($hostgroup$) |fields sourcetype,host | stats values(host) AS hosts by sourcetype</query> <earliest>@d</earliest> <latest>now</latest> <sampleRatio>1</sampleRatio> </search> <option name="count">50</option> <option name="dataOverlayMode">none</option> <option name="drilldown">cell</option> <option name="percentagesRow">false</option> <option name="rowNumbers">false</option> <option name="totalsRow">false</option> <option name="wrap">true</option> </table> </panel> <panel> <table> <title>Sourcetypes by Host</title> <search> <query>| tstats count WHERE $idxs$ by index host sourcetype |search $sourcetypes$ AND ($hostgroup$)| fields sourcetype,host | stats values(sourcetype) AS sourcetype by host</query> <earliest>@d</earliest> <latest>now</latest> </search> <option name="count">50</option> </table> </panel> </row> <row> <panel> <title>Sourcetype by filename Source and Index</title> <table> <search> <query>| tstats count WHERE $idxs$ by index host sourcetype source |search $sourcetypes$ AND ($hostgroup$) | eval host=upper(host) | eval sourcedate=replace(source,"\d\d+","_DATE_") | eval minimal=replace(sourcedate,"ch-(.*)-","ch_USERNAME")|stats values(host) AS hosts values(index) AS index by sourcetype minimal</query> <earliest>@d</earliest> <latest>now</latest> <sampleRatio>1</sampleRatio> </search> <option name="count">100</option> <option name="dataOverlayMode">none</option> <option name="drilldown">cell</option> <option name="percentagesRow">false</option> <option name="rowNumbers">false</option> <option name="totalsRow">false</option> <option name="wrap">true</option> </table> </panel> </row> </form>

JDukeSplunk · ‎03-28-2019

I didn't use the Web Analytics app, but I've gotten IIS logs to ingest before. As I recall when the service starts or the log rotates the new file starts with a header line that begins with "#Fields:" As seen here. https://stackify.com/how-to-interpret-iis-logs/ Anyways, here are my inputs and props.conf that I just bundled up into a deployment app. They still work for me inputs.conf [monitor://C:\inetpub\logs\LogFiles\...\] disabled = 0 whitelist = .log$ followTail = 0 sourcetype=iis ignoreOlderThan = 15d index = iis [monitor://E:\inetpub\logs\LogFiles\...\] disabled = 0 whitelist = .log$ followTail = 0 sourcetype=iis ignoreOlderThan = 15d index = iis props.conf [iis] ... INDEXED_EXTRACTIONS = w3c FIELD_DELIMITER = whitespace FIELD_HEADER_REGEX = ^#Fields:\s*(.*) MISSING_VALUE_REGEX = - TIME_FORMAT = %Y-%m-%d %H:%M:%S TIMESTAMP_FIELDS = date,time TRANSFORMS-set= setnull Transforms.conf - that uses setnull to weed out loadbalancer web hits from the logs. # This stanza removes the loadbalancer noise. [setnull] REGEX = _hc_.txt DEST_KEY = queue FORMAT = nullQueue

JDukeSplunk · ‎03-22-2019

Thanks for your input.

JDukeSplunk · ‎03-22-2019

The forwarder had to be upgraded to get the proper csv extractions. 6.5.2 did not do it, 7.2.3 did. For complete info see this https://answers.splunk.com/answers/739964/need-sedcmd-help.html

JDukeSplunk · ‎03-20-2019

What am I doing wrong? I am trying to get fields from a csv. I imported one csv file into a standalone Splunk server using the "add data" GUI. It picked up the header and displayed the fields perfectly. I then took the props.conf lines it generated and placed it on my indexer in /etc/system/local/props.conf. I then did a /debug/refresh on the indexer to reload the props.conf. When that didn't work I placed the props.conf alongside the inputs.conf on the forwarding host.. Neither worked. When I ingest the data, there are no fields, just single line-by-lines. My configs. Inputs.conf [monitor://E:\Health_Data_Fax_QA\Reporting\*(log$|txt$|csv$)] disabled = 0 followTail = 0 sourcetype=hdfax:qa ignoreOlderThan = 30d index = application crcSalt = [monitor://E:\Health_Data_Fax_UAT\Reporting\*(log$|txt$|csv$)] disabled = 0 followTail = 0 sourcetype=hdfax:uat ignoreOlderThan = 30d index = application crcSalt = props.conf [hdfax:qa] DATETIME_CONFIG = INDEXED_EXTRACTIONS = csv KV_MODE = none LINE_BREAKER = ([\r\n]+) NO_BINARY_CHECK = true SHOULD_LINEMERGE = false category = Structured description = Comma-separated value format. Set header and other settings in "Delimited Settings" disabled = false pulldown_type = true [hdfax:uat] DATETIME_CONFIG = INDEXED_EXTRACTIONS = csv KV_MODE = none LINE_BREAKER = ([\r\n]+) NO_BINARY_CHECK = true SHOULD_LINEMERGE = false category = Structured description = Comma-separated value format. Set header and other settings in "Delimited Settings" disabled = false pulldown_type = true

JDukeSplunk · ‎10-09-2018

Maybe try (index=A AND sourcetype=A1 OR sourcetype=A2) OR (index=B sourcetype=B1)

JDukeSplunk · ‎10-09-2018

You want the search sid, like in the URL? Like this? /en-US/app/search/search?q=search%20index%3Dtest&sid=1539111270.454955_197AB89D-D8CE-4CEE-BF99-1B35C805ED07 The sid is what Splunk uses to quickly pull up a result of a search without having to run it again. It's a job. Every time you run the same search without specifying the exact same time frame and search it will generate a new sid, since the search is different due to time from the last one. This would explain why you are getting different search sids when you run it. Even if you specify the exact same time frame with earliest and latest in the search string, after a certain amount of time the job will expire and you will still get a new sid. You might try to save and schedule the job and run the results link from the command line.

JDukeSplunk · ‎10-09-2018

I can't give you a specific answer for this. However I can tell you how I got mine working. Using ADExplorer or some other LDAP browser I nailed down the OU structure. I copy-pasted to ensure that I got the characters exactly. You can usually go into the properties of the object and copy it there. This assumes users are in the following OU's. OU=Users,OU=Accounts,OU=GA-ATL,OU=America,OU=Sites,DC=domain,DC=com OU=Expire,OU=Accounts,OU=GA-ATL,OU=America,OU=Sites,DC=domain,DC=com OU=WA-SEA,OU=America,OU=Sites,DC=domain,DC=com And the group mappings will only show any group that begins with "Splunk" Here is my working copy of my ..\etc\local\authentication.conf file. Which of course is populated from the GUI. [LDAP Authentication to AD] SSLEnabled = 1 anonymous_referrals = 1 bindDN = CN=splunkadsearch\, svc,CN=Users,DC=domain,DC=com bindDNpassword = XXXXXXXX charset = utf8 emailAttribute = mail groupBaseDN = OU=Security,OU=Groups,OU=GA-ATL,OU=America,OU=Sites,DC=domain,DC=com groupBaseFilter = (CN=Splunk*) groupMappingAttribute = dn groupMemberAttribute = member groupNameAttribute = cn host = PDOM05.domain.com nestedGroups = 0 network_timeout = 20 port = 636 realNameAttribute = displayname sizelimit = 1000 timelimit = 15 userBaseDN = OU=Users,OU=Accounts,OU=GA-ATL,OU=America,OU=Sites,DC=domain,DC=com;OU=Expire,OU=Accounts,OU=GA-ATL,OU=America,OU=Sites,DC=domain,DC=com;OU=WA-SEA,OU=America,OU=Sites,DC=domain,DC=com userNameAttribute = samaccountname Hope this helps.

JDukeSplunk · ‎10-02-2018

Maybe, maybe not. If your powershell/batch executes and writes out to a text/log file but does not echo it to console then Splunk will ingest the command, and the output of the command. @echo off? Something like.. DATE C:\Somefolder\somefile.bat DATE C:\Somefolder\somefile.ps1 and that should be it. You can then tell the file monitor portion of Splunk to read the output file log if you so choose. If the scripts output verbosely to the console then Splunk will probably pick that up. I've not done Windows batch in a while or powershell scripting much at all, but I think you would want your file to do the Linux equivalent of >dev null if you don't want Splunk to pickup the actual script output.

JDukeSplunk · ‎09-28-2018

I use this as my wmi.conf, and the last line for systeminfo works like a charm. # WMI FOR appdev INDEX #replace the index = line with the correct index #place this file in C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\local [settings] initial_backoff = 5 max_backoff = 20 max_retries_at_max_backoff = 0 checkpoint_sync_interval = 2 ## Processes [WMI:LocalProcesses] interval = 120 wql = Select IDProcess,PrivateBytes,Name,PercentProcessorTime,TimeStamp_Sys100NS from Win32_PerfRawData_PerfProc_Process index = wmi disabled = 0 ## Scheduled Jobs ## Use the Win32_ScheduledJob class. Note that this class can only return jobs that are created using either a script or AT.exe. ## It cannot return information about jobs that are either created by or modified by the Scheduled Task wizard. [WMI:ScheduledJobs] disabled = 0 ## Run once per day interval = 86400 wql = SELECT Caption, Command, Description, InstallDate, InteractWithDesktop, JobId, JobStatus, Name, Notify, Priority, RunRepeatedly, Status FROM Win32_ScheduledJob index = wmi ## Services ## http://msdn.microsoft.com/en-us/library/aa394418(VS.85).aspx ## Lists all services registered on the system,if they are running,and the status [WMI:Service] disabled = 0 ## Run once an hour interval = 3600 wql = SELECT Name, Caption, State, Status, StartMode, StartName, PathName, Description FROM Win32_Service index = wmi ## Update [WMI:InstalledUpdates] disabled = 0 ## Run once per day interval = 86400 wql = SELECT Description, FixComments, HotFixID, InstalledBy, InstalledOn, ServicePackInEffect FROM Win32_QuickFixEngineering index = wmi ## Uptime [WMI:Uptime] disabled = 0 ## Run once an hour interval = 3600 wql = SELECT SystemUpTime FROM Win32_PerfFormattedData_PerfOS_System index = wmi ## index = wmi ## Version [WMI:Version] disabled = 0 ## Run once per day interval = 86400 wql = SELECT Caption, ServicePackMajorVersion, ServicePackMinorVersion, Version FROM Win32_OperatingSystem index = wmi ## Model [WMI:SystemInfo] disabled = 0 ## Run once per day interval = 86400 wql = SELECT Model, Manufacturer, SystemType FROM Win32_ComputerSystem index = wmi 9/28/18 2:43:16.791 PM 20180928144316.791677 Manufacturer=LENOVO Model=7033A1U SystemType=x64-based PC wmi_type=SystemInfo host = DATLTS11954 index = wmi source = WMI:SystemInfo sourcetype = WMI:SystemInfo

JDukeSplunk · ‎09-28-2018

We do that very same thing for Tomcat logs since the server team cannot make up their mind where Apache should live. Java and GC Logs [monitor:///opt/apache-tomcat/*/GC.log] disabled = 0 index=application sourcetype = prod:sun_jvm ignoreOlderThan = 30d [monitor:///opt/prpc/prpclogs/*/GC.log] disabled = 0 index=application sourcetype = prod:sun_jvm ignoreOlderThan = 30d [monitor:///opt/prpc/pegalogs/*/GC.log] disabled = 0 index=application sourcetype = prod:sun_jvm ignoreOlderThan = 30d [monitor:///opt/prpc/logs/*/GC.log] disabled = 0 index=application sourcetype = prod:sun_jvm ignoreOlderThan = 30d

JDukeSplunk · ‎09-28-2018

I don't see why not. We have some powershell scripts that we just kick off by way of a batch file. So get a working batch file that runs your powershell the way you would expect, then put the batch file in Splunk at whatever internval/sourcetype whatever you like in Settings>>Data inputs>>Scripts. Or maybe under Powershell v3 Modular Input Settings>>Data inputs » Powershell v3 Modular Input.

JDukeSplunk · ‎09-28-2018

Thanks for responding Paolo. I'll see if I can work my code and your code into something better.

JDukeSplunk · ‎09-28-2018

For now I ended up doing this. | rex field=_raw "LabelID\>(?P<LabelID>[^\<]+)" | rex field=_raw "MemberLastName\<\/apol\:QuestionID\>\<apol\:AnswerList\>\<apol\:Answer\>\<apol\:Value\>(?P<MemberLastName>[^\<]+)" | rex field=_raw "MemberFirstName\<\/apol\:QuestionID\>\<apol\:AnswerList\>\<apol\:Answer\>\<apol\:Value\>(?P<MemberFirstName>[^\<]+)" | rex field=_raw "MentorFirstName\<\/apol\:QuestionID\>\<apol\:AnswerList\>\<apol\:Answer\>\<apol\:Value\>(?P<MentorFirstName>[^\<]+)" | rex field=_raw "MentorLastName\<\/apol\:QuestionID\>\<apol\:AnswerList\>\<apol\:Answer\>\<apol\:Value\>(?P<MentorLastName>[^\<]+)" | rex field=_raw "DOB\<\/apol\:QuestionID\>\<apol\:AnswerList\>\<apol\:Answer\>\<apol\:Value\>(?P<DOB>[^\<]+)" | rex field=_raw "EmployeeIDNum\<\/apol\:QuestionID\>\<apol\:AnswerList\>\<apol\:Answer\>\<apol\:Value\>(?P<EmployeeIDNum>[^\<]+)" | rex field=_raw "Gender\<\/apol\:QuestionID\>\<apol\:AnswerList\>\<apol\:Answer\>\<apol\:LabelID\>(?P<EmployeeGender>[^\<]+)" | rex field=_raw "\<apol\:ClientID\>(?P<ClientID>[^\<]+)" | fillnull value="" | eval Name=MemberFirstName." ".MemberLastName | eval Name=Upper(Name) | stats count(EmployeeIDNum) as Total by Name MemberLastName MemberFirstName MentorFirstName MentorLastName DOB EmployeeIDNum EmployeeGender LabelID ClientID Which results in a table like this. Name MemberLastName MemberFirstName MentorFirstName MentorLastName DOB EmployeeIDNum EmployeeGender LabelID ClientID Total John Doe Doe John Leisha Wilker 11/22/2022 999999999999 MALE PARENT 11504455 2

JDukeSplunk · ‎09-25-2018

I have a log file, that outputs different formats depending on the portion of the application doing the logging. Some of the events output like the XML sample data shown here. I'd like to find some way to extract the key-value pairs out of them. If a transforms/props can be put in place, that will recognize events like this one and extract the fields I need without interfering with the other single-line machine data and JSON entries in the log that would be nice. If multiple lines can be entered to account for XML and JSON even better. Honestly, Id be happy with an inline solution with |extract |xmlkv |xpath or something like that. Otherwise, I will be forced to write some pretty nasty |REX statements for each field. Thanks! Looking for: Clientid 11111111 MemberFirstName Jane MemberLastName Doe Gender FEMALE DOB 11/11/1911 EmployeeIDNum xxxxx MentorFirstName MentorLastName Event Samples: 2018-09-25 12:48:23,599 [tp-bio-8001-exec-151] [ STANDARD] [ ] [ PHSInt:01.01] (og.Domain_FW_Apollo_Int_.Action) INFO hostname01.domain.com|10.200.200.200|HTTP|AssessmentServices|Services|SaveAssessmentAnswers|AD0A0F376B08E09090B78F37816A41733 - INSERTING INTO SERVICE REQUEST LOG:--SERVICEREQUESTTYPE -->:SaveAssessmentAnswers--SERVICEREQUESTSTATUS -->:--TRANSACTIONID-->:3740e6fc-99xx-43f2-ba47-4630da0aaeda--MEMBERELIGID-->:--PID-->:--PARTICIPANTID-->:--DEBUGMESSAGE-->:[hostname01.domain.com] --REQUEST-->:<apol:SaveAssessmentAnswersReq xmlns:apol="http://www.domain.com/xsds/ApolloAssessment.xsd"><apol:TransactionID>3740e6fc-59ee-43f2-ba47-4630da0aaeda</apol:TransactionID><apol:ClientID>11111111</apol:ClientID><apol:AssessmentTypeID>931</apol:AssessmentTypeID><apol:ParticipantAnswerList><apol:PartQuestionAnswer><apol:QuestionID>ImpersonatorDetail</apol:QuestionID><apol:AnswerList><apol:Answer><apol:LabelID>MEMBER</apol:LabelID></apol:Answer></apol:AnswerList></apol:PartQuestionAnswer><apol:PartQuestionAnswer><apol:QuestionID>MemberFirstName</apol:QuestionID><apol:AnswerList><apol:Answer><apol:Value>Jane</apol:Value></apol:Answer></apol:AnswerList></apol:PartQuestionAnswer><apol:PartQuestionAnswer><apol:QuestionID>MemberLastName</apol:QuestionID><apol:AnswerList><apol:Answer><apol:Value>Doe</apol:Value></apol:Answer></apol:AnswerList></apol:PartQuestionAnswer><apol:PartQuestionAnswer><apol:QuestionID>Gender</apol:QuestionID><apol:AnswerList><apol:Answer><apol:LabelID>FEMALE</apol:LabelID></apol:Answer></apol:AnswerList></apol:PartQuestionAnswer><apol:PartQuestionAnswer><apol:QuestionID>DOB</apol:QuestionID><apol:AnswerList><apol:Answer><apol:Value>01/01/1911</apol:Value></apol:Answer></apol:AnswerList></apol:PartQuestionAnswer><apol:PartQuestionAnswer><apol:QuestionID>EmployeeIDNum</apol:QuestionID><apol:AnswerList><apol:Answer><apol:Value>35121212121212</apol:Value></apol:Answer></apol:AnswerList></apol:PartQuestionAnswer></apol:ParticipantAnswerList></apol:SaveAssessmentAnswersReq>--RESPONSE-->:<ns1:SaveAssessmentAnswersRes xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:ns1="http://www.domain.com/xsds/ApolloAssessment.xsd"> <ns1:TransactionID>3740e6fc-59ee-43f2-ba47-4630da0aaeda</ns1:TransactionID> <ns1:AdditionalDataList> <ns1:AdditionalData> <ns1:Name>Message</ns1:Name> <ns1:Value>We’re sorry, we’re not able to verify your account information. Please contact your benefits administrator.</ns1:Value> </ns1:AdditionalData> </ns1:AdditionalDataList> </ns1:SaveAssessmentAnswersRes> --REFERENCEID-->: 2018-09-25 12:47:21,248 [tp-bio-8001-exec-177] [ STANDARD] [ ] [ PHSInt:01.01] (og.Alere_FW_Apollo_Int_.Action) INFO hostname.domain.com|10.214.6.60|HTTP|AssessmentServices|Services|SaveAssessmentAnswers|A6E53D8C7F19456C1484D3F2307AB5FDB - INSERTING INTO SERVICE REQUEST LOG:--SERVICEREQUESTTYPE -->:SaveAssessmentAnswers--SERVICEREQUESTSTATUS -->:--TRANSACTIONID-->:a8667bd9-2be5-4655-8d9a-dd47e8111ce4--MEMBERELIGID-->:--PID-->:--PARTICIPANTID-->:--DEBUGMESSAGE-->:[hostname.domain.com] --REQUEST-->:<apol:SaveAssessmentAnswersReq xmlns:apol="http://www.domain.com/xsds/ApolloAssessment.xsd"><apol:TransactionID>axx67bd9-2be5-4655-8d9a-dd47e8111ce4</apol:TransactionID><apol:ClientID>11121212</apol:ClientID><apol:AssessmentTypeID>931</apol:AssessmentTypeID><apol:ParticipantAnswerList><apol:PartQuestionAnswer><apol:QuestionID>ImpersonatorDetail</apol:QuestionID><apol:AnswerList><apol:Answer><apol:LabelID>PARENT</apol:LabelID></apol:Answer></apol:AnswerList></apol:PartQuestionAnswer><apol:PartQuestionAnswer><apol:QuestionID>MentorFirstName</apol:QuestionID><apol:AnswerList><apol:Answer><apol:Value>Jane</apol:Value></apol:Answer></apol:AnswerList></apol:PartQuestionAnswer><apol:PartQuestionAnswer><apol:QuestionID>MentorLastName</apol:QuestionID><apol:AnswerList><apol:Answer><apol:Value>Doe</apol:Value></apol:Answer></apol:AnswerList></apol:PartQuestionAnswer><apol:PartQuestionAnswer><apol:QuestionID>MemberFirstName</apol:QuestionID><apol:AnswerList><apol:Answer><apol:Value>Aiden</apol:Value></apol:Answer></apol:AnswerList></apol:PartQuestionAnswer><apol:PartQuestionAnswer><apol:QuestionID>MemberLastName</apol:QuestionID><apol:AnswerList><apol:Answer><apol:Value>Doe</apol:Value></apol:Answer></apol:AnswerList></apol:PartQuestionAnswer><apol:PartQuestionAnswer><apol:QuestionID>Gender</apol:QuestionID><apol:AnswerList><apol:Answer><apol:LabelID>MALE</apol:LabelID></apol:Answer></apol:AnswerList></apol:PartQuestionAnswer><apol:PartQuestionAnswer><apol:QuestionID>DOB</apol:QuestionID><apol:AnswerList><apol:Answer><apol:Value>01/1/2001</apol:Value></apol:Answer></apol:AnswerList></apol:PartQuestionAnswer><apol:PartQuestionAnswer><apol:QuestionID>EmployeeIDNum</apol:QuestionID><apol:AnswerList><apol:Answer><apol:Value>351111111111</apol:Value></apol:Answer></apol:AnswerList></apol:PartQuestionAnswer></apol:ParticipantAnswerList></apol:SaveAssessmentAnswersReq>--RESPONSE-->:<ns1:SaveAssessmentAnswersRes xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:ns1="http://www.domain.com/xsds/ApolloAssessment.xsd"> <ns1:TransactionID>axx67bd9-2be5-4655-8d9a-dd47e8111ce4</ns1:TransactionID> <ns1:AdditionalDataList> <ns1:AdditionalData> <ns1:Name>Message</ns1:Name> <ns1:Value>We’re sorry, we’re not able to verify your account information. Please contact your benefits administrator.</ns1:Value> </ns1:AdditionalData> </ns1:AdditionalDataList> </ns1:SaveAssessmentAnswersRes> --REFERENCEID-->: Other Samples in the log that are not XML. 2018-09-252018-09-25 13:17:4613:17:46,,541541 [ [tp-bio-8004-exec-171tp-bio-8004-ex ] [ STANDARD] [ ] [ PHSInt:01.01] (lo_Data_System_BatchLog.Action) INFO hostname06.domain.com|10.200.200.200|HTTP|HealthIndicatorsInt|Services|saveHealthData|A30AC19E66FD562E79942068C75D03XXF - In UpdateBatchLog:ID=20001,Type=ProcessEvent,Action=P-212799085,Status=INFO,Message=Processing of EventNew HD,Exception= JSON I think. 2018-09-25 13:17:45,929 [ PegaRULES-Batch-18] [ STANDARD] [ ] [ ApolloCCBatch:01.01] (on.Domain_FW_Apollo_Int_.Action) INFO - INSERTING INTO SERVICE REQUEST LOG:--SERVICEREQUESTTYPE -->:MPEAPI--SERVICEREQUESTSTATUS -->:200--TRANSACTIONID-->:DOE--MEMBERELIGID-->:99999999--PID-->:999999999--PARTICIPANTID-->:JOHN--DEBUGMESSAGE-->:[hostname04.domain.com] OK [Time Elapsed=697.0ms]--REQUEST-->:{ "MemberProductEligibilityRequest":{ "requestHeader":{ "applicationName":"APPLICATION", "transactionId":"bc99999b547b64cf99a01cabd625e0bc7" }, "consumerDetails":{ "firstName":"JOHN", "lastName":"DOE", "dateOfBirth":"1900-05-09T00:00:00Z", "searchId":"999999999", "contractNumber":"999999" }, "filteringAttributes":{ "includeExtendedAttributes":"true", "applyFilters":"true" }, "requestDetails":{ "requestType":"BIG5", "searchType":"ALL" } }}--RESPONSE-->:{"MemberProductEligibilityResponse":{"responseHeader":{"transactionId":"bc2706b547b64cf99a01cabd625e0bc7"},"consumerDetails":[{"demographics":{**** Section suppressed for logging ****},"contactDetails":{**** Section suppressed for logging ****},"idSet":{**** Section suppressed for logging ****},"populationDetails":{"populationEffectiveDate":"2018-01-01T00:00:00Z","populationCancelDate":"9999-12-31T00:00:00Z","populationId":"POP33477","populationDateAssigned":"2017-12-12T00:00:00Z","populationBrandingType":"Optum Logo","populationBrandingEffectiveDate":"2018-01-01T00:00:00Z"},"coverageDetails":{"recordType":"HEALTH_COVERAGE","employeeStatus":"A","contractNumber":"0999999","eligibilitySourceSystem":"CS","planVariation":"0106","reportingCode":"0106","customerName":"TESLA","coverageType":"M","coverageEffectiveDate":"2018-01-01T00:00:00Z","hireDate":"2001-01-04T00:00:00Z","stateOfIssue":"CA","legalEntity1":"20020","marketSite":"0004422"},"extendedAttributes":{"ecExtended":[],"elExtended":[],"euExtended":[{"typeCode":"EU3","value":"0004422","effectiveDate":"2001-01-01T00:00:00Z","cancelDate":"9999-12-31T00:00:00Z"},{"typeCode":"EU3","value":"0004422","effectiveDate":"2001-01-01T00:00:00Z","cancelDate":"9999-12-31T00:00:00Z"}],"cuExtended":[],"suExtended":[],"muExtended":[]},"productDetails":{"product":[{"source":"Optum","productEvent1":"Productname for Life","productEffectiveDate":"2018-01-01T00:00:00Z","productTerminationDate":"2199-12-31T00:00:00Z"}]}}]}}--REFERENCEID-->:999999

JDukeSplunk · ‎09-20-2018

You should read or watch this excellent session from .conf 2017 - it was a very well attended session. This will give you a best practice syslog server to collect the logs: http://conf.splunk.com/sessions/2017-sessions.html#search=critical%20syslog%20tricks& https://conf.splunk.com/files/2017/slides/the-critical-syslog-tricks-that-no-one-seems-to-know-about.pdf

JDukeSplunk · ‎09-20-2018

So... I would do something like this [monitor://C:\path\*(log$|txt$|csv$)] disabled = 0 recurse = false followTail = 0 sourcetype=YOURSOURCETYPENAME ignoreOlderThan = 7d index = YOURINDEX Or specify file names. These should no recurse down the sub folders since you are specifying exact file names, or partial file names with wildcards. Filename_9-20-18.log would be written like Filename_.log$*. The $ tells it to not pickup files like Filename_DATE.log.gz [monitor://C:\path\sourcetypeA*.file_extension$] disabled = 0 followTail = 0 sourcetype=sourcetypeA ignoreOlderThan = 7d index = YOURINDEX [monitor://C:\path\sourcetypeB*.file_extension$] disabled = 0 followTail = 0 sourcetype=sourcetypeB ignoreOlderThan = 7d index = YOURINDEX Linux paths look a little different, but the rest is about the same. [monitor:///opt/apache-tomcat/*/logs/catalina.out] Will grab catalina.out from /opt/apache-tomcat/8.1/logs AND /opt/apache-tomcat/8.2/logs

JDukeSplunk · ‎09-20-2018

Rex puts your data wrecks in check |eval lower(ALL_CAPS_CHATTERS) Splunk - The Future is relative_time(now(),"+1s@s")

JDukeSplunk · ‎09-20-2018

You may have to experiment with the sed string a little to get it to work. I'm no master at it myself. I tried this on my searches, which worked. | rex mode=sed field=_raw "s/\\\//g" Search snippet before sed. (category:"/Newsletter/Impact\+letter")* And After. (category:"/Newsletter/Impact+letter")

JDukeSplunk · ‎09-18-2018

You could run a SEDCMD in props.conf that log file name. Something like this to replace any backslash with nothing. [source::..path/to/logb/logb.log] SEDCMD-logbslash = s/\\//g Then in theory the field extractions for Log A would work You can test the sed command in search with a |rex mode=sed http://docs.splunk.com/Documentation/SplunkCloud/latest/SearchReference/Rex

JDukeSplunk · ‎09-18-2018

Have you tried |spath or |xmlkv? http://docs.splunk.com/Documentation/SplunkCloud/latest/SearchReference/Xmlkv

JDukeSplunk · ‎09-07-2018

Yours worked in 99% of the events. What I finally ended up with was a really long regex. BREAK_ONLY_BEFORE = \w{3}\s\d{1,2}\s\d{1,2}\:\d{1,2}\:\d{1,2}\s\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\s\[

JDukeSplunk · ‎08-21-2018

It sounds like a job for a lookup table. I don't know if you can do ranges in a lookup table.. https://docs.splunk.com/Documentation/Splunk/7.1.2/Knowledge/LookupexampleinSplunkWeb You may have to have a line entry for each port in the csv file to get it working. Not the best solution, but it is simple and might be the only solution. So your csv lookup file might look like... And with excel if you enter two cells with consecutive numbers, you can click the corner and drag down to populate up to the number you like. port,servicename 1024,ServiceA 1025,ServiceA 1026,ServiceA 1027,ServiceA 1028,ServiceA 1029,ServiceA 1030,ServiceA 1031,ServiceA 1032,ServiceA etc... Then you can either do an inline |inputlookup or do an automatic search that will create a new field called "ServiceName" or whatever.

JDukeSplunk · ‎08-21-2018

I think the posted answer will show saved searches, and not typed searches. I use this one, which is basically the same search as the answer index=_audit action=search info=granted search=* NOT "search_id='scheduler" NOT "search='|history" NOT "user=splunk-system-user" NOT "search='typeahead" NOT "search='| metadata type=* | search totalCount>1" | stats count by user search

Posts	240
Solutions	24
Karma Given	19
Karma Received	61
Member Since	‎07-28-2015

Online Status	Offline
Date Last Visited	‎09-29-2022 02:03 PM

Tstats datamodel combine three sources by common f...

Some time questions.

Use lookup table to populate otherwise empty rows

Getting uptime of a process from PS

Field Extraction regex, stop at word or $

indexes.conf sanity question.

8.0.1 upgraded Heavy Forwarder- TcpOutputProc - Po...

Nest App, Google Migrated Account, 406 errors.

DB Connect, MSSQL Availability Group, Read Only In...

Need SEDCMD Help.

Re: tstats latest time for devices in a lookup

Re: Determine actual forwarder host

Re: No file field in IIS logs

Re: CSV Source not getting field headers.

Re: CSV Source not getting field headers.

CSV Source not getting field headers.

Re: SPlunk search not returning all events

Re: How to get sid for particular job, when i try ...

Re: How come the LDAP config is not picking up use...

Re: Is there a way to run a PowerShell script loca...

Re: Can you help me with my Windows Management Ins...

Re: How do I monitor input on Windows machine with...

Re: Is there a way to run a PowerShell script loca...

Re: Can you help me with an XML field extraction?

Re: Can you help me with an XML field extraction?

Can you help me with an XML field extraction?

Re: Why did Splunk stop collecting syslog logs?

Re: How do I blacklist sub-directories in the inpu...

Re: What are your Splunk t-shirt ideas?

Re: Can you help me delete the '\' character after...

Re: Can you help me delete the '\' character after...

Re: How do you extract the following XML fields?

Re: Line Breaks

Re: Automatic lookup, matching range field?

Re: how to know the search history by user, but on...

Are you a member of the Splunk Community?