About DarthHerm

DarthHerm

The issue has been identified. When the other agency pushed SplunkForwarder 9.2.6.0 to my hosts, NT SERVICE/SplunkForwarder was removed as a member of the performance monitor users’ group. That agency used Ivanti Patch for Endpoint Manager to push the updates. With one of the hosts on 9.2.6.0, I kicked off the repair to SplunkForwarder and perfmon counters started to come in at the interval that was set for that host. I next moved to a powershell command to add NT SERVICE/SplunkForwarder back as a member of the performance monitor users group. I asked the tech for a copy of the syntax used to push SplunkForwarder to my hosts to go over and validate. I’m asking support about it too, has there been any known issues with Ivanti pushing SplunkForwarder updates?

DarthHerm

Thought I would post here in the community as well since I have this opened with support. A couple weeks ago, another agency pushed updates to Splunk Universal Forwarder to half of my hosts without my knowledge or consent. Those hosts were updated to 9.2.6.0 from 9.2.0.1. The updates went unnoticed for a couple weeks since the events from our custom application and Event Viewer continued to get indexed. I started to notice an issue on one dashboard where no perfmon events were coming in. I reviewed another dashboard that checks the status of my forwarders and that's where I saw the updated installs. I went over the index the perfmon counters go to and validated only the hosts that were using Universal Forwarder 9.2.0.1 were coming in. My version of Enterprise was 9.2.1.0 and support recommended I update Enterprise to a newer version. After some testing, I went to Enterprise 9.3.5.0. Not ready for 9.4.X with trying to update the kvstore. Reviewing the Universal Forwarder compatibility matrix, I've kept my Universal Forwarders on 9.2.0.1, 9.2.6.0, and two were updated to 9.3.5.0. Updating Enterprise didn't correct the issue. I went through troubleshooting on the host looking over the config files. I did a rebuild of the resource counters and restarted the splunk forwarder service on one of the hosts using forwarder 9.2.6.0. I've looking at one of the hosts by adding the service account used as a local member of the administrators and Remote Management Users groups, adding a path variable for SPLUNK_HOME at "c:\program files\splunkuniversalforwarder". Chatted with the tech who pushed universal forwarder and they're not going to do that again. The hosts that got updated are members of my custom applications lower environments. I can live without the perfmon counters in the lower environments and none of my hosts in our production environment were updated. I know if I uninstall Forwarder and reinstall 9.2.0.1, the perfmon counters will resume coming in. Convinced its a change I need to do and thought I would check with the community who have updated their forwarders. I attached a copy of the inputs.conf from one of my hosts which is the same for all of them (aside from the environment name)

DarthHerm · ‎05-01-2025

I'm continuing to work on dashboards to report on user activity on our application. Going through the knowledgebase, bootcamp slides, and google, trying to determine the best route to report on the values in logs files such as this one. The dashboards I am creating is showing activity in the various modules, what values are getting select and what is being pulled up. I looked at spath and mvexpand and wasn't getting the results I was hoping for, might have been I wasn't formatting the search correctly and also how green myself and work is to Splunk. Creating field extractions has worked for the most part to pull the specific values I wanted to report but further on, I'm finding incorrect values being pulled in. Below is one such event that's been sanitized and it's in valid JSON format. I'm trying to do a table event showing the userName, date and time, serverHost, SparklingTypeId, PageSize, and PageNumber. The other values not so much. Is spath and MV expand along with eval statements the best course? I was using field extractions in a couple other modules but then found incorrect values were being added. {"auditResultSets":null,"schema":"com","storedProcedureName":"SpongeGetBySearchCriteria","commandText":"com.SpongeGetBySearchCriteria","Locking":null,"commandType":4,"parameters":[{"name":"@RETURN_VALUE","value":0},{"name":"@SpongeTypeId","value":null},{"name":"@CustomerNameStartWith","value":null},{"name":"@IsAssigned","value":null},{"name":"@IsAssignedToIdIsNULL","value":false},{"name":"@SpongeStatusIdsCSV","value":",1,"},{"name":"@RequestingValueId","value":null},{"name":"@RequestingStaffId","value":null},{"name":"@IsParamOther","value":false},{"name":"@AssignedToId","value":null},{"name":"@MALLLocationId","value":8279},{"name":"@AssignedDateFrom","value":null},{"name":"@AssignedDateTo","value":null},{"name":"@RequestDateFrom","value":null},{"name":"@RequestDateTo","value":null},{"name":"@DueDateFrom","value":null},{"name":"@DueDateTo","value":null},{"name":"@ExcludeCustomerFlagTypeIdsCSV","value":",1,"},{"name":"@PageSize","value":25},{"name":"@PageNumber","value":1},{"name":"@SortColumnName","value":"RequestDate"},{"name":"@SortDirection","value":"DESC"},{"name":"@HasAnySparkling","value":null},{"name":"@SparklingTypeId","value":null},{"name":"@SparklingSubTypeId","value":null},{"name":"@SparklingStatusId","value":null},{"name":"@SparklingDateFrom","value":null},{"name":"@SparklingDateTo","value":null},{"name":"@SupervisorId","value":null},{"name":"@Debug","value":null}],"serverIPAddress":"255.255.000.000","serverHost":"WEBSERVER","clientIPAddress":"255.255.255.255","sourceSystem":"WebSite","module":"Vendor.Product.BLL.Community","accessDate":"2025-04-30T15:34:33.3568918-06:00","userId":3231,"userName":"PeterVenkman","traceInformation":[{"type":"Page","class":"Vendor.Product.Web.UI.Website.Community.Operations.SpongeSearch","method":"Page_Load"},{"type":"Manager","class":"Vendor.Product.BLL.Community.SpongeManager","method":"SpongeSearch"}]}

DarthHerm · ‎04-04-2025

I pulled up the original the events and then also downloaded a different external document from our system to generate those logs. I trimmed it down from the 12 events to 3. I reduced the sanitizing to help out. In the dashboards the DocumentTypeId is where I am starting with because that identifies which module the file is located in our application. The DocumentId is the SQL document Id number assigned to the file. Lastly the DocumentFileTypeId identifies the file format. I'm also looking at leverage the DBConnect add-on. I'm also looking at that as an option and use the DocumentId for the search instead. { [-] Locking: null accessDate: 2025-03-21T16:37:14.8614186-06:00 auditResultSets: null clientIPAddress: 255.255.255.255 commandText: ref.DocumentFileTypeGetById commandType: 4 module: Vendor.PRODUCT.BLL.DocumentManagement parameters: [ [-] { [-] name: @RETURN_VALUE value: 0 } { [-] name: @DocumentFileTypeId value: 7 } ] schema: ref serverHost: Webserver serverIPAddress: 255.255.255.255 sourceSystem: WebSite storedProcedureName: DocumentFileTypeGetById traceInformation: [ [-] { [-] class: Vendor.PRODUCT.Web.UI.Website.DocumentManagement.ViewDocument method: Page_Load type: Page } { [-] class: Vendor.PRODUCT.BLL.DocumentManagement.DocumentManager method: Get type: Manager } ] userId: UserNumber userName: Username }{ [-] Locking: null accessDate: 2025-03-21T16:37:14.8614186-06:00 auditResultSets: null clientIPAddress: 255.255.255.255 commandText: ref.DocumentFileTypeGetById commandType: 4 module: Vendor.PRODUCT.BLL.DocumentManagement parameters: [ [-] { [-] name: @RETURN_VALUE value: 0 } { [-] name: @DocumentFileTypeId value: 7 } ] schema: ref serverHost: Webserver serverIPAddress: 255.255.255.255 sourceSystem: WebSite storedProcedureName: DocumentFileTypeGetById traceInformation: [ [-] { [-] class: Vendor.PRODUCT.Web.UI.Website.DocumentManagement.ViewDocument method: Page_Load type: Page } { [-] class: Vendor.PRODUCT.BLL.DocumentManagement.DocumentManager method: Get type: Manager } ] userId: UserNumber userName: Username } { [-] Locking: null accessDate: 2025-03-21T16:37:14.8614186-06:00 auditResultSets: null clientIPAddress: 255.255.255.255 commandText: ref.DocumentAttributeGetByDocumentTypeId commandType: 4 module: Vendor.PRODUCT.BLL.DocumentManagement parameters: [ [-] { [-] name: @RETURN_VALUE value: 0 } { [-] name: @DocumentTypeId value: 92 } { [-] name: @IncludeInactive value: false } ] schema: ref serverHost: Webserver serverIPAddress: 255.255.255.255 sourceSystem: WebSite storedProcedureName: DocumentAttributeGetByDocumentTypeId traceInformation: [ [-] { [-] class: Vendor.PRODUCT.Web.UI.Website.DocumentManagement.ViewDocument method: Page_Load type: Page } { [-] class: Vendor.PRODUCT.BLL.DocumentManagement.DocumentManager method: Get type: Manager } ] userId: UserNumber userName: Username }

DarthHerm · ‎03-31-2025

Yes the data has a valid json in for these events. For the post, I exported the raw events and sanitized it for the post in the community. If that wasn't the best way to go about it, let me know. I'm only starting to post on the Splunk community. Wen I ran the extended search, the one event that ties them all together is return_value with a value of 0. Try running the stats on return_value?

DarthHerm · ‎03-28-2025

This has been scratching my head. I'm working on dashboards on user activity on our application. Multiple dashboards I am working I'm trying to report when an external document was downloaded that is tied to the specific module of our application. Our application utilizes uses SQL. When I download an external document in our application, it generates a total of 12 separate events. I set up field extractions on the DocumentId, DocumentTypeId, and DocmentFileTypeId. What I am trying to do with the 12 separate events is to show in one event with the DocumentTypeId DocumentId and DocumentFileTypeId (not as much needed). The DocumentTypeId is my primary event since it's tied to the dashboard for the specific module followed by the DocumentId. The DocumentTypeId tells me what the file is pertaining to the system, DocumentFileTypeId if it's a PDF or a different type, and the DocumentTypeId is the document number in SQL. For the time, I am using _time instead of accessDate in the events. All 12 was from the single download even with the three accessDate times. Tried a few different commands and searches now and just can't seem to get it to report correctly. When I think I have it correct, it's not the case when I expand the search range where for example all my Document Ids would also list the same DocumentTypeId which I verified against the DB to be incorrect when I used the join command. This is the raw events that were exported and sanitized for the post. The events have a valid _json. T {"auditResultSets":null,"schema":"ref","storedProcedureName":"DocumentFileTypeGetById","commandText":"ref.DocumentFileTypeGetById","Locking":null,"commandType":4,"parameters":[{"name":"@RETURN_VALUE","value":0},{"name":"@DocumentFileTypeId","value":7}],"serverIPAddress":"000.000.000.000","serverHost":"Webserver","clientIPAddress":"000.000.000.000","sourceSystem":"WebSite","module":"Vendor.Product.BLL.DocumentManagement","accessDate":"2025-03-21T16:37:14.8614186-06:00","userId":0000,"userName":"username","traceInformation":[{"type":"Page","class":"Vendor.Product.Web.UI.Website.DocumentManagement.ViewDocument","method":"Page_Load"},{"type":"Manager","class":"Vendor.Product.BLL.DocumentManagement.DocumentManager","method":"Get"}]} {"auditResultSets":null,"schema":"ref","storedProcedureName":"DocumentAttributeGetByDocumentTypeId","commandText":"ref.DocumentAttributeGetByDocumentTypeId","Locking":null,"commandType":4,"parameters":[{"name":"@RETURN_VALUE","value":0},{"name":"@DocumentTypeId","value":00},{"name":"@IncludeInactive","value":false}],"serverIPAddress":"000.000.000.000","serverHost":"Webserver","clientIPAddress":"000.000.000.000","sourceSystem":"WebSite","module":"Vendor.Product.BLL.DocumentManagement","accessDate":"2025-03-21T16:37:14.8614186-06:00","userId":0000,"userName":"username","traceInformation":[{"type":"Page","class":"Vendor.Product.Web.UI.Website.DocumentManagement.ViewDocument","method":"Page_Load"},{"type":"Manager","class":"Vendor.Product.BLL.DocumentManagement.DocumentManager","method":"Get"}]} {"auditResultSets":null,"schema":"ref","storedProcedureName":"DocumentDetailGetByParentId","commandText":"ref.DocumentDetailGetByParentId","Locking":null,"commandType":4,"parameters":[{"name":"@RETURN_VALUE","value":0},{"name":"@DocumentId","value":000000}],"serverIPAddress":"000.000.000.000","serverHost":"Webserver","clientIPAddress":"000.000.000.000","sourceSystem":"WebSite","module":"Vendor.Product.BLL.DocumentManagement","accessDate":"2025-03-21T16:37:14.8614186-06:00","userId":0000,"userName":"username","traceInformation":[{"type":"Page","class":"Vendor.Product.Web.UI.Website.DocumentManagement.ViewDocument","method":"Page_Load"},{"type":"Manager","class":"Vendor.Product.BLL.DocumentManagement.DocumentManager","method":"Get"}]} {"auditResultSets":null,"schema":"ref","storedProcedureName":"DocumentStatusHistoryGetByFK","commandText":"ref.DocumentStatusHistoryGetByFK","Locking":null,"commandType":4,"parameters":[{"name":"@RETURN_VALUE","value":0},{"name":"@DocumentVersionId","value":000000},{"name":"@IncludeInactive","value":""}],"serverIPAddress":"000.000.000.000","serverHost":"Webserver","clientIPAddress":"000.000.000.000","sourceSystem":"WebSite","module":"Vendor.Product.BLL.DocumentManagement","accessDate":"2025-03-21T16:37:14.8614186-06:00","userId":0000,"userName":"username","traceInformation":[{"type":"Page","class":"Vendor.Product.Web.UI.Website.DocumentManagement.ViewDocument","method":"Page_Load"},{"type":"Manager","class":"Vendor.Product.BLL.DocumentManagement.DocumentManager","method":"Get"}]} {"auditResultSets":null,"schema":"ref","storedProcedureName":"DocumentVersionGetByFK","commandText":"ref.DocumentVersionGetByFK","Locking":null,"commandType":4,"parameters":[{"name":"@RETURN_VALUE","value":0},{"name":"@DocumentId","value":000000}],"serverIPAddress":"000.000.000.000","serverHost":"Webserver","clientIPAddress":"000.000.000.000","sourceSystem":"WebSite","module":"Vendor.Product.BLL.DocumentManagement","accessDate":"2025-03-21T16:37:14.8614186-06:00","userId":0000,"userName":"username","traceInformation":[{"type":"Page","class":"Vendor.Product.Web.UI.Website.DocumentManagement.ViewDocument","method":"Page_Load"},{"type":"Manager","class":"Vendor.Product.BLL.DocumentManagement.DocumentManager","method":"Get"}]} {"auditResultSets":null,"schema":"ref","storedProcedureName":"DocumentLinkGetByFK","commandText":"ref.DocumentLinkGetByFK","Locking":null,"commandType":4,"parameters":[{"name":"@RETURN_VALUE","value":0},{"name":"@DocumentId","value":000000}],"serverIPAddress":"000.000.000.000","serverHost":"Webserver","clientIPAddress":"000.000.000.000","sourceSystem":"WebSite","module":"Vendor.Product.BLL.DocumentManagement","accessDate":"2025-03-21T16:37:14.8614186-06:00","userId":0000,"userName":"username","traceInformation":[{"type":"Page","class":"Vendor.Product.Web.UI.Website.DocumentManagement.ViewDocument","method":"Page_Load"},{"type":"Manager","class":"Vendor.Product.BLL.DocumentManagement.DocumentManager","method":"Get"}]} {"auditResultSets":null,"schema":"ref","storedProcedureName":"DocumentGetById","commandText":"ref.DocumentGetById","Locking":null,"commandType":4,"parameters":[{"name":"@RETURN_VALUE","value":0},{"name":"@DocumentId","value":000000}],"serverIPAddress":"000.000.000.000","serverHost":"Webserver","clientIPAddress":"000.000.000.000","sourceSystem":"WebSite","module":"Vendor.Product.BLL.DocumentManagement","accessDate":"2025-03-21T16:37:14.8457543-06:00","userId":0000,"userName":"username","traceInformation":[{"type":"Page","class":"Vendor.Product.Web.UI.Website.DocumentManagement.ViewDocument","method":"Page_Load"},{"type":"Manager","class":"Vendor.Product.BLL.DocumentManagement.DocumentManager","method":"Get"}]} {"auditResultSets":null,"schema":"ref","storedProcedureName":"DocumentFileTypeGetById","commandText":"ref.DocumentFileTypeGetById","Locking":null,"commandType":4,"parameters":[{"name":"@RETURN_VALUE","value":0},{"name":"@DocumentFileTypeId","value":7}],"serverIPAddress":"000.000.000.000","serverHost":"Webserver","clientIPAddress":"000.000.000.000","sourceSystem":"WebSite","module":"Vendor.Product.BLL.DocumentManagement","accessDate":"2025-03-21T16:37:14.736377-06:00","userId":0000,"userName":"username","traceInformation":[{"type":"Page","class":"Vendor.Product.Web.UI.Website.DocumentManagement.DocumentManagementMain","method":"ViewDocument"},{"type":"Manager","class":"Vendor.Product.BLL.DocumentManagement.DocumentManager","method":"GetLatestDocumentwithoutAttributes"}]} {"auditResultSets":null,"schema":"ref","storedProcedureName":"DocumentStatusHistoryGetByFK","commandText":"ref.DocumentStatusHistoryGetByFK","Locking":null,"commandType":4,"parameters":[{"name":"@RETURN_VALUE","value":0},{"name":"@DocumentVersionId","value":000000},{"name":"@IncludeInactive","value":""}],"serverIPAddress":"000.000.000.000","serverHost":"Webserver","clientIPAddress":"000.000.000.000","sourceSystem":"WebSite","module":"Vendor.Product.BLL.DocumentManagement","accessDate":"2025-03-21T16:37:14.736377-06:00","userId":0000,"userName":"username","traceInformation":[{"type":"Page","class":"Vendor.Product.Web.UI.Website.DocumentManagement.DocumentManagementMain","method":"ViewDocument"},{"type":"Manager","class":"Vendor.Product.BLL.DocumentManagement.DocumentManager","method":"GetLatestDocumentwithoutAttributes"}]} {"auditResultSets":null,"schema":"ref","storedProcedureName":"DocumentVersionGetByFK","commandText":"ref.DocumentVersionGetByFK","Locking":null,"commandType":4,"parameters":[{"name":"@RETURN_VALUE","value":0},{"name":"@DocumentId","value":000000}],"serverIPAddress":"000.000.000.000","serverHost":"Webserver","clientIPAddress":"000.000.000.000","sourceSystem":"WebSite","module":"Vendor.Product.BLL.DocumentManagement","accessDate":"2025-03-21T16:37:14.736377-06:00","userId":0000,"userName":"username","traceInformation":[{"type":"Page","class":"Vendor.Product.Web.UI.Website.DocumentManagement.DocumentManagementMain","method":"ViewDocument"},{"type":"Manager","class":"Vendor.Product.BLL.DocumentManagement.DocumentManager","method":"GetLatestDocumentwithoutAttributes"}]} {"auditResultSets":null,"schema":"ref","storedProcedureName":"DocumentLinkGetByFK","commandText":"ref.DocumentLinkGetByFK","Locking":null,"commandType":4,"parameters":[{"name":"@RETURN_VALUE","value":0},{"name":"@DocumentId","value":000000}],"serverIPAddress":"000.000.000.000","serverHost":"Webserver","clientIPAddress":"000.000.000.000","sourceSystem":"WebSite","module":"Vendor.Product.BLL.DocumentManagement","accessDate":"2025-03-21T16:37:14.736377-06:00","userId":0000,"userName":"username","traceInformation":[{"type":"Page","class":"Vendor.Product.Web.UI.Website.DocumentManagement.DocumentManagementMain","method":"ViewDocument"},{"type":"Manager","class":"Vendor.Product.BLL.DocumentManagement.DocumentManager","method":"GetLatestDocumentwithoutAttributes"}]} {"auditResultSets":null,"schema":"ref","storedProcedureName":"DocumentGetById","commandText":"ref.DocumentGetById","Locking":null,"commandType":4,"parameters":[{"name":"@RETURN_VALUE","value":0},{"name":"@DocumentId","value":000000}],"serverIPAddress":"000.000.000.000","serverHost":"Webserver","clientIPAddress":"000.000.000.000","sourceSystem":"WebSite","module":"Vendor.Product.BLL.DocumentManagement","accessDate":"2025-03-21T16:37:14.736377-06:00","userId":0000,"userName":"username","traceInformation":[{"type":"Page","class":"Vendor.Product.Web.UI.Website.DocumentManagement.DocumentManagementMain","method":"ViewDocument"},{"type":"Manager","class":"Vendor.Product.BLL.DocumentManagement.DocumentManager","method":"GetLatestDocumentwithoutAttributes"}]}

DarthHerm · ‎03-07-2025

Please disregard. Overthinking the issue and looking at setting up some field extractions.

DarthHerm · ‎03-07-2025

Both my work and myself are new to Splunk and I'm developing some reports and dashboards for one of our applications. This one dashboard I am working on includes a table of events showing when some reports are downloaded. The log file's sourcetype is _json_AuditFramework. I'm looking to include the parameters name @documentId and it's corresponding value to place into a table. Right now, the table syntax lists the parameters{}.value and when there's multiple parameters{}.name and parameters{}.values in the log, they will all show in the table. Pending on the report, I'm including trace information as well and it's the same thing as the parameters. I haven't had luck with similar posts I found. {"auditResultSets":null,"schema":"ref","storedProcedureName":"DocumentGetById","commandText":"ref.DocumentGetById","Locking":null,"commandType":4,"parameters":[{"name":"@RETURN_VALUE","value":0},{"name":"@DocumentId","value":123123}],"serverIPAddress":"100.100.100.100","serverHost":"WEBSERVER","clientIPAddress":"101.101.101.101","sourceSystem":"WebSite","module":"Vendor.PRODUCT.BLL.DocumentManagement","accessDate":"2025-03-06T17:26:47.4112974-07:00","userId":0000,"userName":"username","traceInformation":[{"type":"Page","class":"Vendor.PRODUCT.Web.UI.Website.DocumentManagement.ViewDocument","method":"Page_Load"},{"type":"Manager","class":"Vendor.PRODUCT.BLL.DocumentManagement.DocumentManager","method":"Get"}]} Show syntax highlighted host = WEBSERVER source = Logfile path sourcetype = _json_AuditFramework

Posts	8
Solutions	0
Karma Given	4
Karma Received	1
Member Since	‎11-15-2024

Online Status	Offline
Date Last Visited	Monday

Perfmon counters not coming in after Universal For...

Best way to report specific multiple values from a...

Combining multiple events from downloading externa...

Specific parameters and trace information in visua...

Re: Perfmon counters not coming in after Universal...

Perfmon counters not coming in after Universal For...

Best way to report specific multiple values from a...

Re: Combining multiple events from downloading ext...

Re: Combining multiple events from downloading ext...

Combining multiple events from downloading externa...

Re: Specific parameters and trace information in v...

Specific parameters and trace information in visua...

Are you a member of the Splunk Community?