About mxanareckless

mxanareckless · ‎11-09-2021

These are dashboards which are scheduled to be emailed out at 8am every morning. The dashboard widgets contain data from the previous 24 hours. Occasionally, certain panels in the emailed report will show "No results found.". If I go to the dashboard immediately after it is emailed out and this message is displayed, the results are populated within seconds, no problem. It seems for some reason the results are not making it to the report. Again: within seconds of the emailed report being sent out with the "No results found", the dashboard can be run manually and the results will populate quickly. So it appears the results are available at the time of report generation. It is not always the same panel that displays "No results found".

mxanareckless · ‎07-07-2021

1. There will be 2 separate charts: CPU usage by process, and RAM usage by process. 2. Sometimes more than one instance of a process is running. For example: there can be 2 splunkd processes, one using 170M and the other using 65M; in the chart I'd like this represented as 1 splunkd and the total of 235M between the 2 splunkd processes. 3. I'd like an overlay, an additional line on the timechart that shows the total RAM/CPU consumed on the server itself. See below screenshots of the search I have constructed so far, and the printout of top on the server to demonstrate the presence of several processes by the same name, that I'd like to aggregate in the timechart's results.

mxanareckless · ‎04-01-2021

Here are the contents of the /opt/splunk/etc/deployment-apps: [splunk@splunk deployment-apps]$ ls README Splunk_TA_nix Splunk_TA_windows [splunk@splunk deployment-apps]$ pwd /opt/splunk/etc/deployment-apps However upon running either splunk reload deploy-server or splunk restart, while no errors are being generated, the deployment apps displayed on the web UI are not updating: The entire path /opt/splunk/etc/deployment-apps and its subfolders are owned by the splunk user. Here are the results when running splunk reload deploy-server -debug: [splunk@splunk deployment-apps]$ splunk reload deploy-server -debug > reload Will setenv SPLUNK_CLI_DEBUG to "v". In check_and_set_splunk_os_user(): In env found SPLUNK_OS_USER var: "splunk" In cmd_rc Will exec (detach=no): LDAPCONF=/disk2/splunk/etc/openldap/ldap.conf HOSTNAME=splunk.cdc.health.local SPLUNK_SERVER_NAME=Splunkd SPLUNK_HOME=/disk2/splunk USER=splunk LD_LIBRARY_PATH=/disk2/splunk/lib:/disk2/splunk/bin/jars/vendors/java/OpenJDK8U-jre_x64_linux_hotspot_8u242b08/lib/amd64:/disk2/splunk/bin/jars/vendors/java/OpenJDK8U-jre_x64_linux_hotspot_8u242b08/lib/amd64/jli PATH=/disk2/splunk/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/splunk/.local/bin:/home/splunk/bin:/disk2/splunk/bin PWD=/opt/splunk/etc/deployment-apps SPLUNK_DB=/export/intindx NODE_PATH=/disk2/splunk/lib/node_modules SPLUNK_OS_USER=splunk /disk2/splunk/bin/splunkd btool web list settings --no-log Will exec (detach=no): LDAPCONF=/disk2/splunk/etc/openldap/ldap.conf HOSTNAME=splunk.cdc.health.local SPLUNK_SERVER_NAME=Splunkd SPLUNK_HOME=/disk2/splunk USER=splunk LD_LIBRARY_PATH=/disk2/splunk/lib:/disk2/splunk/bin/jars/vendors/java/OpenJDK8U-jre_x64_linux_hotspot_8u242b08/lib/amd64:/disk2/splunk/bin/jars/vendors/java/OpenJDK8U-jre_x64_linux_hotspot_8u242b08/lib/amd64/jli PATH=/disk2/splunk/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/splunk/.local/bin:/home/splunk/bin:/disk2/splunk/bin PWD=/opt/splunk/etc/deployment-apps SPLUNK_DB=/export/intindx NODE_PATH=/disk2/splunk/lib/node_modules SPLUNK_OS_USER=splunk /disk2/splunk/bin/splunkd btool server list general --no-log Will exec (detach=no): LDAPCONF=/disk2/splunk/etc/openldap/ldap.conf HOSTNAME=splunk.cdc.health.local SPLUNK_SERVER_NAME=Splunkd SPLUNK_HOME=/disk2/splunk USER=splunk LD_LIBRARY_PATH=/disk2/splunk/lib:/disk2/splunk/bin/jars/vendors/java/OpenJDK8U-jre_x64_linux_hotspot_8u242b08/lib/amd64:/disk2/splunk/bin/jars/vendors/java/OpenJDK8U-jre_x64_linux_hotspot_8u242b08/lib/amd64/jli PATH=/disk2/splunk/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/splunk/.local/bin:/home/splunk/bin:/disk2/splunk/bin PWD=/opt/splunk/etc/deployment-apps SPLUNK_DB=/export/intindx NODE_PATH=/disk2/splunk/lib/node_modules SPLUNK_OS_USER=splunk /disk2/splunk/bin/splunkd btool server list kvstore --no-log Will exec (detach=no): LDAPCONF=/disk2/splunk/etc/openldap/ldap.conf HOSTNAME=splunk.cdc.health.local SPLUNK_SERVER_NAME=Splunkd SPLUNK_HOME=/disk2/splunk USER=splunk LD_LIBRARY_PATH=/disk2/splunk/lib:/disk2/splunk/bin/jars/vendors/java/OpenJDK8U-jre_x64_linux_hotspot_8u242b08/lib/amd64:/disk2/splunk/bin/jars/vendors/java/OpenJDK8U-jre_x64_linux_hotspot_8u242b08/lib/amd64/jli PATH=/disk2/splunk/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/splunk/.local/bin:/home/splunk/bin:/disk2/splunk/bin PWD=/opt/splunk/etc/deployment-apps SPLUNK_DB=/export/intindx NODE_PATH=/disk2/splunk/lib/node_modules SPLUNK_OS_USER=splunk /disk2/splunk/bin/splunkd btool server list dfs --no-log dynamic conf init: httpport=8000 mgmtHostPort=8089 bind_ip= rest_hostname=127.0.0.1 kvStorePort=8191 base splunkdURL from splunkd: https://127.0.0.1:8089 Will setenv SPLUNK_CLI_DEBUG to "v". cmd: reload obj: deploy-server cli args: In build_full_rest_url(): Composing URL from base=https://127.0.0.1:8089 + relative=/static/splunkrc_cmds.xml In build_full_rest_url(): Composed URL=https://127.0.0.1:8089/static/splunkrc_cmds.xml In make_simple_rest_call_online(): using_basic_auth=0 In make_simple_rest_call_online(): [Re-]Initialized HTTP request headers: <none> In make_simple_rest_call_online(): HTTP request response_code=200 In make_simple_rest_call_online(): HTTP response headers: ETag: "s-t2y1XwAAAAD4ZQsAAAAAAJ8CCgAAAAAANsQDAAAAAAA" Date: Thu, 01 Apr 2021 20:44:37 GMT Expires: Thu, 01 Apr 2021 21:44:37 GMT Cache-Control: max-age=3600 Content-Type: text/xml X-Content-Type-Options: nosniff Last-Modified: Wed, 18 Nov 2020 18:49:27 GMT Content-Length: 246838 Connection: Keep-Alive X-Frame-Options: SAMEORIGIN Server: Splunkd Online REST call to /static/splunkrc_cmds.xml returned 0 Visited node: deploy-server Entered node:deploy-server/common Entered node:deploy-server/cmd Metadata: cmd: reload obj: deploy-server rel_url: (no value) eai_id: (no value) implied_arg_name: (no value) action: 0 args map count: 0 args map: default args count: 0 initial args count: 0 hooks count: 0 object help command array count: 0 In call_pcl, will exec_python Will exec (detach=no): HOSTNAME=splunk.cdc.health.local USER=splunk PATH=/disk2/splunk/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/splunk/.local/bin:/home/splunk/bin:/disk2/splunk/bin PWD=/opt/splunk/etc/deployment-apps SPLUNK_HOME=/disk2/splunk SPLUNK_DB=/export/intindx SPLUNK_SERVER_NAME=Splunkd SPLUNK_OS_USER=splunk NODE_PATH=/disk2/splunk/lib/node_modules PYTHONPATH= LD_LIBRARY_PATH=/disk2/splunk/lib:/disk2/splunk/bin/jars/vendors/java/OpenJDK8U-jre_x64_linux_hotspot_8u242b08/lib/amd64:/disk2/splunk/bin/jars/vendors/java/OpenJDK8U-jre_x64_linux_hotspot_8u242b08/lib/amd64/jli LDAPCONF=/disk2/splunk/etc/openldap/ldap.conf /disk2/splunk/bin/python3.7 -u /disk2/splunk/lib/python3.7/site-packages/splunk/clilib/cli.py reload deploy-server DEBUG (cli) Running in debug mode. DEBUG (cli) Command: reload DEBUG (cli) Subcmd: deploy-server DEBUG (cli) Begin parsed arguments: DEBUG (cli) DEBUG (cli) End parsed arguments. DEBUG (cli) noAuthReq: False DEBUG (cli) username: None DEBUG (cli) password: None DEBUG (cli_common) Running btool for 'web.conf'. WARNING (cli_common) btool returned something in stderr: 'Will exec (detach=no): HOSTNAME=splunk.cdc.health.local USER=splunk PATH=/disk2/splunk/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/splunk/.local/bin:/home/splunk/bin:/disk2/splunk/bin PWD=/opt/splunk/etc/deployment-apps SPLUNK_HOME=/disk2/splunk SPLUNK_DB=/export/intindx SPLUNK_SERVER_NAME=Splunkd SPLUNK_OS_USER=splunk NODE_PATH=/disk2/splunk/lib/node_modules LD_LIBRARY_PATH=/disk2/splunk/lib:/disk2/splunk/bin/jars/vendors/java/OpenJDK8U-jre_x64_linux_hotspot_8u242b08/lib/amd64:/disk2/splunk/bin/jars/vendors/java/OpenJDK8U-jre_x64_linux_hotspot_8u242b08/lib/amd64/jli LDAPCONF=/disk2/splunk/etc/openldap/ldap.conf /disk2/splunk/bin/splunkd btool web list ' DEBUG (cli) Contents of ./splunk/authToken file: <auth><username>splunkadmin</username><sessionkey></sessionkey></auth> DEBUG (cli) authInfo: DEBUG (rcUtils) cmd: reload, obj: deploy-server, restArgList: {'authstr':', 'owner': 'splunkadmin'} DEBUG (rcUtils) endpoint: deployments DEBUG (rcUtils) layeredFind:default_eai_parms: DEBUG (rcUtils) authstr poped from argList DEBUG (rcUtils) layeredFind:app_context: DEBUG (rcUtils) namespace: DEBUG (rcUtils) layeredFind:uri: /deployment/server/config/_reload DEBUG (rcUtils) layeredFind:eai_id: DEBUG (rcUtils) layeredFind:required: DEBUG (rcUtils) eai_key_list: [] DEBUG (rcUtils) eai_key_list: [] DEBUG (rcUtils) layeredFind:args: DEBUG (rcUtils) layeredFind:prehooks: DEBUG (rcUtils) after prehooks, eaiArgsList: {} DEBUG (rcUtils) Before buildEndpoint uri: /deployment/server/config/_reload DEBUG (rcUtils) Before buildEndpoint entityName: DEBUG (rcUtils) uri: /services/deployment/server/config/_reload DEBUG (rcUtils) layeredFind:type: edit DEBUG (rcUtils) eaiArgsList: {} DEBUG (rcUtils) In sanitizeArgs: target: , argsMap: {}, argsDict: {} DEBUG (rcUtils) In sanitizeArgs: target: , argsMap: {}, argsDict: {} DEBUG (rcUtils) In sanitizeArgs: target: /deployment/server/config/_reload, argsMap: {}, argsDict: {} DEBUG (rcUtils) In sanitizeArgs: target: /deployment/server/config/_reload, argsMap: {}, argsDict: {} DEBUG (rcUtils) postargs: {} DEBUG (rcUtils) getargs: {} DEBUG (__init__) simpleRequest > POST https://127.0.0.1:8089/services/deployment/server/config/_reload [] sessionSource=direct timeout=30 DEBUG (cli_common) Running btool for 'server.conf'. WARNING (cli_common) btool returned something in stderr: 'Will exec (detach=no): HOSTNAME=splunk.cdc.health.local USER=splunk PATH=/disk2/splunk/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/splunk/.local/bin:/home/splunk/bin:/disk2/splunk/bin PWD=/opt/splunk/etc/deployment-apps SPLUNK_HOME=/disk2/splunk SPLUNK_DB=/export/intindx SPLUNK_SERVER_NAME=Splunkd SPLUNK_OS_USER=splunk NODE_PATH=/disk2/splunk/lib/node_modules LD_LIBRARY_PATH=/disk2/splunk/lib:/disk2/splunk/bin/jars/vendors/java/OpenJDK8U-jre_x64_linux_hotspot_8u242b08/lib/amd64:/disk2/splunk/bin/jars/vendors/java/OpenJDK8U-jre_x64_linux_hotspot_8u242b08/lib/amd64/jli LDAPCONF=/disk2/splunk/etc/openldap/ldap.conf /disk2/splunk/bin/splunkd btool server list ' DEBUG (__init__) simpleRequest < server responded status=200 responseTime=0.1081s DEBUG (rcUtils) In checkStatus: type: edit, server_response: {'date': 'Thu, 01 Apr 2021 20:44:37 GMT', 'expires': 'Thu, 26 Oct 1978 00:00:00 GMT', 'cache-control': 'no-store, no-cache, must-revalidate, max-age=0', 'content-type': 'text/xml; charset=UTF-8', 'x-content-type-options': 'nosniff', 'content-length': '3828', 'vary': 'Authorization', 'connection': 'Close', 'x-frame-options': 'SAMEORIGIN', 'server': 'Splunkd', 'status': '200'} DEBUG (rcDisplay) In displayDeployment: kwargs: {'cmd': 'reload', 'obj': 'deploy-server', 'type': 'edit', 'serverResponse': {'date': 'Thu, 01 Apr 2021 20:44:37 GMT', 'expires': 'Thu, 26 Oct 1978 00:00:00 GMT', 'cache-control': 'no-store, no-cache, must-revalidate, max-age=0', 'content-type': 'text/xml; charset=UTF-8', 'x-content-type-options': 'nosniff', 'content-length': '3828', 'vary': 'Authorization', 'connection': 'Close', 'x-frame-options': 'SAMEORIGIN', 'server': 'Splunkd', 'status': '200'}, 'serverContent': b'<?xml version="1.0" encoding="UTF-8"?>\n\n<?xml-stylesheet type="text/xml" href="/static/atom.xsl"?>\n<feed xmlns="http://www.w3.org/2005/Atom" xmlns:s="http://dev.splunk.com/ns/rest" xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/">\n <title>deploymentserver</title>\n <id>https://127.0.0.1:8089/services/deployment/server/config</id>\n <updated>2021-04-01T20:44:37+00:00</updated>\n <generator build="152fb4b2bb96" version="8.0.6"/>\n <author>\n <name>Splunk</name>\n </author>\n <link href="/services/deployment/server/config/_reload" rel="_reload"/>\n <link href="/services/deployment/server/config/attributesUnsupportedInUI" rel="attributesUnsupportedInUI"/>\n <link href="/services/deployment/server/config/authentication" rel="authentication"/>\n <link href="/services/deployment/server/config/listIsDisabled" rel="listIsDisabled"/>\n <opensearch:totalResults>1</opensearch:totalResults>\n <opensearch:itemsPerPage>30</opensearch:itemsPerPage>\n <opensearch:startIndex>0</opensearch:startIndex>\n <s:messages/>\n <entry>\n <title>config</title>\n <id>https://127.0.0.1:8089/services/deployment/server/config/config</id>\n <updated>1970-01-01T00:00:00+00:00</updated>\n <link href="/services/deployment/server/config/config" rel="alternate"/>\n <author>\n <name>system</name>\n </author>\n <link href="/services/deployment/server/config/config" rel="list"/>\n <link href="/services/deployment/server/config/config/_reload" rel="_reload"/>\n <link href="/services/deployment/server/config/config" rel="edit"/>\n <content type="text/xml">\n <s:dict>\n <s:key name="currentDownloads">0</s:key>\n <s:key name="disabled">0</s:key>\n <s:key name="eai:acl">\n <s:dict>\n <s:key name="app"></s:key>\n <s:key name="can_list">1</s:key>\n <s:key name="can_write">1</s:key>\n <s:key name="modifiable">0</s:key>\n <s:key name="owner">system</s:key>\n <s:key name="perms">\n <s:dict>\n <s:key name="read">\n <s:list>\n <s:item>admin</s:item>\n <s:item>splunk-system-role</s:item>\n </s:list>\n </s:key>\n <s:key name="write">\n <s:list>\n <s:item>admin</s:item>\n <s:item>splunk-system-role</s:item>\n </s:list>\n </s:key>\n </s:dict>\n </s:key>\n <s:key name="removable">0</s:key>\n <s:key name="sharing">system</s:key>\n </s:dict>\n </s:key>\n <s:key name="loadTime">1617309877</s:key>\n <s:key name="repositoryLocation">$SPLUNK_HOME/etc/deployment-apps</s:key>\n </s:dict>\n </content>\n </entry>\n</feed>\n', 'sessionKey': '', 'eaiArgsList': {}}

mxanareckless · ‎03-15-2021

In the first picture, you can see the desired layout of the report in CSV. The second picture displays the corresponding results in Splunk after running the report. However, in the final picture you can see the exported CSV is hard to read, and the values of the far-right columns are one long string without newlines. Here's the SPL. index=index host=host sourcetype="db_backup_size" OR sourcetype=df | eval usage=Size+" "+Name, OldestBackup=OldestBackup+" "+Name | eval Environment="Production", "Server info (LDOM)"="LDOM", "Mount Point"="Mount Point", "Backup Filesystem"="/backup", "Local mount"="/b" | chart latest(Environment) as Environment, latest("Server info (LDOM)") as "Server info (LDOM)", latest("Mount Point") as "Mount Point", latest("Backup Filesystem") as "Backup Filesystem", latest("Local mount") as "Local mount", latest(Size) as "Allocation", latest(Used) as "Usage", values(usage) as "Usage for Each Database", values(OldestBackup) as "Oldest Backup Set" How can I export something that resembles the first picture? Thank you!

mxanareckless · ‎03-10-2021

As seen in the first example, the expression I've constructed will capture the field values I want. However in the second example, all of the values are not being captured for the field I wish to extract. Why is it capturing everything in the first example, but not the second? Capturing between 1-3 digits followed by 1 of any letter.

mxanareckless · ‎03-08-2021

@richgalloway Thanks again for your reply. I've struggled to add the | stats values(*) as * by <common_field_name>, because neither of the searches seem to have any common field names. I've tried using both "Usage" and "Usage for Each Database" as the <common_field_name>, and although the results are different, it's not what we want. How can I add "Usage for Each Database" field to the first search in order for both searches to have a common field?

mxanareckless · ‎02-26-2021

@richgalloway Thanks for your reply. I've tried your second suggested method. Here's the exact results of the subseach, which I would like appended as a column to the result table of the main search: I've tried putting this into an | append in the main search, but I see only a blank row added to the table.

mxanareckless · ‎02-26-2021

Here is what I need to generate: So far I have this search to generate everything but the right-most column: index=Production sourcetype=df host="host*" MountedOn="/b1" | stats latest(Size) as "Allocation", latest(Used) as "Usage" | eval Environment="Production", "Server info (LDOM)"="host1,host2", "Mount Point"="/export/backups", "Backup Filesystem"="/backup01", "Local mount"="/b1" | table Environment, "Server info (LDOM)", "Mount Point", "Backup Filesystem", "Local mount", "Allocation", "Usage" Results: And this search generates the contents for the right-most column in the spreadsheet: index=DB sourcetype=db_size | table source, TotalSizeGB | eval TotalSizeGB=round(TotalSizeGB,2) | sort -TotalSizeGB Results: How can I get the results of search #2 to appear in the final right-most column of search #1's results? Thank you!

mxanareckless · ‎02-23-2021

@bowesmana Thank you very much for your insightful reply. I think I understand the concept, but when I apply your suggestion, all of the fields returned are blank instead. I need the values in the bottom screenshot to be rounded to the nearest whole integer - the excessive decimals make it very difficult to read in reports.

mxanareckless · ‎02-22-2021

@tscroggins Thank you very much for your response. How can I integrate this solution with my current query? I am not well versed in Oracle DB and am only providing Splunk queries for that team. | dbxquery connection="helps" query="select (select sum(bytes) from dba_data_files)+(select sum(bytes) from dba_temp_files)-(select sum(bytes) from dba_free_space) total_size from dual;"

mxanareckless · ‎02-22-2021

@to4kawa

mxanareckless · ‎02-19-2021

I'm interested in creating an alert scheduled to run every 60 minutes, that will search for hosts which have had > 85% CPU load over a span of 5 minutes. Here's the search: index=index sourcetype=cpu | streamstats time_window=5min latest(cpu_load_percent) count by host | eval cpu_load_percent=if(count<18,null,round(cpu_load_percent, 2)) | where cpu_load_percent>85 | dedup host | table host, _time, cpu_load_percent From there, I would like a report generated, wherein for each host a timechart is provided for the last 60 minutes, showing CPU %s for each of the processes run on that host. Ideally this will be a line chart, with a line for each of the top 10 CPU-heavy processes. I've tried using | transaction, and this is what I have so far: index=index sourcetype=cpu AND sourcetype=top host=$host$ | timechart latest(cpu_load_percent) by COMMAND I'd really appreciate any guidance on how to implement an alert of this type.

mxanareckless · ‎02-12-2021

@bowesmana That worked for | timechart avg(cpu_load_percent) as cpuavg However, if I add by host to the end, the rounding no longer works. index=index sourcetype=vmstat | where like(host, "kdc%") | timechart avg(memUsedGB) as avgmem by host | eval avgmem=round(avgmem,2)

mxanareckless · ‎02-10-2021

I need a Single Value widget for a dashboard which displays current RAM usage in real-time. This is what I have so far in SPL: index=index host=host sourcetype=vmstat memUsedMB=* | stats count And this is all I'm getting: How can I get something more like this? :

mxanareckless · ‎02-10-2021

I cannot figure out how to round the values presented on the timechart. My SPL: index=$radio_token$ host=$dropdown_token2$ sourcetype=cpu | eval cpuavg=round(cpu_load_percent, 2) | timechart avg(cpuavg) And these are the results: How can I get that to appear as 1%, instead of that huge clunky number?

mxanareckless · ‎02-05-2021

Need to run a dbxquery command via the REST API, and having trouble defining the search's time range in that context. Below I demonstrate how the queries appear in the web UI, versus the commandline with curl. Either the query is invalid because | dbxquery needs to be at the beginning of the query, or no results are returned when appending | search earliest=-1day latest=now to the end of the query. How can I correctly specify a time range when using | dbxquery via REST API? [root@host ~]# curl -u user:password -k https://192.168.xx.xxx:xxxx/services/search/jobs/export --data-urlencode search='seary connection="xxx" query="SELECT (SELECT sum(bytes) FROM dba_data_files)+ (SELECT sum(bytes) FROM dba_temp_files)- (SELECT sum(bytes) FROM dba_ <?xml version='1.0' encoding='UTF-8'?> <response><messages><msg type="FATAL">Error in 'dbxquery' command: This command must be the first command of a search.</msg></messages></response> [root@host ~]# curl -u user:password -k https://192.168.xx.xxx:xxxx/services/search/jobs/export --data-urlencode search='| dT (SELECT sum(bytes) FROM dba_data_files)+ (SELECT sum(bytes) FROM dba_temp_files)- (SELECT sum(bytes) FROM dba_free_space) total_size FROM dual; <?xml version='1.0' encoding='UTF-8'?> <results preview='0'> <meta> <fieldOrder> <field>TOTAL_SIZE</field> </fieldOrder> </meta> <messages> <msg type="DEBUG">Configuration initialization for /opt/splunk/etc took 15ms when dispatching a search (search ID: 1612554968.128577)</msg> <msg type="DEBUG">The 'dbxquery' command is implemented as an external script and may cause the search to be significantly slower.</msg> <msg type="DEBUG">search context: user="reporting", app="search", bs-pathname="/opt/splunk/etc"</msg> </messages> </results>

mxanareckless · ‎01-20-2021

Really stumped on this. We would like to count the number of instances of each process run on a server, and present the sum of RAM and CPU usage for all those instances of the process. Here is an example, from the server side, of all the instances of a single app being run, that we would like to aggregate: PID USERNAME NLWP PRI NICE SIZE RES STATE TIME CPU COMMAND 22661 cacheuse 1 59 0 452M 441M sleep 0:03 0.00% cache 22664 cacheuse 1 59 0 452M 440M sleep 0:00 0.00% cache 22669 cacheuse 1 59 0 452M 440M sleep 0:00 0.00% cache 22667 cacheuse 1 59 0 452M 440M sleep 0:00 0.00% cache 22665 cacheuse 1 59 0 452M 440M sleep 0:00 0.00% cache 22670 cacheuse 1 59 0 452M 440M sleep 0:00 0.00% cache 22668 cacheuse 1 59 0 452M 440M sleep 0:00 0.00% cache 22666 cacheuse 1 59 0 452M 440M sleep 0:00 0.00% cache 22953 cacheuse 1 59 0 452M 444M sleep 0:03 0.00% cache 23053 cacheuse 1 59 0 452M 444M sleep 0:06 0.00% cache 23052 cacheuse 1 59 0 452M 444M sleep 0:02 0.00% cache 24543 cacheuse 1 59 0 452M 444M sleep 0:03 0.00% cache 22941 cacheuse 1 59 0 452M 440M sleep 0:00 0.00% cache 22945 cacheuse 1 59 0 452M 440M sleep 0:00 0.00% cache 22944 cacheuse 1 59 0 452M 440M sleep 0:00 0.00% cache 22943 cacheuse 1 59 0 452M 440M sleep 0:00 0.00% cache 22946 cacheuse 1 59 0 452M 440M sleep 0:00 0.00% cache 22942 cacheuse 1 59 0 452M 440M sleep 0:00 0.00% cache 22947 cacheuse 1 59 0 452M 440M sleep 0:00 0.00% cache 22940 cacheuse 1 59 0 452M 440M sleep 0:00 0.00% cache 22948 cacheuse 1 59 0 452M 440M sleep 0:00 0.00% cache 22939 cacheuse 1 59 0 452M 440M sleep 0:06 0.00% cache 22938 cacheuse 1 59 0 452M 440M sleep 0:00 0.00% cache 22671 cacheuse 1 59 0 451M 440M sleep 0:00 0.00% cache 22663 cacheuse 1 59 0 451M 440M sleep 0:13 0.00% cache 22662 cacheuse 1 59 0 451M 440M sleep 0:00 0.00% cache 22649 cacheuse 1 59 0 444M 440M sleep 0:33 0.00% cache 22932 cacheuse 1 59 0 443M 440M sleep 0:05 0.00% cache 5863 root 17 59 0 185M 163M sleep 139:37 0.00% sstored 4570 splunk 43 59 0 177M 130M sleep 15:16 0.00% splunkd As you can see, there are 28 instances of the cache program. We would like to roll all of that up into something like this: Program # instances total RAM total CPU cache 28 12GB 0.00% splunkd 1 177M 0.01% sstored 1 185M 0.0.1% For the top sourcetype, the VIRT sourcetype counts RAM in kilobytes. If VIRT's integer value is greater than 1024, we want the integer multiplied by 1024 and suffixed with the letter "M" for megabytes; and if the integer is greater than 1048576, we want that integer multiplied by 1048576 and suffixed with the letter "G" for gigabytes. Here is what we've come up with so far, but it's nowhere near what we need: index=xxxx sourcetype=top host=xxxx COMMAND!="<n/a>" | rename COMMAND as Program, pctCPU as "% CPU", USER as User | regex "% CPU"="(\d+)" | convert rmunit(VIRT) | eval inMB=if(VIRT>=1024,1,0), VIRT=floor(if(inMB=1,VIRT/1024,VIRT*1)) | chart sum(VIRT) by Program Thank you in advance!

mxanareckless · ‎01-19-2021

@scelikok Thank you; but still nothing is being returned after trying for 10 minutes, until eventually an error "Unknown SID" is returned: [user.name@host ~]$ curl -u username:password -k https://192.168.xx.xxx:xxxx/services/search/jobs/export --data-urlencode search=' | dbxquery query=\"select (select sum(bytes) from dba_data_files)+(select sum(bytes) from dba_temp_files)-(select sum(bytes) from dba_free_space) total_size from dual\" connection=\"xxx\"' <?xml version='1.0' encoding='UTF-8'?> <results preview='0'> <meta> <fieldOrder /> </meta> <messages> <msg type="DEBUG">Configuration initialization for /opt/splunk/etc took 19ms when dispatching a search (search ID: 1611076520.164004)</msg> <msg type="DEBUG">The 'dbxquery' command is implemented as an external script and may cause the search to be significantly slower.</msg> <msg type="DEBUG">search context: user="username", app="search", bs-pathname="/opt/splunk/etc"</msg> </messages> </results> # Querying for result for 10 minutes, until below occurs: [user.name@host ~]$ curl -u username:password -k https://192.168.xx.xxx:xxxx/services/search/jobs/1611076520.164004/results/ --get -d output_mode=csv <?xml version="1.0" encoding="UTF-8"?> <response> <messages> <msg type="FATAL">Unknown sid.</msg> </messages> </response>

mxanareckless · ‎01-18-2021

I've checked this, but it hasn't solved the problem for me: https://community.splunk.com/t5/Getting-Data-In/Is-it-possible-to-run-a-curl-command-on-a-dbxquery/m-p/500081#M85221 This is my curl request: curl -u username:password -k https://192.168.xx.xxx:xxxx/services/search/jobs -d search=" | dbxquery query=\"select (select sum(bytes) from dba_data_files)+(select sum(bytes) from dba_temp_files)-(select sum(bytes) from dba_free_space) total_size from dual\" connection=\"XXX\"" And I get an SID back: <?xml version="1.0" encoding="UTF-8"?> <response> <sid>1611013146.153172</sid> </response> However when I try fetching the results, I get nothing back: [user.name@host ~]$ curl -u username:password -k https://192.168.xx.xxx:xxxx/services/search/jobs/1611013146.153172/results/ --get -d output_mode=csv [user.name@host ~]$ I've tried waiting a few minutes in between fetch attempts, still nothing. This same query works find and returns a result immediately when run from the DBX UI: Is there something I'm missing here in order to get the result via the REST API? Thanks.

mxanareckless · ‎01-07-2021

As you can see, top reports most values in megabytes, but a few are in kilobytes. I am working on a timechart that will show memory consumption in gigbytes over a day's time. I need to convert all of the VIRT values containing the letter "K" to megabytes, leave the VIRT values containing the letter "M" untouched, before finally converting all of the VIRT values into gigabytes. This is my query so far: index=os_evt_prd sourcetype=top host=kdcprd-db00 | rename COMMAND as Program | regex Program="(\w+[[:lower:]])" | replace *oracle WITH oracle | convert rmunit(VIRT) | eval "Used GB Mem."=floor(VIRT*0.0009765625) | timechart avg("Used GB Mem.") BY Program However, the values don't appear to be calculated properly due to the existence of VIRT values in both MB & KB. How can I get around this?

mxanareckless · ‎07-10-2020

@gcusello I did suspect this was an NFS issue, as the bottleneck first appeared after I migrated the indexes over. Turns out Splunk was using the slower backup pool, and not the iops-optimized pool. Thanks for pointing me in the right direction!

mxanareckless · ‎07-06-2020

There doesn't seem to be a lot of documentation or discussions online which cover the setup of an intermediate, heavy forwarder. We need this for the following reasons: * to scrub/anonymize personal information from data coming from universal forwarders * to reduce load on indexing server, whose parsing queues are consistently full Here is the deployment: [uf] > [hf] > [indexer] Does anybody have example .conf files that would support this? So far, mine look as such: Universal forwarder's outputs.conf: [tcpout] defaultGroup = pspr-heavy-forwarder [tcpout:pspr-heavy-forwarder] disabled = false server = 192.168.60.213:9997 Heavy forwarder's outputs.conf: [tcpout] defaultGroup = central-indexer indexAndForward = false sendCookedData = true useACK = true [tcpout:central-indexer] disabled = false server = 192.168.60.211:9997 Indexer's inputs.conf: [default] queue = indexQueue I've directed all universal forwarders to send to the intermediate forwarder, but the main indexer's still showing saturated queues. Local monitoring is limited to Splunk's own logs. Is there a way I can view what exactly is going into these queues, so I know where to chase the problem?

Posts	22
Solutions	0
Karma Given	24
Karma Received	0
Member Since	‎07-06-2020

Online Status	Offline
Date Last Visited	‎01-24-2023 05:50 PM

Email report shows "No results found."

Timechart RAM or CPU usage by Linux process

Deployment apps not reloading

Poor formatting for report export to CSV

Working regex elsewhere does not work in Splunk

Insert results of subsearch as single column in ta...

Timechart CPU by Process

Single Value: Current Server RAM Usage

How to round result of timechart avg()?

REST API: DBX: Specify time range

Email report shows "No results found."

Timechart RAM or CPU usage by Linux process

Deployment apps not reloading

Poor formatting for report export to CSV

Working regex elsewhere does not work in Splunk

Re: Insert results of subsearch as single column i...

Re: Insert results of subsearch as single column i...

Insert results of subsearch as single column in ta...

Re: Round result of timechart avg()

Re: REST API: DBX: Specify time range

Re: Timechart CPU by Process

Timechart CPU by Process

Re: Round result of timechart avg()

Single Value: Current Server RAM Usage

How to round result of timechart avg()?

REST API: DBX: Specify time range

Chart # of instances for a process, w/ sum of RAM ...

Re: Run DBX query via REST API

Run DBX query via REST API

top reports values in both MB & KB: How to convert...

Re: Intermediate Heavy Forwarder setup

Intermediate Heavy Forwarder setup: How to view wh...