All Posts

Find Answers
Ask questions. Get answers. Find technical product solutions from passionate members of the Splunk community.

All Posts

Hey guys I have a question about CSS in splunk, I want to move these displays like this: And I can't use dashboard studio because I need some special things that dahsboard studio doesn't have ye... See more...
Hey guys I have a question about CSS in splunk, I want to move these displays like this: And I can't use dashboard studio because I need some special things that dahsboard studio doesn't have yet, like depends functionality, I'm trying with css grid but I couldn't find a way to do this yet
Hi gcusello, Thank you for your time to review this, and your quick reply! We do in fact have heavy forwarders at each tenant, so there is local control over any traffic passing through those. That ... See more...
Hi gcusello, Thank you for your time to review this, and your quick reply! We do in fact have heavy forwarders at each tenant, so there is local control over any traffic passing through those. That said though, I am told we have over 40,000 endpoints reporting directly to the cloud , so we definitely do not know the hostnames for everything. Each tenant does have their own HEC key for those submissions, so that would be about the only way to identify and separate that traffic. In my original posting, my intended means to make these changes was via a custom inputs.conf file on all the HFs and UFs rather than modifying the events on arrival, since the inputs file is already at the source. Aside from the overhead and logistics of having to deploy that file to that many workstations and servers, I was wondering if there's any downside to this approach from the perspective of the Splunk agent itself. The ability to do override some/all of any portion of any .conf files is built into the design, so it seems like a custom inputs .conf wouldn't be a problem in any way. Any feedback on that approach? Thank you!
I just posted a question that was immediately rejected. How do I get it approved please?
I use the OpenTelemetry Java agent to monitor FusionAuth in one Docker container, and send the output to the Splunk OpenTelemetry Docker container in Gateway mode.   Here's a diagram of my system a... See more...
I use the OpenTelemetry Java agent to monitor FusionAuth in one Docker container, and send the output to the Splunk OpenTelemetry Docker container in Gateway mode.   Here's a diagram of my system architecture:   ```mermaid graph LR subgraph I[Your server] direction LR subgraph G[Docker] H[(Postgresql)] end subgraph C[Docker] direction BT D(OpenTelemetry for Java) --> A(FusionAuth) end subgraph E[Docker] B(Splunk OpenTelemetry collector) end end C --> G C --> B E --> F(Splunk web server) style I fill:#111 ```   The Splunk container runs correctly and exports sample data to Splunk Observability Cloud. I can see it in the dashboard.   FusionAuth and the Java agent run correctly. But the Otel sender cannot send to the Otel collector. I get network errors: ```sh | [otel.javaagent 2024-06-07 13:52:40:936 +0000] [OkHttp http://otel:4317/...] ERROR io.opentelemetry.exporter.internal.http.HttpExporter - Failed to export logs. The request could not be executed. Full error message: Connection reset fa | java.net.SocketException: Connection reset fa | at java.base/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:328) fa | at java.base/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:355) ...   [otel.javaagent 2024-06-07 13:52:42:847 +0000] [OkHttp http://otel:4317/...] ERROR io.opentelemetry.exporter.internal.http.HttpExporter - Failed to export spans. The request could not be executed. Full error message: Connection reset by peer fa | java.net.SocketException: Connection reset by peer fa | at java.base/sun.nio.ch.NioSocketImpl.implWrite(NioSocketImpl.java:425) fa | at java.base/sun.nio.ch.NioSocketImpl.write(NioSocketImpl.java:445) fa | at java.base/sun.nio.ch.NioSocketImpl$2.write(NioSocketImpl.java:831) fa | at java.base/java.net.Socket$SocketOutputStream.write(Socket.java:1035) ```   I'm using the standard configuration file for Splunk Linux Collector - https://github.com/signalfx/splunk-otel-collector/blob/main/cmd/otelcol/config/collector/otlp_config_linux.yaml Below is my docker compose file   ```yaml services: db: image: postgres:latest container_name: fa_db ports: - "5432:5432" environment: PGDATA: /var/lib/postgresql/data/pgdata POSTGRES_USER: ${POSTGRES_USER} POSTGRES_PASSWORD: ${POSTGRES_PASSWORD} healthcheck: test: [ "CMD-SHELL", "pg_isready -U postgres" ] interval: 5s timeout: 5s retries: 5 networks: - db_net volumes: - db_data:/var/lib/postgresql/data fa: # image: fusionauth/fusionauth-app:latest image: faimage container_name: fa # command: "tail -f /dev/null" depends_on: db: condition: service_healthy environment: DATABASE_URL: jdbc:postgresql://db:5432/fusionauth DATABASE_ROOT_USERNAME: ${POSTGRES_USER} DATABASE_ROOT_PASSWORD: ${POSTGRES_PASSWORD} DATABASE_USERNAME: ${DATABASE_USERNAME} DATABASE_PASSWORD: ${DATABASE_PASSWORD} FUSIONAUTH_APP_MEMORY: ${FUSIONAUTH_APP_MEMORY} FUSIONAUTH_APP_RUNTIME_MODE: ${FUSIONAUTH_APP_RUNTIME_MODE} FUSIONAUTH_APP_URL: http://fusionauth:9011 SEARCH_TYPE: database FUSIONAUTH_APP_KICKSTART_FILE: ${FUSIONAUTH_APP_KICKSTART_FILE} networks: - db_net ports: - 9011:9011 volumes: - fusionauth_config:/usr/local/fusionauth/config - ./kickstart:/usr/local/fusionauth/kickstart extra_hosts: - "host.docker.internal:host-gateway" otel: image: quay.io/signalfx/splunk-otel-collector:latest container_name: fa_otel environment: SPLUNK_ACCESS_TOKEN: "secret" SPLUNK_REALM: "us1" SPLUNK_LISTEN_INTERFACE: "0.0.0.0" SPLUNK_MEMORY_LIMIT_MIB: "1000" SPLUNK_CONFIG: /config.yaml volumes: - ./config.yaml:/config.yaml networks: - db_net # no host ports are needed as communication is inside the docker network # ports: # - "13133:13133" # - "14250:14250" # - "14268:14268" # - "4317:4317" # - "6060:6060" # - "7276:7276" # - "8888:8888" # - "9080:9080" # - "9411:9411" # - "9943:9943" networks: db_net: driver: bridge volumes: db_data: fusionauth_config: ```   The FusionAuth Dockerfile starts FusionAuth like this:   ```sh exec "${JAVA_HOME}/bin/java" -javaagent:/usr/local/fusionauth/otel.jar -Dotel.resource.attributes=service.name=fusionauth -Dotel.traces.exporter=otlp -Dotel.exporter.otlp.endpoint=http://otel:4317 -cp "${CLASSPATH}" ${JAVA_OPTS} io.fusionauth.app.FusionAuthMain <&- >> "${LOG_DIR}/fusionauth-app.log" 2>&1 ``` Why can't the FusionAuth container connect to http://otel:4317 please?
Hello, I've recently tested a sourcetype for a new input via the props.conf file on my standalone dev environment, and it worked perfectly -datas were correctly parsed -. But when I put it in my pro... See more...
Hello, I've recently tested a sourcetype for a new input via the props.conf file on my standalone dev environment, and it worked perfectly -datas were correctly parsed -. But when I put it in my prod environment, the data which where attributed the sourcetype weren't parsed at all. Now, my prod environment is distributed (HFs->DS->Indexers->SH) but I've been careful to put the sourcetype both in the Heavy forwarder and in the searchhead as recommended, and i've restart both the HF and the SH but it still doesn't work. Does anyone have an idea of what I can do to fix it?
Hello, we have a Red status for Ingestion Latency,  it says the following:    Red: The feature has severe issues and is negatively impacting the functionality of your deployment. For details, see R... See more...
Hello, we have a Red status for Ingestion Latency,  it says the following:    Red: The feature has severe issues and is negatively impacting the functionality of your deployment. For details, see Root Cause.   However, I can't figure out how to see the "Root Cause".  What report should I look at, that would show me where this latency is occurring? Thanks for all of the help, Tom  
I use Eggplant https://www.keysight.com/us/en/product/EG1000A/eggplant-test.html    ( https://support.eggplantsoftware.com/downloads/eggplant-functional-downloads ) Eggplant understands GUI controls... See more...
I use Eggplant https://www.keysight.com/us/en/product/EG1000A/eggplant-test.html    ( https://support.eggplantsoftware.com/downloads/eggplant-functional-downloads ) Eggplant understands GUI controls natively - doesn't just check the HTML source - it actually checks what the user sees. Had great success testing swing clients, mobile apps across multiple platforms. You would probaly have to write some scripting to check the math.  
Hello @deepakc, thank you for your effort to respond. Maybe I was unclear with my question, let me rephrase it. TL;DR: how can changes on forwarder affect search-head's behavior during search-time? ... See more...
Hello @deepakc, thank you for your effort to respond. Maybe I was unclear with my question, let me rephrase it. TL;DR: how can changes on forwarder affect search-head's behavior during search-time? My primary goal is to understand how the extractions work particularly in this detail. Longer version: The issue here is with search-time extractions. To my knowledge, these extractions are done by search-head and/or respective indexer when the data is being searched (= after all writing to the index is finished, no matter how the data was cooked). Forwarders are not involved in searching, therefore I don't understand how forwarder can affect the search. The events themselves seem to be indexed the same way before and after upgrade, I don't see any distinction in _raw field contents and generally no reason for search-time extractions to stop working. What in my assumptions and knowledge above is not correct? Your suggestions apply - as of my knowledge - to index-time extractions, however that is not the case here.
Hello @Amadou , I believe SSE should be used in conjunction with Splunk Enterprise Security. The correlation searches within ES, ESCU, and SSE have MITRE tactics associated with each of them. The nu... See more...
Hello @Amadou , I believe SSE should be used in conjunction with Splunk Enterprise Security. The correlation searches within ES, ESCU, and SSE have MITRE tactics associated with each of them. The number of correlation searches with MITRE tactics are highlighted with dark color on the SSE dashboard and the less number would be light in shade.  I do not think this is possible to achieve on Splunk Enterprise. However, if you are able to achieve this through custom solution using .css or .js, let the community know.   Thanks, Tejas.
Hello @deangoris, The Splunk packaging toolkit will also work in the similar fashion. The packages created using the toolkit should not have the local folder within it. Otherwise it'll fail on the U... See more...
Hello @deangoris, The Splunk packaging toolkit will also work in the similar fashion. The packages created using the toolkit should not have the local folder within it. Otherwise it'll fail on the UI itself. The best way to deal with this situation is to have a Barebone app created from the UI and have all the KOs migrated to the custom private app. This way it helps modifying the objects from UI in future as well.   Thanks, Tejas.   --- If the above solution helps, an upvote is appreciated..!! 
Hi Team, Need your assistance for the configuration changes in Splunk. The requirement is to change the Timezone based on different “source” (not sourcetype). We have different sources defined in o... See more...
Hi Team, Need your assistance for the configuration changes in Splunk. The requirement is to change the Timezone based on different “source” (not sourcetype). We have different sources defined in our application. All of them are in their respective server timezone, except for the below 2 sources (these 2 are in EST timezone & our requirement is to change it into CET timezone)     source=/applications/testscan/*/testscn01/* source=/applications/testscan/*/testcpdom/*     For rest of the other sources, I do not want make any change in the Timezone. For example:     source=/applications/testscan/*/testscn02/* source=/applications/testscan/*/testnycus/* source=/applications/testscan/*/testnyus2/* source=/applications/testscan/*/testshape/* source=/applications/testscan/*/testshape2/* source=/applications/testscan/*/testshape3/*     Please note, we do not have any "props.conf" file available or configured in the server.  We are maintaining splunk configuration in only "inputs.conf" file. The present content of "inputs.conf" as below:     [monitor:///applications/testscan/.../] whitelist = (?:tools\/test\/log\/|TODAY\/LOGS\/)*\.(?:log|txt)$ index = testscan_prod sourcetype = testscan _TCP_ROUTING = in_prod [monitor:///applications/testscan/*/*/tools/test_transfer/log] index = testscan_prod sourcetype = testscan _TCP_ROUTING = in_prod [monitor:///applications/testscan/*/*/tools/test_reports/log] index = testscan_prod sourcetype = testscan _TCP_ROUTING = in_prod       Please suggest what changes to be done so that Timezone can be managed based on the "source" information provided. @ITWhisperer 
In this case a full file systems caused this file below to be empty, even after splunkd restart, it was still empty.. That was the cause of this error: [splunk@hf001 ~]$ ll /opt/splunk/quarantined_f... See more...
In this case a full file systems caused this file below to be empty, even after splunkd restart, it was still empty.. That was the cause of this error: [splunk@hf001 ~]$ ll /opt/splunk/quarantined_files/ total 8 -rwxr-x--- 1 splunk splunk 0 Jun 7 14:18 quarantine_manifest.json -rwxr-x--- 1 splunk splunk 208 Mar 16 2023 README.md Adding Enterprise default config to the file solved the issue: [splunk@hf001 ~]$ cat /opt/splunk/quarantined_files/quarantine_manifest.json {"enable_jQuery2": "not-restricted", "enable_unsupported_hotlinked_imports": "not-restricted"}
So basically I'd like to do concatenation between DeviceProcess and DeviceRegistry events in advanced hunting query | advhunt   
Hello @jrs42, In Dashboard studio, there's no option to specify a drilldown for a particular cell or a row. When you enable the drilldown, by default it gets applied to a cell. You can find the foll... See more...
Hello @jrs42, In Dashboard studio, there's no option to specify a drilldown for a particular cell or a row. When you enable the drilldown, by default it gets applied to a cell. You can find the following JSON source code as an example for a drilldown to set token in dashboard studio. { "visualizations": { "viz_dNS83Gj5": { "type": "splunk.table", "dataSources": { "primary": "ds_aQ7285AG" }, "eventHandlers": [ { "type": "drilldown.setToken", "options": { "tokens": [ { "token": "log_level_tok", "key": "row.log_level.value" } ] } } ] }, "viz_qGr86Sbm": { "type": "splunk.events", "options": {}, "dataSources": { "primary": "ds_MmJUCreO" } } }, "dataSources": { "ds_aQ7285AG": { "type": "ds.search", "options": { "query": "index=_internal source=\"*splunkd.log\"\n| stats count by log_level", "queryParameters": { "earliest": "$global_time.earliest$", "latest": "$global_time.latest$" } }, "name": "Search_1" }, "ds_MmJUCreO": { "type": "ds.search", "options": { "query": "index=_internal source=\"*splunkd.log\" log_level=\"$log_level_tok$\"", "queryParameters": { "earliest": "$global_time.earliest$", "latest": "$global_time.latest$" } }, "name": "Search_2" } }, "defaults": { "dataSources": { "ds.search": { "options": { "queryParameters": { "latest": "$global_time.latest$", "earliest": "$global_time.earliest$" } } } } }, "inputs": { "input_global_trp": { "type": "input.timerange", "options": { "token": "global_time", "defaultValue": "-24h@h,now" }, "title": "Global Time Range" } }, "layout": { "type": "absolute", "options": { "width": 1440, "height": 960, "display": "auto" }, "structure": [ { "item": "viz_dNS83Gj5", "type": "block", "position": { "x": 0, "y": 0, "w": 300, "h": 300 } }, { "item": "viz_qGr86Sbm", "type": "block", "position": { "x": 300, "y": 0, "w": 1140, "h": 300 } } ], "globalInputs": [ "input_global_trp" ] }, "description": "", "title": "Test Input Placeholder" }   Thanks, Tejas.   --- If the above solution helps, an upvote is appreciated..!!  
Hi @Cyner__, you have to enable receiving on Splunk Enterprise, then you have to check the route from the Universal Forwarder on port 9997 to the Spunk Enterprise (using telnet), then you have to ... See more...
Hi @Cyner__, you have to enable receiving on Splunk Enterprise, then you have to check the route from the Universal Forwarder on port 9997 to the Spunk Enterprise (using telnet), then you have to configure your outputs.con (as described in the above link) in the Universal Forwarder. Ciao. Giuseppe
Thanks for your answer. I'm not sure if this is what I want. Because the advanced hunting app requires an API call with a limit of calls, I start doing a call on DeviceProcessEvents. Then I'm not sur... See more...
Thanks for your answer. I'm not sure if this is what I want. Because the advanced hunting app requires an API call with a limit of calls, I start doing a call on DeviceProcessEvents. Then I'm not sure if I need to do another API call on DeviceRegistryEvents, since I'd like to joint these two instances. 
Hi @anandhalagaras1 , you should take the searches in Workload and adapt them to your requirements. Ciao. Giuseppe
I have the same question, which capabilities are needed for the "Add Data" button?
@gcusello    We are using Splunk Cloud version 9.1.2308.203. Following your instructions, I navigated to Cloud Monitoring Console --> License Usage and found the following options in the Cloud Moni... See more...
@gcusello    We are using Splunk Cloud version 9.1.2308.203. Following your instructions, I navigated to Cloud Monitoring Console --> License Usage and found the following options in the Cloud Monitoring Console App: - Entitlement - Ingest - Workload - Storage Summary - Searchable Storage (DDAS) - Archive Storage (DDAA) - Federated Search for Amazon S3 Our Cloud Monitoring Console app is version 3.25.0. Please let me know how to pull the top 20 or top 50 sources with the index and sourcetype information.
Hi rsreese,  i know this post is some years old already but maybe it can help someone in the future. The McAfee ePO or now called Trellix Orchestrator can only sent data to tcp ports via SSL.  So s... See more...
Hi rsreese,  i know this post is some years old already but maybe it can help someone in the future. The McAfee ePO or now called Trellix Orchestrator can only sent data to tcp ports via SSL.  So switch the input from [tcp://514] to [tcp-ssl:514]. Be sure to fulfill the configuration requirements for tcp-ssl inputs.