I am doing a short term gig building dashboards in Splunk and I have a production standalone Splunk Enterprise single instance deployment which I don't have admin access to. But I do have admin access to the Dev instance. Dev instance however has no data in it. My gut tells me I can make the production instance a search peer to my Dev box and start using production data to build dashboards in Dev. But I see this in Splunk documentation
Important: A search head should not perform a dual function as a search peer. The only exception to this rule is for the distributed management console, which functions as a "search head of search heads." I could not find anymore details whether this is a technical infeasibility or a performance best practice.
Has anybody tried this before?
Just a thought I had. If I get the relevant buckets with suitable time periods copied over from Dev to Prod, I should be able to achieve my goal. It is a standalone Splunk instance so I don't think the instance GUID is part of. I know the sysadmin is going to give me the looks. I also know this is not exactly the answer to my question. But just presenting it as a solve to achieve the end goal.
When in an environment where I need to do "dev on a budget" I've configured a dev search head to peer the prod indexers. This has some limitations, but is generally a pretty reliable way to build and test apps as you get a full dataset to utilize and you get to ensure that your new saved searches don't over-schedule a block of time.
I am a bit skeptical after reading the Splunk docs as the Production set up that I am dealing with is a standalone single instance deployment and not an indexer only instance. I don't know if making it a search peer to my Dev instance will impact its active prod SH duties.
To comprehensively cover all use cases, I need at least 8 days worth of data. But the Dev is pooling license with prod. So I can't import that much logs into Dev without license violations.