topic generate dummy data -eventgen in Getting Data In

generate dummy data -eventgen

net1993 — Tue, 02 Jul 2019 10:48:36 GMT

Hello
Can you provide some working solution for eventgen with testdata and eventgen.conf which is working for you?

I am 0 interested in how it is working or their docs as I've used hours to try make it working without success so I just need someone who has already working set of conf with some simple sample data.

Re: generate dummy data -eventgen

adonio — Tue, 02 Jul 2019 12:50:45 GMT

you can download the eventgen app and look in its own docs.
also, many TAs has sample data and eventgen.conf in them, turning eventgen on alongside with the TA will generate fake data.
explore the evengen.conf in the to further understand how it works
try and download this TA for Cisco ASA: https://splunkbase.splunk.com/app/1620/
it supposed to have eventgen.conf in it

Re: generate dummy data -eventgen

net1993 — Wed, 03 Jul 2019 06:31:28 GMT

Hi.
I dont need to see how eventgen its working, I need a working config file and data sample so I can just run and not lose time with that simple thing.

Re: generate dummy data -eventgen

koshyk — Wed, 03 Jul 2019 08:43:26 GMT

There are officially two versions to run eventgen
1. as a python package (needs pypy etc)
2. As a Splunk app. Be cautious that this is NOT accidently pushed to PROD

I'm not a big fan of both above, as I need it to be run as a standalone softare. Below are some hints..

Are you using the old eventgen or the new eventgen?
The old eventgen had a youtube video: https://www.youtube.com/watch?v=wLYMY9dwBXI

For the new eventgen, if you want to simulate similar fashion, you need to extract the SA-Eventgen/lib/splunk_eventgen . and do as above. When I get time, I will post this as step by step process into a blog.

Re: generate dummy data -eventgen

net1993 — Wed, 03 Jul 2019 08:47:26 GMT

I got last version. of eventgen
I work on TEST machine:)
It will be great if you create a guide for this. I tried yesterday but it simply doesnt insert data in splunk.
I can see that it generate more data in my sample file but nothing more, it doesnt send data to splunk.
I will try this with extract of splunk_gen

Re: generate dummy data -eventgen

adonio — Wed, 03 Jul 2019 10:39:47 GMT

yes, the working config file is within the TA,go to the default directory and look for eventgen.conf
here is the content of the file from that TA version 3.2.5

################
####   ASA  ####
################

[samplelog.cisco.asa]
sourcetype=cisco:asa
interval = 150
earliest = -60m
latest = now

##replace timestamp
token.0.token = ^(\w{3}\s+\d{1,2}\s+\d{1,2}:\d{1,2}:\d{1,2})
token.0.replacementType = timestamp
token.0.replacement = %b %d %H:%M:%S

##replace timestamp 2
token.1.token = ^(\d{4}\-\d{2}\-\d{2}\s+\d{1,2}:\d{1,2}:\d{1,2})
token.1.replacementType = timestamp
token.1.replacement = %Y-%m-%d %H:%M:%S

##replace timestamp 3
token.2.token = ^(\w{3}\s\d{1,2}\s\d{1,4}\s\d{1,2}:\d{1,2}:\d{1,2})
token.2.replacementType = timestamp
token.2.replacement = %b %d %Y %H:%M:%S

##repalce user
token.3.token = (UUUUUUUU)
token.3.replacementType = file
token.3.replacement = $SPLUNK_HOME\etc\apps\Splunk_TA_cisco-asa\samples\userName.sample

##replace local address
token.4.token = \sladdr\s(XXX\.XXX\.XXX\.XXX)
token.4.replacementType = file
token.4.replacement = $SPLUNK_HOME\etc\apps\Splunk_TA_cisco-asa\samples\internal_ips.sample

##replace foreign address
token.5.token = \sfaddr\s(XXX\.XXX\.XXX\.XXX)
token.5.replacementType = random
token.5.replacement = ipv4

##replace outside ips
token.6.token = (?:O|o)utside\S*(?::|/)(XXX\.XXX\.XXX\.XXX)
token.6.replacementType = random
token.6.replacement = ipv4

##replace inside ips
token.7.token = (?:I|i)nside\S*(?::|/)(XXX\.XXX\.XXX\.XXX)
token.7.replacementType = file
token.7.replacement = $SPLUNK_HOME\etc\apps\Splunk_TA_cisco-asa\samples\internal_ips.sample

##replace username part of email
token.8.token = YYYYYYYYYY
token.8.replacementType = file
token.8.replacement = $SPLUNK_HOME\etc\apps\Splunk_TA_cisco-asa\samples\userName.sample

##replacing remaining ip
token.9.token = XXX\.XXX\.XXX\.XXX
token.9.replacementType = random
token.9.replacement = ipv4

##replacing Hostname
token.10.token = (HHHHHHHH)
token.10.replacementType = file
token.10.replacement = $SPLUNK_HOME\etc\apps\Splunk_TA_cisco-asa\samples\hostname.sample

##replacing ipv6
token.11.token = VVVVVVVVVV
token.11.replacementType = random
token.11.replacement = ipv6

##replacing internal IP
token.12.token = (##INTERNAL_IP##)
token.12.replacementType = file
token.12.replacement = $SPLUNK_HOME\etc\apps\Splunk_TA_cisco-asa\samples\internal_ips.sample

##replacing IPv6
token.13.token = (##IP_V6##)
token.13.replacementType = random
token.13.replacement = ipv6

################
####  FWSM  ####
################

[samplelog.cisco.fwsm]
sourcetype=cisco:fwsm
interval = 150
earliest = -60m
latest = now

#replace timestamp 1
token.0.token = ^(\w{3}\s+\d{1,2}\s\d{1,2}:\d{1,2}:\d{1,2})
token.0.replacementType = timestamp
token.0.replacement = %b %d %H:%M:%S

##replace timestamp 2
token.1.token = ^(\w{3}\s\d{1,2}\s\d{1,4}\s\d{1,2}:\d{1,2}:\d{1,2})
token.1.replacementType = timestamp
token.1.replacement = %b %d %Y %H:%M:%S

##replace user
token.2.token = (UUUUUUUU)
token.2.replacementType = file
token.2.replacement = $SPLUNK_HOME\etc\apps\Splunk_TA_cisco-asa\samples\userName.sample

##replace local address
token.3.token = \sladdr\s(XXX\.XXX\.XXX\.XXX)
token.3.replacementType = file
token.3.replacement = $SPLUNK_HOME\etc\apps\Splunk_TA_cisco-asa\samples\internal_ips.sample

##replace foreign address
token.4.token = \sfaddr\s(XXX\.XXX\.XXX\.XXX)
token.4.replacementType = random
token.4.replacement = ipv4

##replace outside ips
token.5.token = (?:O|o)utside\S*(?::|/)(XXX\.XXX\.XXX\.XXX)
token.5.replacementType = random
token.5.replacement = ipv4

##replace inside ips
token.6.token = (?:I|i)nside\S*(?::|/)(XXX\.XXX\.XXX\.XXX)
token.6.replacementType = file
token.6.replacement = $SPLUNK_HOME\etc\apps\Splunk_TA_cisco-asa\samples\internal_ips.sample

##replacing remaining ip
token.7.token = XXX\.XXX\.XXX\.XXX
token.7.replacementType = random
token.7.replacement = ipv4

##replacing Hostname
token.8.token = (HHHHHHHH)
token.8.replacementType = file
token.8.replacement = $SPLUNK_HOME\etc\apps\Splunk_TA_cisco-asa\samples\hostname.sample

################
####   PIX  ####
################

[samplelog.cisco.pix]
sourcetype=cisco:pix
interval = 150
earliest = -60m
latest = now

#replace timestamp 1
token.0.token = ^(\w{3}\s+\d{1,2}\s\d{1,2}:\d{1,2}:\d{1,2})
token.0.replacementType = timestamp
token.0.replacement = %b %d %H:%M:%S

##replace timestamp 2
token.1.token = ^(\w{3}\s\d{1,2}\s\d{1,4}\s\d{1,2}:\d{1,2}:\d{1,2})
token.1.replacementType = timestamp
token.1.replacement = %b %d %Y %H:%M:%S

##replace user
token.2.token = (UUUUUUUU)
token.2.replacementType = file
token.2.replacement = $SPLUNK_HOME\etc\apps\Splunk_TA_cisco-asa\samples\userName.sample

##replace outside ips
token.3.token = outside\s?:\s?(?:Allocated ip = )?(XXX\.XXX\.XXX\.XXX)
token.3.replacementType = random
token.3.replacement = ipv4

##replace inside ips
token.4.token = inside\s?:\s?(?:.*\()?(XXX\.XXX\.XXX\.XXX)
token.4.replacementType = file
token.4.replacement = $SPLUNK_HOME\etc\apps\Splunk_TA_cisco-asa\samples\internal_ips.sample

##replacing remaining ip
token.5.token = XXX\.XXX\.XXX\.XXX
token.5.replacementType = random
token.5.replacement = ipv4

##replacing Hostname
token.6.token = (HHHHHHHH)
token.6.replacementType = file
token.6.replacement = $SPLUNK_HOME\etc\apps\Splunk_TA_cisco-asa\samples\hostname.sample

Re: generate dummy data -eventgen

net1993 — Wed, 03 Jul 2019 10:41:59 GMT

Thanks. I will try it and response if ok.

Re: generate dummy data -eventgen

koshyk — Wed, 03 Jul 2019 11:43:33 GMT

if you take a copy of splunk_eventgen into say a temporary directory and the jinja2/markupsafe

cp -r SA-Eventgen/lib/splunk_eventgen /tmp/
cp -r SA-Eventgen/lib/markupsafe /tmp/splunk_eventgen/lib/
cp -r SA-Eventgen/lib/jinja2 /tmp/splunk_eventgen/lib/

Copy one of the sample tutorial config to your name

cp /tmp/splunk_eventgen/README/eventgen.conf.tutorial1 /tmp/splunk_eventgen/README/mytest.tutorial

Validate the mytest.tutorial file and ensure it has . (comment out all outputMode other than splunkstream)

outputMode=splunkstream
splunkHost = localhost
splunkUser = <your_admin_user>
splunkPass = <your_admin_passwd>

rest should remain the same.

then

cd /tmp/splunk_eventgen
/opt/splunk/bin/splunk cmd python __main_.py generate README/mytest.tutorial

This will pump the sample data directly into Splunk

Re: generate dummy data -eventgen

net1993 — Fri, 05 Jul 2019 11:41:10 GMT

@adonio, I see only ver. 3.4.0 in splunk base and cannot find older realeses. In the one I get, there is not eventconf.conf neither sample data.
I guess, I can use eventconf.conf with content which you have post above but I don't have the sample data. Can you provide sample data?

Re: generate dummy data -eventgen

rendi7936 — Mon, 06 Jan 2020 02:54:44 GMT

Thank you, sir for your answer. But, i still confuse about different outputMode in Eventgen.

What is the point of using different outputMode in Eventgen ?

Because when i use the same sample log for different outputMode like tcpout and udpout. Both, will get the same result. Even, i cannot identify which one from tcpout or udpout.

At documentation ( https://splunk.github.io/eventgen/REFERENCE.html ), every outputMode should have different MUST HAVE configuration. In your example, when using splunkstream. You must have splunkHost, splunkUser, and splunkPass configuration.

When i use httpevent outputMode, i also set httpeventServers too. In httpeventServers, i also include valid token. And it works. But, when i use wrong token, it still works too. What's going on here ?

Is it supposed behavior that EventGen do ?

Re: generate dummy data -eventgen

koshyk — Fri, 10 Jan 2020 10:35:45 GMT

it depends on how you want to get the data into Splunk.

outputMode = modinput | s2s | file | splunkstream | stdout | devnull | spool | httpevent | syslogout | tcpout | udpout | metric_httpevent
    * Specifies how to output log data. Modinput is default.
    * If setting spool, should set spoolDir
    * If setting file, should set fileName
    * If setting splunkstream, should set splunkHost, splunkPort, splunkMethod,
      splunkUser and splunkPassword if not Splunk embedded
    * If setting s2s, should set splunkHost and splunkPort
    * If setting syslogout, should set syslogDestinationHost and syslogDestinationPort
    * If setting httpevent, should set httpeventServers
    * If setting metric_httpevent, should set httpeventServers and make sure your index is a splunk metric index

essentially, the simplified settings would be (in my experience)
1. use splunkstream if you want to stream data directly into Splunk. Good for DEV systems
2. use file if you want to output into a file and then use inputs.conf to read it. Good for remote/clustered systems
3. use syslogout if you want to integrate with syslog

Rest of settings are rarely used. Try one of the above and see if it works (baby steps). Then once working, you can extend it to complex output types.

regarding your query about httpevent, ensure below settings are in place.

httpeventServers = 
    * valid json that contains a list of server objects
    * valid server objects contain a protocol, a address, a port and a session key
    * {"servers":[{ "protocol":"https", "address":"127.0.0.1", "port":"8088", "key":"12345-12345-123123123123123123"}]}

When you say wrong token, do you mean a separate "key" ? (Please beware that accessToken is NOT for httpevent)

Re: generate dummy data -eventgen

ag0x00 — Fri, 01 May 2020 18:42:08 GMT

I agree, the docs hosted at http://splunk.github.io/eventgen/ are incredibly confusing for first-timers.

They don't make clear a simple fact, for example, that you need a combination of event templates and files containing rotating values (users, hosts, etc) in order to get the tool running. You have to find the templates based on real log strings or build your own. local/eventgen.conf stores this combined config. Everything else is secondary.

rav3n's Splunk EventGen — Quick Tutorial on Medium helped me to get started and to wrap my head around how eventgen works. Check it out here: https://medium.com/@rav3n/splunk-eventgen-quick-tutorial-593f526bafc1.

Re: generate dummy data -eventgen

TheLawsOfChaos — Sun, 11 Aug 2024 03:59:03 GMT

I'm not normally one to resurrect dead posts, but as I was myself trying to accomplish the same task and via Google found this post, figured I'd give an update.

Per the documentation for the TA (https://docs.splunk.com/Documentation/AddOns/released/CiscoASA/Releasehistory) they removed the eventgen support in version 3.2.5