Splunk Homelab Guide (9.4)

Downloads (Splunk 9.4)

Grab Splunk 9.4.x from the official download pages. If you want a specific 9.4 patch build, the Previous Releases page makes it easy to pick one.

Baseline best practices (9.4)

1. Keep cluster configs consistent across members and keep deployer owned assets centralized.
2. Use a dedicated deployer for search head clusters. Do not place the deployer on a cluster member.
3. Configure indexer clusters from the manager node and push indexes from there.
4. Use the deployment server for forwarders only, not for indexer cluster peers.

Easy lab: Single box (all-in-one)

One Linux VM running a single Splunk Enterprise instance with search head + indexer on the same host. Fast to set up and perfect for SPL experiments and dashboards.

In this tier you are learning the basics: how Splunk stores data, how searches run, and how to get from raw logs to dashboards. Everything lives in one place so you can focus on workflows instead of wiring.

1) Install Splunk Enterprise

This puts the Splunk binaries and default configuration in place. On Linux the default install path is /opt/splunk. On Windows it is under C:\Program Files\Splunk unless you choose a custom path.

sudo dpkg -i splunk-9.4.x-linux-amd64.deb

sudo rpm -i splunk-9.4.x-linux-amd64.rpm

2) Start Splunk and set admin password

The first start initializes the instance and prompts for an admin password. Enabling boot-start keeps Splunk running across reboots.

sudo /opt/splunk/bin/splunk start --accept-license
sudo /opt/splunk/bin/splunk enable boot-start

sudo /opt/splunk/bin/splunk start --accept-license
sudo /opt/splunk/bin/splunk enable boot-start

cd \"C:\\Program Files\\Splunk\\bin\"
splunk.exe start --accept-license

3) Enable receiving port for forwarders (optional)

If you want to forward logs from other hosts, enable a listening port and open it in your firewall (example uses 9997):

This port is the “landing zone” for Universal Forwarders. You can skip it if all your data is local to the same host.

sudo /opt/splunk/bin/splunk enable listen 9997
sudo /opt/splunk/bin/splunk restart

cd \"C:\\Program Files\\Splunk\\bin\"
splunk.exe enable listen 9997
splunk.exe restart

4) Add a quick data input

This gives you instant data to search. We’re using standard OS logs because they always exist and generate steady events.

sudo /opt/splunk/bin/splunk add monitor /var/log/syslog -index main -sourcetype syslog

cd \"C:\\Program Files\\Splunk\\bin\"
splunk.exe add monitor \"C:\\Windows\\System32\\winevt\\Logs\" -index main -sourcetype WinevtLog

Standard lab: Split roles (Search head + Indexer + Deployment server + Forwarder)

This layout mirrors a small production deployment and keeps indexing work away from the search UI. It’s the sweet spot for a realistic lab without a huge footprint.

The goal here is separation of duties: indexers ingest and store data, search heads handle users and searches, and the deployment server pushes configs to forwarders.

Topology

1) Install Splunk Enterprise on SH1, IDX1, DS1

Use the same 9.4.x build on SH1, IDX1, and DS1.

Matching versions keeps auth and bundle replication simple. You can mix later, but for labs it’s best to keep everything in lockstep.

sudo dpkg -i splunk-9.4.x-linux-amd64.deb

sudo rpm -i splunk-9.4.x-linux-amd64.rpm

2) Configure IDX1 to receive data

The indexer needs a receiving port so forwarders can send data. We’ll use 9997 (the Splunk default).

[splunktcp://9997]
connection_host = ip

sudo /opt/splunk/bin/splunk restart

[splunktcp://9997]
connection_host = ip

net stop splunkd
net start splunkd

3) Point SH1 at IDX1 as a search peer

This tells the search head where the data lives. Once added, searches on SH1 will fan out to IDX1 and return results.

sudo /opt/splunk/bin/splunk add search-server https://IDX1:8089 -auth admin:YOURPASSWORD

cd \"C:\\Program Files\\Splunk\\bin\"
splunk.exe add search-server https://IDX1:8089 -auth admin:YOURPASSWORD

4) Install Universal Forwarder on UF1

Forwarders are lightweight collectors. They tail files, package events, and send them to your indexer.

sudo dpkg -i splunkforwarder-9.4.x-linux-amd64.deb
sudo /opt/splunkforwarder/bin/splunk start --accept-license
sudo /opt/splunkforwarder/bin/splunk enable boot-start

sudo rpm -i splunkforwarder-9.4.x-linux-amd64.rpm
sudo /opt/splunkforwarder/bin/splunk start --accept-license
sudo /opt/splunkforwarder/bin/splunk enable boot-start

cd \"C:\\Program Files\\SplunkForwarder\\bin\"
splunk.exe start --accept-license

5) Forward UF1 data to IDX1

Here we define the destination indexer and add a monitor input. You can add multiple monitors as your lab grows.

sudo /opt/splunkforwarder/bin/splunk add forward-server IDX1:9997
sudo /opt/splunkforwarder/bin/splunk add monitor /var/log/syslog -index main -sourcetype syslog
sudo /opt/splunkforwarder/bin/splunk restart

cd \"C:\\Program Files\\SplunkForwarder\\bin\"
splunk.exe add forward-server IDX1:9997
splunk.exe add monitor \"C:\\Windows\\System32\\winevt\\Logs\" -index main -sourcetype WinevtLog
splunk.exe restart

6) (Optional) Enable deployment server for forwarders

On UF1, set deployment server polling:

This is how you scale. Instead of configuring every forwarder manually, you assign server classes and let the deployment server push apps and inputs.

sudo /opt/splunkforwarder/bin/splunk set deploy-poll DS1:8089
sudo /opt/splunkforwarder/bin/splunk restart

cd \"C:\\Program Files\\SplunkForwarder\\bin\"
splunk.exe set deploy-poll DS1:8089
splunk.exe restart

You can now create server classes on DS1 to push apps and inputs to forwarders.

Advanced lab: Clustered search + clustered indexers

This is a production grade layout: clustered indexers and clustered search heads, plus a management tier. It’s heavier, but it’s the closest thing to how real Splunk deployments are usually run.

You’re building for resilience here. Indexers replicate data for redundancy, search heads replicate knowledge objects, and management nodes keep everything coordinated.

Topology

1) Configure the indexer cluster manager (CM1)

The manager orchestrates the indexer cluster: it tells peers how to replicate data and maintains cluster health. The replication and search factors define durability and search availability.

[clustering]
mode = manager
replication_factor = 3
search_factor = 2
pass4SymmKey = yourClusterKey
cluster_label = lab-indexer-cluster

sudo /opt/splunk/bin/splunk restart

[clustering]
mode = manager
replication_factor = 3
search_factor = 2
pass4SymmKey = yourClusterKey
cluster_label = lab-indexer-cluster

net stop splunkd
net start splunkd

2) Configure each indexer peer (IDX1-3)

Peers do the actual indexing. The replication port is used for peer to peer bucket replication. The receiving port handles forwarder data.

[replication_port://9887]

[clustering]
mode = peer
manager_uri = https://CM1:8089
pass4SymmKey = yourClusterKey

[splunktcp://9997]
connection_host = ip

sudo /opt/splunk/bin/splunk restart

[replication_port://9887]

[clustering]
mode = peer
manager_uri = https://CM1:8089
pass4SymmKey = yourClusterKey

[splunktcp://9997]
connection_host = ip

net stop splunkd
net start splunkd

3) Configure a cluster search head (SHC1-3)

Search head members connect to the indexer cluster manager and receive the cluster configuration. In the next step we will join them into an SHC.

[clustering]
mode = searchhead
manager_uri = https://CM1:8089
pass4SymmKey = yourClusterKey

[clustering]
mode = searchhead
manager_uri = https://CM1:8089
pass4SymmKey = yourClusterKey

Then configure each search head cluster member to join the SHC and keep the same security key + label across all members.

4) Configure the SHC deployer (DEP1)

The deployer is the control plane for search head apps and configs. You push bundles to the deployer, and it replicates to SHC members.

[shclustering]
pass4SymmKey = yourShcKey
shcluster_label = lab-shc

sudo /opt/splunk/bin/splunk restart

[shclustering]
pass4SymmKey = yourShcKey
shcluster_label = lab-shc

net stop splunkd
net start splunkd

5) Initialize the search head cluster

On each SHC member (replace SHC1 with the local host):

This command tells each search head where the deployer is and how to replicate config. After all members are initialized, you bootstrap the captain.

sudo /opt/splunk/bin/splunk init shcluster-config \
  -mgmt_uri https://SHC1:8089 \
  -replication_port 8191 \
  -replication_factor 3 \
  -conf_deploy_fetch_url https://DEP1:8089 \
  -secret yourShcKey

cd \"C:\\Splunk\\bin\"
splunk.exe init shcluster-config ^
  -mgmt_uri https://SHC1:8089 ^
  -replication_port 8191 ^
  -replication_factor 3 ^
  -conf_deploy_fetch_url https://DEP1:8089 ^
  -secret yourShcKey

Bootstrap the captain (run on a single member):

The captain coordinates searches and bundle replication. Only one member should be bootstrapped to avoid split brain.

sudo /opt/splunk/bin/splunk bootstrap shcluster-captain -servers_list \"https://SHC1:8089,https://SHC2:8089,https://SHC3:8089\"

cd \"C:\\Splunk\\bin\"
splunk.exe bootstrap shcluster-captain -servers_list \"https://SHC1:8089,https://SHC2:8089,https://SHC3:8089\"

6) Deployment server for forwarders (DS1)

Use forwarder management on DS1 for universal forwarders only. Don’t push updates to indexer cluster peers with a deployment server.

Forwarders talk to DS1, indexers talk to the cluster manager, and search heads talk to the deployer. Keeping those lanes separate prevents config collisions.

Verification checklist

• Search head opens Splunk Web and runs a test search.
• Indexers are green and searchable in the Monitoring Console.
• Forwarders appear under Forwarder Management and are phoning home.
• Cluster manager shows the expected replication/search factor.
• SHC captain is elected and bundle deploys successfully.