SPLK-2002 Splunk Enterprise Certified Architect Questions and Answers

According to the Splunk SmartStore Architecture Guide, the primary benefit of SmartStore is the separation of storage from compute resources within an indexer cluster. SmartStore enables Splunk to decouple indexer storage (data at rest) from the compute layer (indexers that perform searches and indexing).

With SmartStore, active (hot/warm) data remains on local disk for fast access, while older, less frequently searched (remote) data is stored in an external object storage system such as Amazon S3, Google Cloud Storage, or on-premises S3-compatible storage. This separation reduces the storage footprint on indexers, allowing organizations to scale compute and storage independently.

This architecture improves cost efficiency and scalability by:

Lowering on-premises storage costs using object storage for retention.

Enabling dynamic scaling of indexers without impacting total data availability.

Reducing replication overhead since SmartStore manages data objects efficiently.

SmartStore does not affect replication or search factors (Option A), does not handle Knowledge Object replication (Option C), and the Cluster Manager is still required (Option D) to coordinate cluster activities.

References (Splunk Enterprise Documentation):

• SmartStore Overview and Architecture Guide

• SmartStore Deployment and Configuration Manual

• Managing Storage and Compute Independence in Indexer Clusters

• Splunk Enterprise Capacity Planning – SmartStore Sizing Guidelines

According to Splunk’s Enterprise Security (ES) Installation and Sizing Guide, the majority of processing and computational load generated by the Splunk App for Enterprise Security is concentrated on the Search Head(s).

This is because Splunk ES is built around a search-driven correlation model — it continuously runs scheduled correlation searches, data model accelerations, and notables generation jobs. These operations rely on the search head tier’s CPU, memory, and I/O resources rather than on indexers. ES also performs extensive data model summarization, CIM normalization, and real-time alerting, all of which are search-intensive operations.

While indexers handle data ingestion and indexing, they are not heavily affected by ES beyond normal search request processing. The Cluster Manager only coordinates replication and plays no role in search execution, and Heavy Forwarders serve as data collection or parsing points with minimal analytical load.

Splunk officially recommends deploying ES on a dedicated Search Head Cluster (SHC) to isolate its high CPU and memory demands from other workloads. For large-scale environments, horizontal scaling via SHC ensures consistent performance and stability.

References (Splunk Enterprise Documentation):

• Splunk Enterprise Security Installation and Configuration Guide

• Search Head Sizing for Splunk Enterprise Security

• Enterprise Security Overview – Workload Distribution and Performance Impact

• Splunk Architecture and Capacity Planning for ES Deployments

Indexed extraction configurations are processed in the indexing phase of the Splunk Enterprise data pipeline. The data pipeline is the process that Splunk uses to ingest, parse, index, and search data. Indexed extraction configurations are settings that determine how Splunk extracts fields from data at index time, rather than at search time. Indexed extraction can improve search performance, but it also increases the size of the index. Indexed extraction configurations are applied in the indexing phase, which is the phase where Splunk writes the data and the .tsidx files to the index. The input phase is the phase where Splunk receives data from various sources and formats. The parsing phase is the phase where Splunk breaks the data into events, timestamps, and hosts. The search phase is the phase where Splunk executes search commands and returns results.

Data Model acceleration is a feature that enables faster searches over large data sets by summarizing the raw data into a more efficient format. Data Model acceleration consumes additional disk space, as it stores both the raw data and the summarized data. The amount of disk space required depends on the size and complexity of the Data Model, the retention period of the summarized data, and the compression ratio of the data. According to the Splunk Enterprise Security Planning and Installation Manual, Data Model acceleration is one of the factors that strongly impacts storage sizing requirements for Enterprise Security. The other factors are the volume and type of data sources, the retention policy of the data, and the replication factor and search factor of the index cluster. The number of scheduled (correlation) searches, the number of Splunk users configured, and the number of source types used in the environment are not directly related to storage sizing requirements for Enterprise Security1

1: https://docs.splunk.com/Documentation/ES/6.6.0/Install/Plan#Storage_sizing_requirements

A dedicated deployment server is a Splunk instance that manages the distribution of configuration updates and apps to a set of deployment clients, such as forwarders, indexers, or search heads. A dedicated deployment server should be used when there are more than 50 deployment clients, because this number exceeds the recommended limit for a non-dedicated deployment server. A non-dedicated deployment server is a Splunk instance that also performs other roles, such as indexing or searching. Using a dedicated deployment server can improve the performance, scalability, and reliability of the deployment process. Option C is the correct answer. Option A is incorrect because the number of search peers does not affect the need for a dedicated deployment server. Search peers are indexers that participate in a distributed search. Option B is incorrect because the number of apps to deploy does not affect the need for a dedicated deployment server. Apps are packages of configurations and assets that provide specific functionality or views in Splunk. Option D is incorrect because the number of server classes does not affect the need for a dedicated deployment server. Server classes are logical groups of deployment clients that share the same configuration updates and apps12

1: https://docs.splunk.com/Documentation/Splunk/9.1.2/Updating/Aboutdeploymentserver 2: https://docs.splunk.com/Documentation/Splunk/9.1.2/Updating/Whentousedeploymentserver

 The following artifacts are included in a Splunk diag file:

Internal logs. These are the log files that Splunk generates to record its own activities, such as splunkd.log, metrics.log, audit.log, and others. These logs can help troubleshoot Splunk issues and monitor Splunk performance.

Configuration files. These are the files that Splunk uses to configure various aspects of its operation, such as server.conf, indexes.conf, props.conf, transforms.conf, and others. These files can help understand Splunk settings and behavior. The following artifacts are not included in a Splunk diag file:

OS settings. These are the settings of the operating system that Splunk runs on, such as the kernel version, the memory size, the disk space, and others. These settings are not part of the Splunk diag file, but they can be collected separately using the diag --os option.

Customer data. These are the data that Splunk indexes and makes searchable, such as the rawdata and the tsidx files. These data are not part of the Splunk diag file, as they may contain sensitive or confidential information. For more information, see Generate a diagnostic snapshot of your Splunk Enterprise deployment in the Splunk documentation.

According to the Splunk Search Head Clustering Troubleshooting Guide, when a Search Head Cluster (SHC) member is reverted from a backup or experiences configuration drift (e.g., an outdated Raft state), it can fail to rejoin the cluster due to inconsistent Raft metadata. The Raft database stores the SHC’s internal consensus and replication state, including knowledge object synchronization, captain election history, and peer membership information.

If this Raft metadata becomes corrupted or outdated (as in the scenario where a node is restored from backup), the recommended and Splunk-supported remediation is to clean the Raft metadata using:

splunk clean raft

This command resets the node’s local Raft state so it can re-synchronize with the current SHC captain and rejoin the cluster cleanly.

The steps generally are:

Stop the affected SHC member.

Run splunk clean raft on that node.

Restart Splunk.

Verify that it successfully rejoins the SHC.

Deleting configuration stanzas or forcing re-addition (Options B and C) can lead to further inconsistency or data loss. Reviewing logs (Option A) helps diagnose issues but does not resolve Raft corruption.

References (Splunk Enterprise Documentation):

• Troubleshooting Raft Metadata Corruption in Search Head Clusters

• splunk clean raft Command Reference

• Search Head Clustering: Recovering from Backup and Membership Failures

• Splunk Enterprise Admin Manual – Raft Consensus and SHC Maintenance

Splunk provides various methods to change the internal logging levels in a Splunk environment to troubleshoot an issue. All of the options are valid ways to do so. Option A is correct because the Monitoring Console (MC) allows the administrator to view and modify the logging levels of various Splunk components through a graphical interface. Option B is correct because the Splunk command line provides the splunk set log-level command to change the logging levels of specific components or categories. Option C is correct because the Splunk Web provides the Settings > Server settings > Server logging page to change the logging levels of various components through a web interface. Option D is correct because the log-local.cfg file allows the administrator to manually edit the logging levels of various components by overriding the default settings in the log.cfg file123

1: https://docs.splunk.com/Documentation/Splunk/9.1.2/Troubleshooting/Enabledebuglogging 2: https://docs.splunk.com/Documentation/Splunk/9.1.2/Admin/Serverlogging 3: https://docs.splunk.com/Documentation/Splunk/9.1.2/Admin/Loglocalcfg

The Splunk Enterprise Capacity Planning Manual states that running Splunk in a virtualized environment typically results in a performance reduction of approximately 20% to 45% compared to equivalent deployments on physical hardware.

This degradation is primarily due to the virtualization overhead inherent in hypervisor environments (such as VMware, Hyper-V, or KVM), which can affect:

Disk I/O throughput and latency — the most critical factor for indexers.

CPU scheduling efficiency, particularly for multi-threaded indexing processes.

Network latency between clustered components.

Splunk’s documentation strongly emphasizes that while virtualized environments offer operational flexibility, they cannot match bare-metal performance, especially under heavy indexing loads.

To mitigate performance loss, Splunk recommends:

Reserving dedicated CPU and I/O resources for Splunk VMs.

Avoiding over-commitment of hardware resources.

Using high-performance SSD storage or paravirtualized disk controllers.

These optimizations can narrow the performance gap, but a 20–45% reduction remains a realistic expectation under typical conditions.

References (Splunk Enterprise Documentation):

• Splunk Enterprise Capacity Planning Manual – Virtualization Performance Considerations

• Splunk on Virtual Infrastructure – Best Practices and Performance Tuning

• Indexer and Search Head Hardware Recommendations

• Performance Testing Guidelines for Splunk Deployments

 The following is a good practice for a search head cluster deployer: The deployer must be used to distribute non-replicable configurations to search head cluster members. Non-replicable configurations are the configurations that are not replicated by the search factor, such as the apps and the server.conf settings. The deployer is the Splunk server role that distributes these configurations to the search head cluster members, ensuring that they have the same configuration. The deployer does not only distribute configurations to search head cluster members when they “phone home”, as this would cause configuration inconsistencies and delays. The deployer does not distribute configurations to search head cluster members to be valid configurations, as this implies that the configurations are invalid without the deployer. The deployer does not only distribute configurations to search head cluster members with splunk apply shcluster-bundle, as this would require manual intervention by the administrator. For more information, see Use the deployer to distribute apps and configuration updates in the Splunk documentation.

To resync a stale KV Store member in a search head cluster, you need to stop the search head that has the stale KV Store member, run the command splunk clean kvstore --local, and then restart the search head. This triggers the initial synchronization from other KV Store members12.

The command splunk resync kvstore [-source sourceId] is used to resync the entire KV Store cluster from one of the members, not a single member. This command can only be invoked from the node that is operating as search head cluster captain2.

The command splunk clean eventdata -local is used to delete all indexed data from a standalone indexer or a cluster peer node, not to resync the KV Store3.

 To reduce the captain’s work load in a search head cluster, the setting that will prevent scheduled searches from running on the captain is captain_is_adhoc_searchhead = true (on the current captain). This setting will designate the current captain as an ad hoc search head, which means that it will not run any scheduled searches, but only ad hoc searches initiated by users. This will reduce the captain’s work load and improve the search head cluster performance. The adhoc_searchhead = true (on all members) setting will designate all search head cluster members as ad hoc search heads, which means that none of them will run any scheduled searches, which is not desirable. The adhoc_searchhead = true (on the current captain) setting will have no effect, as this setting is ignored by the captain. The captain_is_adhoc_searchhead = true (on all members) setting will have no effect, as this setting is only applied to the current captain. For more information, see Configure the captain as an ad hoc search head in the Splunk documentation.

In the deployment planning process, a person should identify who gets to see network data in the data policy definition step. This step involves defining the data access policies and permissions for different users and roles in Splunk. The deployment schedule step involves defining the timeline and milestones for the deployment project. The topology diagramming step involves creating a visual representation of the Splunk architecture and components. The data source inventory step involves identifying and documenting the data sources and types that will be ingested by Splunk

The Splunk Capacity Planning Manual defines the total required storage for indexes as a function of daily ingest rate, compression factor, retention period, and an additional padding buffer for index management and growth.

The formula is:

Storage = (Daily Data * Compression Factor * Retention Days) + Padding

Given the values:

Daily rate = 20 GB

Compression factor = 0.5 (50% reduction)

Retention period = 30 days

Padding = 100 GB

Plugging these into the formula gives:

20 * 0.5 * 30 + 100 = 400 GB

This result represents the estimated storage needed to retain 30 days of compressed indexed data with an additional buffer to accommodate growth and Splunk’s bucket management overhead.

Compression factor values typically range between 0.5 and 0.7 for most environments, depending on data type. Using compression in calculations is critical, as indexed data consumes less space than raw input after Splunk’s tokenization and compression processes.

Other options either misapply the compression ratio or the order of operations, producing incorrect totals.

References (Splunk Enterprise Documentation):

• Capacity Planning for Indexes – Storage Sizing and Compression Guidelines

• Managing Index Storage and Retention Policies

• Splunk Enterprise Admin Manual – Understanding Index Bucket Sizes

• Indexing Performance and Storage Optimization Guide

 Search head clustering is a Splunk feature that allows a group of search heads to share configurations, apps, and knowledge objects, and to provide high availability and scalability for searching. Search head clustering has the following characteristics:

A deployer is required. A deployer is a Splunk instance that distributes the configurations and apps to the members of the search head cluster. The deployer is not a member of the cluster, but a separate instance that communicates with the cluster master.

At least three search heads are needed. A search head cluster must have at least three search heads to form a quorum and to ensure high availability. If the cluster has less than three search heads, it cannot function properly and will enter a degraded mode.

The deployer must have sufficient CPU and network resources to process service requests and push configurations. The deployer is responsible for handling the requests from the cluster master and the cluster members, and for pushing the configurations and apps to the cluster members. Therefore, the deployer must have enough CPU and network resources to perform these tasks efficiently and reliably.

Search heads do not need to meet the high-performance reference server requirements, as this is not a mandatory condition for search head clustering. The high-performance reference server requirements are only recommended for optimal performance and scalability of Splunk deployments, but they are not enforced by Splunk.

 The following artifacts are included in a Splunk diag file:

Server specs. These are the specifications of the server that Splunk runs on, such as the CPU model, the memory size, the disk space, and the network interface. These specs can help understand the Splunk hardware requirements and performance.

Current open connections. These are the connections that Splunk has established with other Splunk instances or external sources, such as forwarders, indexers, search heads, license masters, deployment servers, and data inputs. These connections can help understand the Splunk network topology and communication.

Internal Splunk log files. These are the log files that Splunk generates to record its own activities, such as splunkd.log, metrics.log, audit.log, and others. These logs can help troubleshoot Splunk issues and monitor Splunk performance.

Index listings. These are the listings of the indexes that Splunk has created and configured, such as the index name, the index location, the index size, and the index attributes. These listings can help understand the Splunk data management and retention. The following artifacts are not included in a Splunk diag file:

Search history. This is the history of the searches that Splunk has executed, such as the search query, the search time, the search results, and the search user. This history is not part of the Splunk diag file, but it can be accessed from the Splunk Web interface or the audit.log file.

Splunk users and their roles. These are the users that Splunk has created and assigned roles to, such as the user name, the user password, the user role, and the user capabilities. These users and roles are not part of the Splunk diag file, but they can be accessed from the Splunk Web interface or the authentication.conf and authorize.conf files.

KV store listings. These are the listings of the KV store collections and documents that Splunk has created and stored, such as the collection name, the collection schema, the document ID, and the document fields. These listings are not part of the Splunk diag file, but they can be accessed from the Splunk Web interface or the mongod.log file.

Indexed data. These are the data that Splunk indexes and makes searchable, such as the rawdata and the tsidx files. These data are not part of the Splunk diag file, as they may contain sensitive or confidential information. For more information, see Generate a diagnostic snapshot of your Splunk Enterprise deployment in the Splunk documentation.

The following may explain the problem of why a colleague cannot see the src_ip field in their search results: The field was extracted as a private knowledge object, and the colleague did not explicitly use the field in the search and the search was set to Fast Mode. A knowledge object is a Splunk entity that applies some knowledge or intelligence to the data, such as a field extraction, a lookup, or a macro. A knowledge object can have different permissions, such as private, app, or global. A private knowledge object is only visible to the user who created it, and it cannot be shared with other users. A field extraction is a type of knowledge object that extracts fields from the raw data at index time or search time. If a field extraction is created as a private knowledge object, then only the user who created it can see the extracted field in their search results. A search mode is a setting that determines how Splunk processes and displays the search results, such as Fast, Smart, or Verbose. Fast mode is the fastest and most efficient search mode, but it also limits the number of fields and events that are displayed. Fast mode only shows the default fields, such as _time, host, source, sourcetype, and _raw, and any fields that are explicitly used in the search. If a field is not used in the search and it is not a default field, then it will not be shown in Fast mode. The events are tagged as communicate, but are missing the network tag, and the Typing Queue, which does regular expression replacements, is blocked, are not valid explanations for the problem. Tags are labels that can be applied to fields or field values to make them easier to search. Tags do not affect the visibility of fields, unless they are used as filters in the search. The Typing Queue is a component of the Splunk data pipeline that performs regular expression replacements on the data, such as replacing IP addresses with host names. The Typing Queue does not affect the field extraction process, unless it is configured to do so

Data retention is optional in the data source assessment because it is not directly related to the ingestion process. Data retention is determined by the index configuration and the storage capacity of the Splunk platform. Data format, data location, and data volume are all essential information for planning how to collect, parse, and index the data source.

Splunk Enterprise documentation clearly states that the best method to secure log traffic between Universal Forwarders (UFs) and Indexers is to implement Transport Layer Security (TLS) using signed SSL certificates. When Universal Forwarders send data to Indexers, this communication can be encrypted using SSL/TLS to prevent eavesdropping, data tampering, or interception while in transit.

Splunk provides default self-signed certificates out of the box, but these are only for testing or lab environments and should not be used in production. Production-grade security requires custom, signed SSL certificates — either from an internal Certificate Authority (CA) or a trusted public CA. These certificates validate both the sender (forwarder) and receiver (indexer), ensuring data integrity and authenticity.

In practice, this involves:

Generating or obtaining CA-signed certificates.

Configuring the forwarder’s outputs.conf to use SSL encryption (sslCertPath, sslPassword, and sslRootCAPath).

Configuring the indexer’s inputs.conf and server.conf to require and validate client certificates.

This configuration ensures end-to-end encryption for all log data transmitted from forwarders to indexers.

Routing traffic through a WAF (Option C) does not provide end-to-end encryption for Splunk’s internal communication, and securing search head–to–indexer communication (Option D) is unrelated to forwarder data flow.

References (Splunk Enterprise Documentation):

• Securing Splunk Enterprise: Encrypting Data in Transit Using SSL/TLS

• Configure Forwarder-to-Indexer Encryption

• Server and Forwarder Authentication with Signed Certificates

• Best Practices for Forwarder Management and Security Configuration

Increasing the number of parallel ingestion pipelines in server.conf is most likely to improve indexing performance when indexing is slow and real-time search results are delayed in a Splunk environment with two indexers and one search head. The parallel ingestion pipelines allow Splunk to process multiple data streams simultaneously, which increases the indexing throughput and reduces the indexing latency. Increasing the maximum number of hot buckets in indexes.conf will not improve indexing performance, but rather increase the disk space consumption and the bucket rolling time. Decreasing the maximum size of the search pipelines in limits.conf will not improve indexing performance, but rather reduce the search performance and the search concurrency. Decreasing the maximum concurrent scheduled searches in limits.conf will not improve indexing performance, but rather reduce the search capacity and the search availability. For more information, see Configure parallel ingestion pipelines in the Splunk documentation.

 A Technical Add-On (TA) is a Splunk app that contains configurations for data collection, parsing, and enrichment. It can also enable event data for a data model, which is useful for creating dashboards and reports. Therefore, before installing a TA, it is important to identify the number of scheduled or real-time searches that will use the data model, and to validate if the TA enables event data for a data model. The number of forwarders that the TA can support is not relevant, as the TA is installed on the indexer or search head, not on the forwarder. The installation location of the TA depends on the type of data and the use case, so it is not a fixed requirement

The correct answer is C. Knowledge Object replication. This is a valid use case that a search head cluster addresses, as it ensures that all the search heads in the cluster have the same set of knowledge objects, such as saved searches, dashboards, reports, and alerts1. The search head cluster replicates the knowledge objects across the cluster members, and synchronizes any changes or updates1. This provides a consistent user experience and avoids data inconsistency or duplication1. The other options are not valid use cases that a search head cluster addresses. Option A, providing redundancy in the event a search peer fails, is not a use case for a search head cluster, but for an indexer cluster, which maintains multiple copies of the indexed data and can recover from indexer failures2. Option B, search affinity, is not a use case for a search head cluster, but for a multisite indexer cluster, which allows the search heads to preferentially search the data on the local site, rather than on a remote site3. Option D, increased Search Factor (SF), is not a use case for a search head cluster, but for an indexer cluster, which determines how many searchable copies of each bucket are maintained across the indexers4. Therefore, option C is the correct answer, and options A, B, and D are incorrect.

1: About search head clusters 2: About indexer clusters and index replication 3: Configure search affinity 4: Configure the search factor

The Indexer Discovery feature enables forwarders to dynamically connect to the available peer nodes in an indexer cluster. To use this feature, the manager node must be configured with the [indexer_discovery] stanza and a pass4SymmKey value. The forwarders must also be configured with the same pass4SymmKey value and the master_uri of the manager node. The pass4SymmKey value must be encrypted using the splunk _encrypt command. Therefore, option A indicates that the Indexer Discovery feature has not been fully configured on the manager node, because the pass4SymmKey value is not encrypted. The other options are not related to the Indexer Discovery feature. Option B shows the configuration of a forwarder that is part of an indexer cluster. Option C shows the configuration of a manager node that is part of an indexer cluster. Option D shows an invalid configuration of the [indexer_discovery] stanza, because the pass4SymmKey value is not encrypted and does not match the forwarders’ pass4SymmKey value12

1: https://docs.splunk.com/Documentation/Splunk/9.1.2/Indexer/indexerdiscovery 2: https://docs.splunk.com/Documentation/Splunk/9.1.2/Security/Secureyourconfigurationfiles#Encrypt_the_pass4SymmKey_setting_in_server.conf

The master node is the Splunk server role that regulates the functioning of the indexer cluster. The master node coordinates the activities of the peer nodes, such as data replication, data searchability, and data recovery. The master node also manages the cluster configuration bundle and distributes it to the peer nodes. The indexer is the Splunk server role that indexes the incoming data and makes it searchable. The deployer is the Splunk server role that distributes apps and configuration updates to the search head cluster members. The monitoring console is the Splunk server role that monitors the health and performance of the Splunk deployment. For more information, see About indexer clusters and index replication in the Splunk documentation.

Splunk Enterprise officially requires a minimum of three search heads and one deployer for a supported Search Head Cluster (SHC) configuration. This ensures both high availability and data consistency within the cluster.

The Splunk documentation explains that a search head cluster uses RAFT-based consensus to elect a captain responsible for managing configuration replication, scheduling, and user workload distribution. The RAFT protocol requires a quorum of members to maintain consistency. In practical terms, this means a minimum of three members (search heads) to achieve fault tolerance — allowing one member to fail while maintaining operational stability.

The deployer is a separate Splunk instance responsible for distributing configuration bundles (apps, settings, and user configurations) to all members of the search head cluster. The deployer is not part of the SHC itself but is mandatory for its proper management.

Running with fewer than three search heads or replacing the deployer with a Deployment Server (as in Options B, C, or D) is unsupported and violates Splunk best practices for SHC resiliency and management.

References (Splunk Enterprise Documentation):

• Search Head Clustering Overview – Minimum Supported Architecture

• Deploy and Configure the Deployer for a Search Head Cluster

• High Availability and Fault Tolerance with RAFT in SHC

The correct tasks that the cluster manager performs in an indexer cluster are A. Generates and maintains the list of primary searchable buckets, B. If Indexer Discovery is enabled, provides the list of available peer nodes to forwarders, and D. Distributes app bundles to peer nodes. According to the Splunk documentation1, the cluster manager is responsible for these tasks, as well as managing the replication and search factors, coordinating the replication and search activities, and providing a web interface for monitoring and managing the cluster. Option C, ensuring all peer nodes are always using the same version of Splunk, is not a task of the cluster manager, but a requirement for the cluster to function properly2. Therefore, option C is incorrect, and options A, B, and D are correct.

1: About the cluster manager 2: Requirements and compatibility for indexer clusters

The splunk clean kvstore command is used to clear the KV store. This command will delete all the collections and documents in the KV store and reset it to an empty state. This command can be useful for troubleshooting KV store issues or resetting the KV store data. The splunk clear kvstore, splunk delete kvstore, and splunk reinitialize kvstore commands are not valid Splunk commands. For more information, see Use the CLI to manage the KV store in the Splunk documentation.

A deployment plan should include business continuity and disaster recovery plans, current logging details and data source inventory, and current and future topology diagrams of the IT environment. These elements are essential for planning, designing, and implementing a Splunk deployment that meets the business and technical requirements. A comprehensive list of stakeholders, either direct or indirect, is not part of the deployment plan, but rather part of the project charter. For more information, see Deployment planning in the Splunk documentation.

According to the Splunk documentation1, any search head in the cluster can update the KV store collection. The KV store collection is replicated across all the cluster members, and any write operation is delegated to the KV store captain, who then synchronizes the changes with the other members. The KV store primary search head is not a valid term, as there is no such role in a search head cluster. The other options are false because:

The search head cluster captain is not the only node that can update the KV store collection, as any member can initiate a write operation1.

Any search head except the captain can also update the KV store collection, as the write operation will be delegated to the captain1.

The splunk validate cluster-bundle --check-restart command is the officially documented Splunk Enterprise method to determine if a configuration bundle push will trigger a rolling restart within an indexer cluster.

When configuration changes are made on the Cluster Manager (Master Node)—for example, updates to indexes.conf, props.conf, or transforms.conf—Splunk administrators must validate the bundle before pushing it to all peer nodes. Using this command allows the Cluster Manager to simulate the deployment and verify whether the configuration modifications necessitate a restart across peer indexers to take effect.

The --check-restart flag specifically reports whether:

The configuration changes are minor (no restart required).

The changes affect components that require a full or rolling restart (e.g., changes to indexing paths, volume definitions, or replication factors).

Running this validation prior to an actual splunk apply cluster-bundle command prevents service disruption during production operations.

Other commands such as splunk show cluster-bundle-status display deployment status but not restart requirements, and splunk apply cluster-bundle executes the actual deployment, not validation.

References (Splunk Enterprise Documentation):

• Indexer Clustering: Deploy Configuration Bundles with Validation

• splunk validate cluster-bundle Command Reference

• Managing Indexer Clusters – Rolling Restarts and Bundle Deployment Best Practices

• Splunk Enterprise Admin Manual – Cluster Manager Maintenance Commands

 A multi-site indexer cluster can be configured by directly editing SPLUNK_HOME/etc/system/local/server.conf or running a splunk edit cluster-config command from the CLI. These methods allow the administrator to specify the site attribute for each indexer node and the site_replication_factor and site_search_factor for the cluster. Configuring a multi-site indexer cluster via Splunk Web or directly editing SPLUNK_HOME/etc/system/default/server.conf are not supported methods. For more information, see Configure the indexer cluster with server.conf in the Splunk documentation.

 When planning a search head cluster, the following statement is true: All indexers must belong to the underlying indexer cluster (no standalone indexers). A search head cluster is a group of search heads that share configurations, apps, and search jobs. A search head cluster requires an indexer cluster as its data source, meaning that all indexers that provide data to the search head cluster must be members of the same indexer cluster. Standalone indexers, or indexers that are not part of an indexer cluster, cannot be used as data sources for a search head cluster. All search heads do not have to use the same operating system, as long as they are compatible with the Splunk version and the indexer cluster. All search heads do not have to be members of the cluster, as standalone search heads can also search the indexer cluster, but they will not have the benefits of configuration replication and load balancing. The search head captain does not have to be assigned to the largest search head in the cluster, as the captain is dynamically elected from among the cluster members based on various criteria, such as CPU load, network latency, and search load.

The master node distributes configuration bundles to peer nodes in the slave-apps directory under $SPLUNK_HOME/etc. The configuration bundle method is the only supported method for managing common configurations and app deployment across the set of peers. It ensures that all peers use the same versions of these files1. Bundles typically contain a subset of files (configuration files and assets) from $SPLUNK_HOME/etc/system, $SPLUNK_HOME/etc/apps, and $SPLUNK_HOME/etc/users2. The process of distributing knowledge bundles means that peers by default receive nearly the entire contents of the search head’s apps3.

According to Splunk Enterprise administrative documentation, the btool utility is used to troubleshoot configuration settings by merging and displaying effective configurations from all configuration files (system, app, and local levels). When diagnosing license master configuration issues on a search peer or any cluster node, Splunk recommends using the command that specifically lists the license-related stanzas from server.conf.

The correct command is:

splunk cmd btool server list license --debug

This command reveals all configuration parameters under the [license] stanza, including those related to license master connections such as master_uri, pass4SymmKey, and disabled flags. The --debug flag ensures full path tracing of each configuration file, making it easy to identify conflicting or missing parameters that cause communication or validation errors between a search peer and the license master.

Other commands listed, such as btool server list clustering, are meant for diagnosing cluster configuration issues (like replication or search factors), not licensing. The check --debug command only validates syntax and structure, not specific configuration errors tied to licensing. Therefore, the correct and Splunk-documented method for diagnosing license configuration problems on a search peer is to inspect the license stanza using the btool server list license --debug command.

References (Splunk Enterprise Documentation):

• btool Command Reference – Troubleshooting Configuration Issues

• server.conf – License Configuration Parameters

• Managing Distributed License Configurations in Search Peer Clusters

• Troubleshooting License Master and Peer Connectivity

The following statements describe licensing in a clustered Splunk deployment: Free licenses do not support clustering, and replicated data does not count against licensing. Free licenses are limited to 500 MB of daily indexing volume and do not allow distributed searching or clustering. To enable clustering, a license with a higher volume limit and distributed features is required. Replicated data is data that is copied from one peer node to another for the purpose of high availability and load balancing. Replicated data does not count against licensing, because it is not new data that is ingested by Splunk. Only the original data that is indexed by the peer nodes counts against licensing. Each cluster member does not require its own clustering license, because clustering licenses are shared among the cluster members. Cluster members must share the same license pool and license master, because the license master is responsible for distributing licenses to the cluster members and enforcing the license limits

Increasing the search factor in the cluster will best address the requirement of high availability for searchable data. The search factor determines how many copies of searchable data are maintained by the cluster. A higher search factor means that more indexers can serve the data in case of a failure or a maintenance event. Increasing the replication factor will improve the availability of raw data, but not searchable data. Increasing the number of search heads or CPUs on the indexers will improve the search performance, but not the availability of searchable data. For more information, see Replication factor and search factor in the Splunk documentation.

All of the options are possible causes of a crash in Splunk. According to the Splunk documentation1, incorrect ulimit settings can lead to file descriptor exhaustion, which can cause Splunk to crash or hang. Insufficient disk IOPS can also cause Splunk to crash or become unresponsive, as Splunk relies heavily on disk performance2. Insufficient memory can cause Splunk to run out of memory and crash, especially when running complex searches or handling large volumes of data3. Running out of disk space can cause Splunk to stop indexing data and crash, as Splunk needs enough disk space to store its data and logs4.

1: Configure ulimit settings for Splunk Enterprise 2: Troubleshoot Splunk performance issues 3: Troubleshoot memory usage 4: Troubleshoot disk space issues

Searching for closed_txn=0 in this search filters results to situations where Splunk was started, but not stopped. This means that the transaction was not completed, and Splunk crashed before it could finish the pipelines. The closed_txn field is added by the transaction command, and it indicates whether the transaction was closed by an event that matches the endswith condition1. A value of 0 means that the transaction was not closed, and a value of 1 means that the transaction was closed1. Therefore, option D is the correct answer, and options A, B, and C are incorrect.

1: transaction command overview

When installing Enterprise Security on a Search Head Cluster (SHC), the following steps should be done: Install Enterprise Security on the deployer, and use the deployer to deploy Enterprise Security to the cluster members. Enterprise Security is a premium app that provides security analytics and monitoring capabilities for Splunk. Enterprise Security can be installed on a SHC by using the deployer, which is a standalone instance that distributes apps and other configurations to the SHC members. Enterprise Security should be installed on the deployer first, and then deployed to the cluster members using the splunk apply shcluster-bundle command. Enterprise Security should not be installed on a staging instance, because a staging instance is not part of the SHC deployment process. Enterprise Security configurations should not be copied to the deployer, because they are already included in the Enterprise Security app package.

Setting site=site0 on all Search Head Cluster members disables search site affinity. Search site affinity is a feature that allows search heads to preferentially search the peer nodes that are in the same site as the search head, to reduce network latency and bandwidth consumption. By setting site=site0, which is a special value that indicates no site, the search heads will search all peer nodes regardless of their site. Setting site=site0 does not set all members to dynamic captaincy, enable multisite search artifact replication, or enable automatic search site affinity discovery. Dynamic captaincy is a feature that allows any member to become the captain, and it is enabled by default. Multisite search artifact replication is a feature that allows search artifacts to be replicated across sites, and it is enabled by setting site_replication_factor to a value greater than 1. Automatic search site affinity discovery is a feature that allows search heads to automatically determine their site based on the network latency to the peer nodes, and it is enabled by setting site=auto

The frequency in which a deployment client contacts the deployment server is controlled by the phoneHomeIntervalInSecs attribute in deploymentclient.conf. This attribute specifies how often the deployment client checks in with the deployment server to get updates on the apps and configurations that it should receive. The polling_interval attribute in outputs.conf controls how often the forwarder sends data to the indexer or another forwarder. The polling_interval attribute in deploymentclient.conf and the phoneHomeIntervalInSecs attribute in outputs.conf are not valid Splunk attributes. For more information, see Configure deployment clients and Configure forwarders with outputs.conf in the Splunk documentation.

Splunk indexing is read/write intensive, as it involves reading data from various sources, writing data to disk, and reading data from disk for searching and reporting. Therefore, it is important to select the appropriate disk storage solution for each deployment, based on the performance, reliability, and cost requirements. The recommended RAID setup for Splunk indexers is RAID 10 (1 + 0), as it provides the best balance of performance and reliability. RAID 10 combines the advantages of RAID 1 (mirroring) and RAID 0 (striping), which means that it offers both data redundancy and data distribution. RAID 10 can tolerate multiple disk failures, as long as they are not in the same mirrored pair, and it can improve the read and write speed, as it can access multiple disks in parallel2

High performance SAN (Storage Area Network) can be used for Splunk indexers, but it is not recommended, as it is more expensive and complex than local disks. SAN also introduces additional network latency and dependency, which can affect the performance and availability of Splunk indexers. SAN is more suitable for Splunk search heads, as they are less read/write intensive and more CPU intensive2

NFS (Network File System) should not be used for storing hot and warm buckets, as it can cause data corruption, data loss, and performance degradation. NFS is a network-based file system that allows multiple clients to access the same files on a remote server. NFS is not compatible with Splunk index replication and search head clustering, as it can cause conflicts and inconsistencies among the Splunk instances. NFS is also slower and less reliable than local disks, as it depends on the network bandwidth and availability. NFS can be used for storing cold and frozen buckets, as they are less frequently accessed and less critical for Splunk operations2

Virtualized environments are not usually preferred over bare metal for Splunk indexers, as they can introduce additional overhead and complexity. Virtualized environments can affect the performance and reliability of Splunk indexers, as they share the physical resources and the network with other virtual machines. Virtualized environments can also complicate the monitoring and troubleshooting of Splunk indexers, as they add another layer of abstraction and configuration. Virtualized environments can be used for Splunk indexers, but they require careful planning and tuning to ensure optimal performance and availability2

The site_mappings attribute should be added to the master node’s server.conf file when decommissioning a site in an indexer cluster. The site_mappings attribute is used to specify how the master node should reassign the buckets from the decommissioned site to the remaining sites. The site_mappings attribute is a comma-separated list of site pairs, where the first site is the decommissioned site and the second site is the destination site. For example, site_mappings = site1:site2,site3:site4 means that the buckets from site1 will be moved to site2, and the buckets from site3 will be moved to site4. The available_sites attribute is used to specify which sites are currently available in the cluster, and it is automatically updated by the master node. The site_search_factor and site_replication_factor attributes are used to specify the number of searchable and replicated copies of each bucket for each site, and they are not affected by the decommissioning process

The average run time of all searches increases as the number of CPU cores on the indexers decreases. The CPU cores are the processing units that execute the instructions and calculations for the data. The number of CPU cores on the indexers affects the search performance, because the indexers are responsible for retrieving and filtering the data from the indexes. The more CPU cores the indexers have, the faster they can process the data and return the results. The less CPU cores the indexers have, the slower they can process the data and return the results. Therefore, the average run time of all searches is inversely proportional to the number of CPU cores on the indexers. The average run time of all searches is not independent of the number of CPU cores on the indexers, because the CPU cores are an important factor for the search performance. The average run time of all searches does not decrease as the number of CPU cores on the indexers decreases, because this would imply that the search performance improves with less CPU cores, which is not true. The average run time of all searches does not increase as the number of CPU cores on the indexers increases, because this would imply that the search performance worsens with more CPU cores, which is not true

Primary rebalancing automatically occurs when a rolling restart completes, a master node rejoins the cluster, or a peer node joins or rejoins the cluster. These events can cause the distribution of primary buckets to become unbalanced, so the master node will initiate a rebalancing process to ensure that each peer node has roughly the same number of primary buckets. Primary rebalancing does not occur when a captain joins or rejoins the cluster, because the captain is a search head cluster component, not an indexer cluster component. The captain is responsible for search head clustering, not indexer clustering

According to Splunk Enterprise Distributed Clustering documentation, when you add a search head to an indexer cluster, you must configure it to communicate with the Cluster Manager (previously known as Master Node). The proper way to initialize this connection is by editing the cluster configuration using the splunk edit cluster-config command.

The correct syntax for a search head is:

splunk edit cluster-config -mode searchhead -manager_uri https:// :8089 -secret

Here:

-mode searchhead specifies that this node will function as a search head that participates in distributed search across the indexer cluster.

-manager_uri provides the management URI of the cluster manager.

-secret defines the shared secret key used for secure communication between the manager and cluster members.

Once this configuration is applied, the search head must be restarted for the changes to take effect.

Using -mode peer (Option B) is for indexers joining the cluster, not search heads. The add cluster-manager command (Options C and D) is not a valid Splunk command.

References (Splunk Enterprise Documentation):

• Configure the Search Head for an Indexer Cluster

• Indexer Clustering: Configure the Cluster Manager, Peer, and Search Head Nodes

• Splunk Enterprise Admin Manual: splunk edit cluster-config Command Reference

To activate replication for an index in an indexer cluster, the repFactor attribute must be configured in indexes.conf on all peer nodes. This attribute specifies the replication factor for the index, which determines how many copies of raw data are maintained by the cluster. Setting the repFactor attribute to auto will enable replication for the index. The replicate attribute in indexes.conf is not a valid Splunk attribute. The repFactor attribute in outputs.conf and the replicate attribute in deploymentclient.conf are not related to replication for an index in an indexer cluster. For more information, see Configure indexes for indexer clusters in the Splunk documentation.

When building a deployment plan, the architect should perform the following tasks:

Use case checklist. A use case checklist is a document that lists the use cases that the deployment will support, along with the data sources, the data volume, the data retention, the data model, the dashboards, the reports, the alerts, and the roles and permissions for each use case. A use case checklist helps to define the scope and the functionality of the deployment, and to identify the dependencies and the requirements for each use case1

Inventory data sources. An inventory of data sources is a document that lists the data sources that the deployment will ingest, along with the data type, the data format, the data location, the data collection method, the data volume, the data frequency, and the data owner for each data source. An inventory of data sources helps to determine the data ingestion strategy, the data parsing and enrichment, the data storage and retention, and the data security and compliance for the deployment1

Review network topology. A review of network topology is a process that examines the network infrastructure and the network connectivity of the deployment, along with the network bandwidth, the network latency, the network security, and the network monitoring for the deployment. A review of network topology helps to optimize the network performance and reliability, and to identify the network risks and mitigations for the deployment1

Installing Splunk apps is not a task that the architect should perform when building a deployment plan, as it is a task that the administrator should perform when implementing the deployment plan. Installing Splunk apps is a technical activity that requires access to the Splunk instances and the Splunk configurations, which are not available at the planning stage

The telnet and tcpdump tools can be leveraged to diagnose connection problems between an indexer and forwarder. The telnet tool can be used to test the connectivity and port availability between the indexer and forwarder. The tcpdump tool can be used to capture and analyze the network traffic between the indexer and forwarder. The splunk btool command can be used to check the configuration files of the indexer and forwarder, but it cannot diagnose the connection problems. The splunk btprobe command does not exist, and it is not a valid tool.

When converting from a single-site to a multi-site cluster, existing single-site clustered buckets will maintain replication as required according to the single-site policies, but never age out. Single-site clustered buckets are buckets that were created before the conversion to a multi-site cluster. These buckets will continue to follow the single-site replication and search factors, meaning that they will have the same number of copies and searchable copies across the cluster, regardless of the site. These buckets will never age out, meaning that they will never be frozen or deleted, unless they are manually converted to multi-site buckets. Single-site clustered buckets will not continue to replicate within the origin site, because they will be distributed across the cluster according to the single-site policies. Single-site clustered buckets will not be replicated across all peers in the multi-site cluster, because they will follow the single-site replication factor, which may be lower than the multi-site total replication factor. Single-site clustered buckets will not stop replicating within the single-site and remain on the indexer they reside on, because they will still be subject to the replication and availability rules of the cluster

The Splunk server name of the member can typically be determined by the serverName attribute in the server.conf file, which is not explicitly shown in the provided snippet. However, based on the provided configuration snippet, we can infer that this search head cluster member is configured to communicate with a cluster master (master_uri) located at node1 and a management node (mgmt_uri) located at node3. The serverName is not the same as the master_uri or mgmt_uri; these URIs indicate the location of the master and management nodes that this member interacts with.

Since the serverName is not provided in the snippet, one would typically look for a setting under the [general] stanza in server.conf. However, given the options and the common naming conventions in a Splunk environment, node3 would be a reasonable guess for the server name of this member, since it is indicated as the management URI within the [shclustering] stanza, which suggests it might be the name or address of the server in question.

For accurate identification, you would need to access the full server.conf file or the Splunk Web on the search head cluster member and look under Settings > Server settings > General settings to find the actual serverName. Reference for these details would be found in the Splunk documentation regarding the configuration files, particularly server.conf.

According to the Indexer Clustering Administration Guide, configuration bundles pushed from the Cluster Manager (Master Node) overwrite the contents of the $SPLUNK_HOME/etc/slave-apps/ directory on each search peer (indexer). However, Splunk provides a special persistent location — the _cluster app’s local directory — for files that must survive bundle redeployments.

Specifically, any configuration files placed in:

$SPLUNK_HOME/etc/slave-apps/_cluster/local/

will persist after future bundle pushes because this directory is excluded from the automatic overwrite process.

This is particularly useful for maintaining local overrides or custom configurations that should not be replaced by the Cluster Manager, such as environment-specific inputs, temporary test settings, or monitoring configurations unique to that peer.

Other directories under slave-apps are overwritten each time a configuration bundle is pushed, ensuring consistency across the cluster. Likewise, master-apps exists only on the Cluster Manager and is used for deployment, not persistence.

Thus, the _cluster/local folder is the only safe, Splunk-documented location for configurations that need to survive bundle redeployment.

References (Splunk Enterprise Documentation):

• Indexer Clustering: How Configuration Bundles Work

• Maintaining Local Configurations on Clustered Indexers

• slave-apps and _cluster App Structure and Behavior

• Splunk Enterprise Admin Manual – Cluster Configuration Management Best Practices

 The following statements are true regarding Splunk Enterprise performance:

Adding search peers increases the search throughput as search load increases. This is because adding more search peers distributes the search workload across more indexers, which reduces the load on each indexer and improves the search speed and concurrency.

Adding search heads provides additional CPU cores to run more concurrent searches. This is because adding more search heads increases the number of search processes that can run in parallel, which improves the search performance and scalability. The following statements are false regarding Splunk Enterprise performance:

Adding search peers does not increase the maximum size of search results. The maximum size of search results is determined by the maxresultrows setting in the limits.conf file, which is independent of the number of search peers.

Adding RAM to an existing search head does not provide additional search capacity. The search capacity of a search head is determined by the number of CPU cores, not the amount of RAM. Adding RAM to a search head may improve the search performance, but not the search capacity. For more information, see Splunk Enterprise performance in the Splunk documentation.

Splunk Enterprise allows administrators to dynamically increase logging verbosity for a specific search by adding a | noop log_debug=* command immediately after the base search. This method provides temporary, search-specific debug logging without requiring global configuration changes or restarts.

The noop (no operation) command passes all results through unchanged but can trigger internal logging actions. When paired with the log_debug=* argument, it instructs Splunk to record detailed debug-level log messages for that specific search execution in search.log and the relevant internal logs.

This approach is officially documented for troubleshooting complex search issues such as:

Unexpected search behavior or slow performance.

Field extraction or command evaluation errors.

Debugging custom search commands or macros.

Using this method is safer and more efficient than modifying server-wide logging configurations (server.conf or limits.conf), which can affect all users and increase log noise. The “Server logging” page in Splunk Web (Option D) adjusts global logging levels, not per-search debugging.

References (Splunk Enterprise Documentation):

• Search Debugging Techniques and the noop Command

• Understanding search.log and Per-Search Logging Control

• Splunk Search Job Inspector and Debugging Workflow

• Troubleshooting SPL Performance and Field Extraction Issues

According to the Splunk documentation1, in a search head cluster, the KV Store Primary is the same node as the search head cluster captain. The KV Store Primary is responsible for coordinating the replication of KV Store data across the cluster members. When any node receives a write request, the KV Store delegates the write to the KV Store Primary. The KV Store keeps the reads local, however. This ensures that the KV Store data is consistent and available across the cluster.

 In a distributed environment, knowledge object bundles are replicated from the search head to the SPLUNK_HOME/var/run/searchpeers directory on the search peer(s). A knowledge object bundle is a compressed file that contains the knowledge objects, such as fields, lookups, macros, and tags, that are required for a search. A search peer is a Splunk instance that provides data to a search head in a distributed search. A search head is a Splunk instance that coordinates and executes a search across multiple search peers. When a search head initiates a search, it creates a knowledge object bundle and replicates it to the search peers that are involved in the search. The search peers store the knowledge object bundle in the SPLUNK_HOME/var/run/searchpeers directory, which is a temporary directory that is cleared when the Splunk service restarts. The search peers use the knowledge object bundle to apply the knowledge objects to the data and return the results to the search head. The SPLUNK_HOME/var/lib/searchpeers, SPLUNK_HOME/var/log/searchpeers, and SPLUNK_HOME/var/spool/searchpeers directories are not the locations where the knowledge object bundles are replicated, because they do not exist in the Splunk file system

Comprehensive and Detailed Explanation (From Splunk Enterprise Documentation)Splunk’s multisite clustering documentation describes that search affinity is controlled by the site attribute in server.conf on the search head. Splunk explicitly states that assigning site=site0 on a search head removes site affinity, causing the search head to treat all sites as equal and search remotely as needed. The documentation describes site0 as the special value that disables local-site preference and forces the system to behave like a single-site cluster.

The customer wants each site’s search head to pull results only from its local site. This behavior works only if the search head’s site value matches the local site name (e.g., site1 or site2). By setting it to site0, all locality restrictions are removed, which prevents the desired reduction of network traffic.

The site search factor options (B and D) affect replication and searchable copy placement on indexers, not search head behavior. The number of indexers per site (C) also does not disable search affinity. Therefore only option A disables local-only searching.