SPLK-2002 Splunk Enterprise Certified Architect Question and Answers

 In a distributed environment, knowledge object bundles are replicated from the search head to the SPLUNK_HOME/var/run/searchpeers directory on the search peer(s). A knowledge object bundle is a compressed file that contains the knowledge objects, such as fields, lookups, macros, and tags, that are required for a search. A search peer is a Splunk instance that provides data to a search head in a distributed search. A search head is a Splunk instance that coordinates and executes a search across multiple search peers. When a search head initiates a search, it creates a knowledge object bundle and replicates it to the search peers that are involved in the search. The search peers store the knowledge object bundle in the SPLUNK_HOME/var/run/searchpeers directory, which is a temporary directory that is cleared when the Splunk service restarts. The search peers use the knowledge object bundle to apply the knowledge objects to the data and return the results to the search head. The SPLUNK_HOME/var/lib/searchpeers, SPLUNK_HOME/var/log/searchpeers, and SPLUNK_HOME/var/spool/searchpeers directories are not the locations where the knowledge object bundles are replicated, because they do not exist in the Splunk file system

 Changing the limits.conf value for max_searches_per_cpu to a higher value is the best option to increase scheduled search capacity on the search head cluster when a large number of searches are skipped across time. This value determines how many concurrent scheduled searches can run on each CPU core of the search head. Increasing this value will allow more scheduled searches to run at the same time, which will reduce the number of skipped searches. Creating a job server on the cluster, running the server.conf captain_is_adhoc_searchhead = true command, or adding another search head to the cluster are not the best options to increase scheduled search capacity on the search head cluster. For more information, see [Configure limits.conf] in the Splunk documentation.

According to the Splunk Search Job Inspector documentation, the search.log file for a completed search job can be accessed through Splunk Web by navigating to the job’s detailed inspection view.

To access it:

Open the completed search in Splunk Web.

Click the Job menu (top right of the search interface).

Select Inspect Job.

In the Job Inspector window, click the search.log link.

This log provides in-depth diagnostic details about how the search was parsed, distributed, and executed across the search head and indexers. It contains valuable performance metrics, command execution order, event sampling information, and any error or warning messages encountered during search processing.

The search.log file is generated for every search job—scheduled, ad-hoc, or background—and is stored in the job’s dispatch directory ($SPLUNK_HOME/var/run/splunk/dispatch//).

Other listed options are incorrect:

Option A queries the _internal index, which does not store per-search logs.

Option B is used to view search configurations, not logs.

Option C is not a valid Splunk Web navigation option.

Thus, the only correct and Splunk-documented method is via Job → Inspect Job → search.log.

References (Splunk Enterprise Documentation):

• Search Job Inspector Overview and Usage

• Analyzing Search Performance Using search.log

• Search Job Management and Dispatch Directory Structure

• Splunk Enterprise Admin Manual – Troubleshooting Searches

Data retention is optional in the data source assessment because it is not directly related to the ingestion process. Data retention is determined by the index configuration and the storage capacity of the Splunk platform. Data format, data location, and data volume are all essential information for planning how to collect, parse, and index the data source.

The following may explain the problem of why a colleague cannot see the src_ip field in their search results: The field was extracted as a private knowledge object, and the colleague did not explicitly use the field in the search and the search was set to Fast Mode. A knowledge object is a Splunk entity that applies some knowledge or intelligence to the data, such as a field extraction, a lookup, or a macro. A knowledge object can have different permissions, such as private, app, or global. A private knowledge object is only visible to the user who created it, and it cannot be shared with other users. A field extraction is a type of knowledge object that extracts fields from the raw data at index time or search time. If a field extraction is created as a private knowledge object, then only the user who created it can see the extracted field in their search results. A search mode is a setting that determines how Splunk processes and displays the search results, such as Fast, Smart, or Verbose. Fast mode is the fastest and most efficient search mode, but it also limits the number of fields and events that are displayed. Fast mode only shows the default fields, such as _time, host, source, sourcetype, and _raw, and any fields that are explicitly used in the search. If a field is not used in the search and it is not a default field, then it will not be shown in Fast mode. The events are tagged as communicate, but are missing the network tag, and the Typing Queue, which does regular expression replacements, is blocked, are not valid explanations for the problem. Tags are labels that can be applied to fields or field values to make them easier to search. Tags do not affect the visibility of fields, unless they are used as filters in the search. The Typing Queue is a component of the Splunk data pipeline that performs regular expression replacements on the data, such as replacing IP addresses with host names. The Typing Queue does not affect the field extraction process, unless it is configured to do so

The curl https://serverhost:8089/services/admin/inputstatus/TailingProcessor:FileStatus command is used to check the status of the tailed files when troubleshooting monitor inputs. Monitor inputs are inputs that monitor files or directories for new data and send the data to Splunk for indexing. The TailingProcessor:FileStatus endpoint returns information about the files that are being monitored by the Tailing Processor, such as the file name, path, size, position, and status. The splunk cmd btool inputs list | tail command is used to list the inputs configurations from the inputs.conf file and pipe the output to the tail command. The splunk cmd btool check inputs layer command is used to check the inputs configurations for syntax errors and layering. The curl https://serverhost:8089/services/admin/inputstatus/TailingProcessor:Tailstatus command does not exist, and it is not a valid endpoint.

A deployment plan should include business continuity and disaster recovery plans, current logging details and data source inventory, and current and future topology diagrams of the IT environment. These elements are essential for planning, designing, and implementing a Splunk deployment that meets the business and technical requirements. A comprehensive list of stakeholders, either direct or indirect, is not part of the deployment plan, but rather part of the project charter. For more information, see Deployment planning in the Splunk documentation.

 Configurations from the deployer are merged into the SPLUNK_HOME/etc/apps/APP_HOME/local directory on the search head cluster member. The deployer distributes apps and other configurations to the search head cluster members in the form of a configuration bundle. The configuration bundle contains the contents of the SPLUNK_HOME/etc/shcluster/apps directory on the deployer. When a search head cluster member receives the configuration bundle, it merges the contents of the bundle into its own SPLUNK_HOME/etc/apps directory. The configurations in the local directory take precedence over the configurations in the default directory. The SPLUNK_HOME/etc/system/local directory is used for system-level configurations, not app-level configurations. The SPLUNK_HOME/etc/apps/search/default directory is used for the default configurations of the search app, not the configurations from the deployer.

The frequency in which a deployment client contacts the deployment server is controlled by the phoneHomeIntervalInSecs attribute in deploymentclient.conf. This attribute specifies how often the deployment client checks in with the deployment server to get updates on the apps and configurations that it should receive. The polling_interval attribute in outputs.conf controls how often the forwarder sends data to the indexer or another forwarder. The polling_interval attribute in deploymentclient.conf and the phoneHomeIntervalInSecs attribute in outputs.conf are not valid Splunk attributes. For more information, see Configure deployment clients and Configure forwarders with outputs.conf in the Splunk documentation.

According to Splunk Enterprise Search Performance and Deployment Optimization guidelines, the phone home interval (configured for deployment clients communicating with a Deployment Server) has no impact on search performance.

The phone home mechanism controls how often deployment clients check in with the Deployment Server for configuration updates or new app bundles. This process occurs independently of the search subsystem and does not consume indexer or search head resources that affect query speed, indexing throughput, or search concurrency.

In contrast:

Increasing the number of indexers (Option B) improves search performance by distributing indexing and search workloads across more nodes.

Workload Management (Option C) allows admins to prioritize compute and memory resources for critical searches, optimizing performance under load.

Increasing search heads (Option D) can enhance concurrency and user responsiveness by distributing search scheduling and ad-hoc query workloads.

Therefore, adjusting the phone home interval is strictly an administrative operation and has no measurable effect on Splunk search or indexing performance.

References (Splunk Enterprise Documentation):

• Deployment Server: Managing Phone Home Intervals

• Search Performance Optimization and Resource Management

• Distributed Search Architecture and Scaling Best Practices

• Workload Management Overview – Resource Allocation in Search Operations

This is where in the Job Inspector details can be found to help determine where performance is affected, as it shows the time and resources spent by each component of the search, such as commands, subsearches, lookups, and post-processing1. The Execution Costs > Components section can help identify the most expensive or inefficient parts of the search, and suggest ways to optimize or improve the search performance1. The other options are not as useful as the Execution Costs > Components section for finding performance issues. Option A, Search Job Properties > runDuration, shows the total time, in seconds, that the search took to run2. This can indicate the overall performance of the search, but it does not provide any details on the specific components or factors that affected the performance. Option B, Search Job Properties > runtime, shows the time, in seconds, that the search took to run on the search head2. This can indicate the performance of the search head, but it does not account for the time spent on the indexers or the network. Option C, Job Details Dashboard > Total Events Matched, shows the number of events that matched the search criteria3. This can indicate the size and scope of the search, but it does not provide any information on the performance or efficiency of the search. Therefore, option D is the correct answer, and options A, B, and C are incorrect.

1: Execution Costs > Components 2: Search Job Properties 3: Job Details Dashboard

The correct answer is C. Knowledge Object replication. This is a valid use case that a search head cluster addresses, as it ensures that all the search heads in the cluster have the same set of knowledge objects, such as saved searches, dashboards, reports, and alerts1. The search head cluster replicates the knowledge objects across the cluster members, and synchronizes any changes or updates1. This provides a consistent user experience and avoids data inconsistency or duplication1. The other options are not valid use cases that a search head cluster addresses. Option A, providing redundancy in the event a search peer fails, is not a use case for a search head cluster, but for an indexer cluster, which maintains multiple copies of the indexed data and can recover from indexer failures2. Option B, search affinity, is not a use case for a search head cluster, but for a multisite indexer cluster, which allows the search heads to preferentially search the data on the local site, rather than on a remote site3. Option D, increased Search Factor (SF), is not a use case for a search head cluster, but for an indexer cluster, which determines how many searchable copies of each bucket are maintained across the indexers4. Therefore, option C is the correct answer, and options A, B, and D are incorrect.

1: About search head clusters 2: About indexer clusters and index replication 3: Configure search affinity 4: Configure the search factor

The deployer is a Splunk Enterprise instance that distributes apps and other configurations to the members of a search head cluster1.

The deployer cannot share functionality with any other Splunk Enterprise instance, including the license master, the master node, or the monitoring console2.

However, the search head cluster members can share functionality with the master node and the monitoring console, as long as they are not designated as the captain of the cluster3.

Therefore, the correct answer is B. License master, as it is the only instance that cannot share functionality with the deployer under any circumstances.

The KV store forms its own cluster within a SHC. The maximum number of SHC members KV store will form is 50. The KV store cluster is a subset of the SHC members that are responsible for replicating and storing the KV store data. The KV store cluster can have up to 50 members, but only 20 of them can be active at any given time. The other members are standby members that can take over if an active member fails. The KV store cluster cannot have more than 50 members, nor can it have an unlimited number of members. The KV store cluster cannot have 25 or 100 members, because these numbers are not multiples of 5, which is the minimum replication factor for the KV store cluster

Before deploying Splunk, it is important to gather some information about the current environment, such as:

Overall goals for the deployment: This includes the business objectives, the use cases, the expected outcomes, and the success criteria for the Splunk deployment. This information helps to define the scope, the requirements, the design, and the validation of the Splunk solution1.

Key users: This includes the roles, the responsibilities, the expectations, and the needs of the different types of users who will interact with the Splunk deployment, such as administrators, analysts, developers, and end users. This information helps to determine the user access, the user experience, the user training, and the user feedback for the Splunk solution1.

Data sources: This includes the types, the formats, the volumes, the locations, and the characteristics of the data that will be ingested, indexed, and searched by the Splunk deployment. This information helps to estimate the data throughput, the data retention, the data quality, and the data analysis for the Splunk solution1.

Option B, C, and D are the correct answers because they reflect the essential information that is needed before deploying Splunk. Option A is incorrect because the list of vendors for network devices is not a relevant information for the Splunk deployment. The network devices may be part of the data sources, but the vendors are not important for the Splunk solution.

The following clarification steps should be taken if apps are not appearing on a deployment client:

Check serverclass.conf of the deployment server. This file defines the server classes and the apps and configurations that they should receive from the deployment server. Make sure that the deployment client belongs to the correct server class and that the server class has the desired apps and configurations.

Check deploymentclient.conf of the deployment client. This file specifies the deployment server that the deployment client contacts and the client name that it uses. Make sure that the deployment client is pointing to the correct deployment server and that the client name matches the server class criteria.

Search for relevant events in splunkd.log of the deployment server. This file contains information about the deployment server activities, such as sending apps and configurations to the deployment clients, detecting client check-ins, and logging any errors or warnings. Look for any events that indicate a problem with the deployment server or the deployment client.

Checking the content of SPLUNK_HOME/etc/apps of the deployment server is not a necessary clarification step, as this directory does not contain the apps and configurations that are distributed to the deployment clients. The apps and configurations for the deployment server are stored in SPLUNK_HOME/etc/deployment-apps. For more information, see Configure deployment server and clients in the Splunk documentation.

This is the fall-back method for Splunk if .delta replication fails during knowledge bundle replication. Knowledge bundle replication is the process of distributing the knowledge objects, such as lookups, macros, and field extractions, from the search head cluster to the indexer cluster1. Splunk uses two methods of knowledge bundle replication: .delta replication and .bundle replication1. .Delta replication is the default and preferred method, as it only replicates the changes or updates to the knowledge objects, which reduces the network traffic and disk space usage1. However, if .delta replication fails for some reason, such as corrupted files or network errors, Splunk automatically switches to .bundle replication, which replicates the entire knowledge bundle, regardless of the changes or updates1. This ensures that the knowledge objects are always synchronized between the search head cluster and the indexer cluster, but it also consumes more network bandwidth and disk space1. The other options are not valid fall-back methods for Splunk. Option A, restarting splunkd, is not a method of knowledge bundle replication, but a way to restart the Splunk daemon on a node2. This may or may not fix the .delta replication failure, but it does not guarantee the synchronization of the knowledge objects. Option B, .delta replication, is not a fall-back method, but the primary method of knowledge bundle replication, which is assumed to have failed in the question1. Option D, restarting mongod, is not a method of knowledge bundle replication, but a way to restart the MongoDB daemon on a node3. This is not related to the knowledge bundle replication, but to the KV store replication, which is a different process3. Therefore, option C is the correct answer, and options A, B, and D are incorrect.

1: How knowledge bundle replication works 2: Start and stop Splunk Enterprise 3: Restart the KV store

As a best practice, the internal licensing logs should be stored on the license server. The license server is a Splunk instance that manages the distribution and enforcement of licenses in a Splunk deployment. The license server generates internal licensing logs that contain information about the license usage, violations, warnings, and pools. The internal licensing logs should be stored on the license server itself, because they are relevant to the license server’s role and function. Storing the internal licensing logs on the license server also simplifies the license monitoring and troubleshooting process. The internal licensing logs should not be stored on the indexing layer, the deployment layer, or the search head layer, because they are not related to the roles and functions of these layers. Storing the internal licensing logs on these layers would also increase the network traffic and disk space consumption

When converting from a single-site to a multi-site cluster, existing single-site clustered buckets will maintain replication as required according to the single-site policies, but never age out. Single-site clustered buckets are buckets that were created before the conversion to a multi-site cluster. These buckets will continue to follow the single-site replication and search factors, meaning that they will have the same number of copies and searchable copies across the cluster, regardless of the site. These buckets will never age out, meaning that they will never be frozen or deleted, unless they are manually converted to multi-site buckets. Single-site clustered buckets will not continue to replicate within the origin site, because they will be distributed across the cluster according to the single-site policies. Single-site clustered buckets will not be replicated across all peers in the multi-site cluster, because they will follow the single-site replication factor, which may be lower than the multi-site total replication factor. Single-site clustered buckets will not stop replicating within the single-site and remain on the indexer they reside on, because they will still be subject to the replication and availability rules of the cluster

 When adding or rejoining a member to a search head cluster, and the following error is displayed: Error pulling configurations from the search head cluster captain; consider performing a destructive configuration resync on this search head cluster member.

The corrective action that should be taken is to run the splunk resync shcluster-replicated-config command on this member. This command will delete the existing configuration files on this member and replace them with the latest configuration files from the captain. This will ensure that the member has the same configuration as the rest of the cluster. Restarting the search head, running the splunk apply shcluster-bundle command from the deployer, or running the clean raft command on all members of the search head cluster are not the correct actions to take in this scenario. For more information, see Resolve configuration inconsistencies across cluster members in the Splunk documentation.

According to the Splunk documentation1, in a search head cluster, the KV Store Primary is the same node as the search head cluster captain. The KV Store Primary is responsible for coordinating the replication of KV Store data across the cluster members. When any node receives a write request, the KV Store delegates the write to the KV Store Primary. The KV Store keeps the reads local, however. This ensures that the KV Store data is consistent and available across the cluster.

A lookup table is a file that contains a list of values that can be used to enrich or modify the data during search time1. Lookup tables can be stored in CSV files or in the KV Store1. Troubleshooting lookup tables involves identifying and resolving issues that prevent the lookup tables from being accessed, updated, or applied correctly by the Splunk searches. Some of the tools and methods that can help with troubleshooting lookup tables are:

web_access.log: This is a file that contains information about the HTTP requests and responses that occur between the Splunk web server and the clients2. This file can help troubleshoot issues related to lookup table permissions, availability, and errors, such as 404 Not Found, 403 Forbidden, or 500 Internal Server Error34.

btool output: This is a command-line tool that displays the effective configuration settings for a given Splunk component, such as inputs, outputs, indexes, props, and so on5. This tool can help troubleshoot issues related to lookup table definitions, locations, and precedence, as well as identify the source of a configuration setting6.

search.log: This is a file that contains detailed information about the execution of a search, such as the search pipeline, the search commands, the search results, the search errors, and the search performance. This file can help troubleshoot issues related to lookup table commands, arguments, fields, and outputs, such as lookup, inputlookup, outputlookup, lookup_editor, and so on .

Option B is the correct answer because web_access.log is the best place to start troubleshooting lookup table issues, as it can provide the most relevant and immediate information about the lookup table access and status. Option A is incorrect because btool output is not a log file, but a command-line tool. Option C is incorrect because health.log is a file that contains information about the health of the Splunk components, such as the indexer cluster, the search head cluster, the license master, and the deployment server. This file can help troubleshoot issues related to Splunk deployment health, but not necessarily related to lookup tables. Option D is incorrect because configuration_change.log is a file that contains information about the changes made to the Splunk configuration files, such as the user, the time, the file, and the action. This file can help troubleshoot issues related to Splunk configuration changes, but not necessarily related to lookup tables.

The Monitoring Console (MC) is the Splunk Enterprise monitoring tool that lets you view detailed topology and performance information about your Splunk Enterprise deployment1. The MC can be installed on any Splunk Enterprise instance that can access the data from all the instances in the deployment2. However, following the Splunk recommendations, the MC should be installed on the search head cluster deployer, which is a dedicated instance that manages the configuration bundle for the search head cluster members3. This way, the MC can monitor the search head cluster as well as the indexer cluster and the forwarders, without affecting the performance or availability of the other instances4. The other options are not recommended because they either introduce additional load on the existing instances (such as A and D) or do not have access to the data from the search head cluster (such as B).

1: About the Monitoring Console - Splunk Documentation 2: Add Splunk Enterprise instances to the Monitoring Console 3: Configure the deployer - Splunk Documentation 4: [Monitoring Console setup and use - Splunk Documentation]

A search head cluster is a group of Splunk Enterprise search heads that share configurations, job scheduling, and search artifacts1. The deployer is a Splunk Enterprise instance that distributes apps and other configurations to the cluster members1. The local folder of each Splunk app contains the custom configurations that override the default settings2. The default folder of each Splunk app contains the default configurations that are provided by the app2.

By default, when the deployer pushes an app to the search head cluster, it merges the local folder of the app into the default folder and deploys the merged folder to the search heads3. This means that the custom configurations in the local folder will take precedence over the default settings in the default folder. However, this also means that the local folder of the app on the search heads will be empty, unless the app is modified through the search head UI3.

Option B is the correct answer because it reflects the default behavior of the deployer when pushing apps to the search head cluster. Option A is incorrect because the local folder is not copied to the local folder on the search heads, but merged into the default folder. Option C is incorrect because all the .conf files in the local folder are deployed to the search heads, not only certain ones. Option D is incorrect because the local folder is not ignored, but merged into the default folder.

According to the Splunk documentation1, the CHARSET setting in props.conf specifies the character set encoding of the source data. This setting has the least impact on indexing performance, as it only affects how Splunk interprets the bytes of the data, not how it processes or transforms the data. The other options are false because:

The SHOULD_LINEMERGE setting in props.conf determines whether Splunk breaks events based on timestamps or newlines. This setting has a significant impact on indexing performance, as it affects how Splunk parses the data and identifies the boundaries of the events2.

The TRUNCATE setting in props.conf specifies the maximum number of characters that Splunk indexes from a single line of a file. This setting has a moderate impact on indexing performance, as it affects how much data Splunk reads and writes to the index3.

The TIME_PREFIX setting in props.conf specifies the prefix that directly precedes the timestamp in the event data. This setting has a moderate impact on indexing performance, as it affects how Splunk extracts the timestamp and assigns it to the event

According to Splunk Enterprise administrative documentation, the btool utility is used to troubleshoot configuration settings by merging and displaying effective configurations from all configuration files (system, app, and local levels). When diagnosing license master configuration issues on a search peer or any cluster node, Splunk recommends using the command that specifically lists the license-related stanzas from server.conf.

The correct command is:

splunk cmd btool server list license --debug

This command reveals all configuration parameters under the [license] stanza, including those related to license master connections such as master_uri, pass4SymmKey, and disabled flags. The --debug flag ensures full path tracing of each configuration file, making it easy to identify conflicting or missing parameters that cause communication or validation errors between a search peer and the license master.

Other commands listed, such as btool server list clustering, are meant for diagnosing cluster configuration issues (like replication or search factors), not licensing. The check --debug command only validates syntax and structure, not specific configuration errors tied to licensing. Therefore, the correct and Splunk-documented method for diagnosing license configuration problems on a search peer is to inspect the license stanza using the btool server list license --debug command.

References (Splunk Enterprise Documentation):

• btool Command Reference – Troubleshooting Configuration Issues

• server.conf – License Configuration Parameters

• Managing Distributed License Configurations in Search Peer Clusters

• Troubleshooting License Master and Peer Connectivity

Splunk Enterprise allows administrators to dynamically increase logging verbosity for a specific search by adding a | noop log_debug=* command immediately after the base search. This method provides temporary, search-specific debug logging without requiring global configuration changes or restarts.

The noop (no operation) command passes all results through unchanged but can trigger internal logging actions. When paired with the log_debug=* argument, it instructs Splunk to record detailed debug-level log messages for that specific search execution in search.log and the relevant internal logs.

This approach is officially documented for troubleshooting complex search issues such as:

Unexpected search behavior or slow performance.

Field extraction or command evaluation errors.

Debugging custom search commands or macros.

Using this method is safer and more efficient than modifying server-wide logging configurations (server.conf or limits.conf), which can affect all users and increase log noise. The “Server logging” page in Splunk Web (Option D) adjusts global logging levels, not per-search debugging.

References (Splunk Enterprise Documentation):

• Search Debugging Techniques and the noop Command

• Understanding search.log and Per-Search Logging Control

• Splunk Search Job Inspector and Debugging Workflow

• Troubleshooting SPL Performance and Field Extraction Issues

Splunk Enterprise documentation clearly states that the best method to secure log traffic between Universal Forwarders (UFs) and Indexers is to implement Transport Layer Security (TLS) using signed SSL certificates. When Universal Forwarders send data to Indexers, this communication can be encrypted using SSL/TLS to prevent eavesdropping, data tampering, or interception while in transit.

Splunk provides default self-signed certificates out of the box, but these are only for testing or lab environments and should not be used in production. Production-grade security requires custom, signed SSL certificates — either from an internal Certificate Authority (CA) or a trusted public CA. These certificates validate both the sender (forwarder) and receiver (indexer), ensuring data integrity and authenticity.

In practice, this involves:

Generating or obtaining CA-signed certificates.

Configuring the forwarder’s outputs.conf to use SSL encryption (sslCertPath, sslPassword, and sslRootCAPath).

Configuring the indexer’s inputs.conf and server.conf to require and validate client certificates.

This configuration ensures end-to-end encryption for all log data transmitted from forwarders to indexers.

Routing traffic through a WAF (Option C) does not provide end-to-end encryption for Splunk’s internal communication, and securing search head–to–indexer communication (Option D) is unrelated to forwarder data flow.

References (Splunk Enterprise Documentation):

• Securing Splunk Enterprise: Encrypting Data in Transit Using SSL/TLS

• Configure Forwarder-to-Indexer Encryption

• Server and Forwarder Authentication with Signed Certificates

• Best Practices for Forwarder Management and Security Configuration

According to Splunk’s Deployment Planning and Implementation Guidelines, one of the most critical elements of a Splunk deployment plan is a comprehensive data source inventory and current logging details. This information defines the scope of data ingestion and directly influences sizing, architecture design, and licensing.

A proper deployment plan should identify:

All data sources (such as syslogs, application logs, network devices, OS logs, databases, etc.)

Expected daily ingest volume per source

Log formats and sourcetypes

Retention requirements and compliance constraints

This data forms the foundation for index sizing, forwarder configuration, and storage planning. Without a well-defined data inventory, Splunk architects cannot accurately determine hardware capacity, indexing load, or network throughput requirements.

While stakeholder mapping, topology diagrams, and continuity plans (Options A, B, D) are valuable in a broader IT project, Splunk’s official guidance emphasizes logging details and source inventory as mandatory for a deployment plan. It ensures that the Splunk environment is properly sized, licensed, and aligned with business data visibility goals.

References (Splunk Enterprise Documentation):

• Splunk Enterprise Deployment Planning Manual – Data Source Inventory Requirements

• Capacity Planning for Indexer and Search Head Sizing

• Planning Data Onboarding and Ingestion Strategies

• Splunk Architecture and Implementation Best Practices

Splunk’s KV Store management documentation specifies that when converting a static CSV lookup to a KV Store lookup, the lookup data is stored in a MongoDB-based collection defined in collections.conf. To ensure that the KV Store lookup is replicated and available across all search head cluster members, administrators must include the attribute replicate=true within the collections.conf file.

This configuration instructs Splunk to replicate the KV Store collection’s data to all members in the Search Head Cluster (SHC), enabling consistent access and reliability across the cluster. Without this attribute, the KV Store collection would remain local to a single search head, making it unavailable for automatic lookups performed by other members.

Here’s an example configuration snippet from collections.conf:

[customer_lookup]

replicate = true

field.name = string

field.age = number

The attribute repFactor=true (mentioned in Options A and D) is unrelated to KV Store behavior—it applies to index replication, not KV Store replication. Similarly, replicate=true in lookups.conf (Option B) has no effect, as KV Store replication is controlled exclusively via collections.conf.

Once properly configured, the lookup can be defined in transforms.conf and referenced in props.conf for automatic lookup functionality.

References (Splunk Enterprise Documentation):

• KV Store Collections and Configuration – collections.conf Reference

• Managing KV Store Data in Search Head Clusters

• Automatic Lookup Configuration Using KV Store

• Splunk Enterprise Admin Manual – Distributed KV Store Replication Settings

The Splunk deployer sends apps to the search head cluster members by default to the path etc/shcluster//default. The deployer is a Splunk component that distributes apps and configurations to members of a search head cluster.

Splunk's documentation recommends placing the configuration bundle in the $SPLUNK_HOME/etc/shcluster/apps directory on the deployer, which then gets distributed to the search head cluster members. However, it should be noted that within each app's directory, configurations can be under default or local subdirectories, with local taking precedence over default for configurations. The reference to etc/shcluster//default is not a standard directory structure and might be a misunderstanding. The correct path where the deployer pushes configuration bundles is $SPLUNK_HOME/etc/shcluster/apps

 When planning a search head cluster, the following statement is true: All indexers must belong to the underlying indexer cluster (no standalone indexers). A search head cluster is a group of search heads that share configurations, apps, and search jobs. A search head cluster requires an indexer cluster as its data source, meaning that all indexers that provide data to the search head cluster must be members of the same indexer cluster. Standalone indexers, or indexers that are not part of an indexer cluster, cannot be used as data sources for a search head cluster. All search heads do not have to use the same operating system, as long as they are compatible with the Splunk version and the indexer cluster. All search heads do not have to be members of the cluster, as standalone search heads can also search the indexer cluster, but they will not have the benefits of configuration replication and load balancing. The search head captain does not have to be assigned to the largest search head in the cluster, as the captain is dynamically elected from among the cluster members based on various criteria, such as CPU load, network latency, and search load.

The Splunk Capacity Planning Manual defines the total required storage for indexes as a function of daily ingest rate, compression factor, retention period, and an additional padding buffer for index management and growth.

The formula is:

Storage = (Daily Data * Compression Factor * Retention Days) + Padding

Given the values:

Daily rate = 20 GB

Compression factor = 0.5 (50% reduction)

Retention period = 30 days

Padding = 100 GB

Plugging these into the formula gives:

20 * 0.5 * 30 + 100 = 400 GB

This result represents the estimated storage needed to retain 30 days of compressed indexed data with an additional buffer to accommodate growth and Splunk’s bucket management overhead.

Compression factor values typically range between 0.5 and 0.7 for most environments, depending on data type. Using compression in calculations is critical, as indexed data consumes less space than raw input after Splunk’s tokenization and compression processes.

Other options either misapply the compression ratio or the order of operations, producing incorrect totals.

References (Splunk Enterprise Documentation):

• Capacity Planning for Indexes – Storage Sizing and Compression Guidelines

• Managing Index Storage and Retention Policies

• Splunk Enterprise Admin Manual – Understanding Index Bucket Sizes

• Indexing Performance and Storage Optimization Guide

The TailingProcessor channel in the splunkd.log file would help troubleshoot this issue, because it contains information about the files that Splunk monitors and indexes, such as the file path, size, modification time, and CRC checksum. It also logs any errors or warnings that occur during the file monitoring process, such as permission issues, file rotation, or file truncation. The TailingProcessor channel can help identify if Splunk is reading the new data from the monitor input file or not, and what might be causing the problem. Option B is the correct answer. Option A is incorrect because the ModularInputs channel logs information about the modular inputs that Splunk uses to collect data from external sources, such as scripts, APIs, or custom applications. It does not log information about the monitor input file. Option C is incorrect because the ChunkedLBProcessor channel logs information about the load balancing process that Splunk uses to distribute data among multiple indexers. It does not log information about the monitor input file. Option D is incorrect because the ArchiveProcessor channel logs information about the archive process that Splunk uses to move data from the hot/warm buckets to the cold/frozen buckets. It does not log information about the monitor input file12

1: https://docs.splunk.com/Documentation/Splunk/9.1.2/Troubleshooting/WhatSplunklogsaboutitself#splunkd.log 2: https://docs.splunk.com/Documentation/Splunk/9.1.2/Troubleshooting/Didyouloseyourfishbucket#Check_the_splunkd.log_file

A multisite indexer cluster is a type of indexer cluster that spans multiple geographic locations or sites. A multisite indexer cluster requires only one cluster manager, also known as the master node, for the entire cluster. The cluster manager is responsible for coordinating the replication and search activities among the peer nodes across all sites. The cluster manager can reside in any site, but it must be accessible by all peer nodes and search heads in the cluster. Option C is the correct answer. Option A is incorrect because having two cluster managers for the entire cluster would introduce redundancy and complexity. Option B is incorrect because having one cluster manager for each site would create separate clusters, not a multisite cluster. Option D is incorrect because having two cluster managers for each site would be unnecessary and inefficient12

1: https://docs.splunk.com/Documentation/Splunk/9.1.2/Indexer/Multisiteoverview 2: https://docs.splunk.com/Documentation/Splunk/9.1.2/Indexer/Clustermanageroverview

The __introspection log file contains data about the impact of the Splunk software on the host system, such as CPU, memory, disk, and network usage, as well as KV store performance1. This log file is monitored by default and the contents are sent to the _introspection index1. The other options are not related to the __introspection log file. File monitor input configurations are stored in inputs.conf2. File monitor checkpoint offset is stored in fishbucket3. User activities and knowledge objects are stored in the _audit and _internal indexes respectively4.

According to the Splunk documentation1, the Splunk Deployer deploys app content to the etc/slave-apps/ directory on the search head cluster members by default. This directory contains the apps that the deployer distributes to the members as part of the configuration bundle. The other options are false because:

The etc/apps/ directory contains the apps that are installed locally on each member, not the apps that are distributed by the deployer2.

The etc/shcluster/ directory contains the configuration files for the search head cluster, not the apps that are distributed by the deployer3.

The etc/deploy-apps/ directory is not a valid Splunk directory, as it does not exist in the Splunk file system structure4.

The Splunk Managing Indexes and Storage Documentation explains that when multiple data sources with significantly different ingestion rates share a single index, index bucket management is governed by volume-based rotation, not by source or time. This means that high-volume data causes buckets to fill and roll more quickly, which in turn causes low-volume data to age out prematurely, even if it is relatively recent — hence Option C is correct.

Additionally, because Splunk organizes data within index buckets based on event time and storage characteristics, low-volume data mixed with high-volume data results in inefficient searches for smaller datasets. Queries that target the low-volume source will have to scan through the same large number of buckets containing the high-volume data, leading to slower-than-necessary search performance — Option B.

Compression efficiency (Option A) and performance optimization through data mixing (Option D) are not influenced by mixing volume patterns; these are determined by the event structure and compression algorithm, not source diversity. Splunk best practices recommend separating data sources into different indexes based on usage, volume, and retention requirements to optimize both performance and lifecycle management.

References (Splunk Enterprise Documentation):

• Managing Indexes and Storage – How Splunk Manages Buckets and Data Aging

• Splunk Indexing Performance and Data Organization Best Practices

• Splunk Enterprise Architecture and Data Lifecycle Management

• Best Practices for Data Volume Segregation and Retention Policies

Splunk provides various methods to change the internal logging levels in a Splunk environment to troubleshoot an issue. All of the options are valid ways to do so. Option A is correct because the Monitoring Console (MC) allows the administrator to view and modify the logging levels of various Splunk components through a graphical interface. Option B is correct because the Splunk command line provides the splunk set log-level command to change the logging levels of specific components or categories. Option C is correct because the Splunk Web provides the Settings > Server settings > Server logging page to change the logging levels of various components through a web interface. Option D is correct because the log-local.cfg file allows the administrator to manually edit the logging levels of various components by overriding the default settings in the log.cfg file123

1: https://docs.splunk.com/Documentation/Splunk/9.1.2/Troubleshooting/Enabledebuglogging 2: https://docs.splunk.com/Documentation/Splunk/9.1.2/Admin/Serverlogging 3: https://docs.splunk.com/Documentation/Splunk/9.1.2/Admin/Loglocalcfg

 The following is a good practice for a search head cluster deployer: The deployer must be used to distribute non-replicable configurations to search head cluster members. Non-replicable configurations are the configurations that are not replicated by the search factor, such as the apps and the server.conf settings. The deployer is the Splunk server role that distributes these configurations to the search head cluster members, ensuring that they have the same configuration. The deployer does not only distribute configurations to search head cluster members when they “phone home”, as this would cause configuration inconsistencies and delays. The deployer does not distribute configurations to search head cluster members to be valid configurations, as this implies that the configurations are invalid without the deployer. The deployer does not only distribute configurations to search head cluster members with splunk apply shcluster-bundle, as this would require manual intervention by the administrator. For more information, see Use the deployer to distribute apps and configuration updates in the Splunk documentation.

 A multi-site indexer cluster can be configured by directly editing SPLUNK_HOME/etc/system/local/server.conf or running a splunk edit cluster-config command from the CLI. These methods allow the administrator to specify the site attribute for each indexer node and the site_replication_factor and site_search_factor for the cluster. Configuring a multi-site indexer cluster via Splunk Web or directly editing SPLUNK_HOME/etc/system/default/server.conf are not supported methods. For more information, see Configure the indexer cluster with server.conf in the Splunk documentation.

All of the forwarders would still be phoning home to the old deployment server, because the forwarders are configured to use the old deployment server in $SPLUNK_HOME/etc/system/local. This is the local configuration directory that contains the settings that override the default settings in $SPLUNK_HOME/etc/system/default. The deploymentclient.conf file in the local directory specifies the targetUri of the deployment server that the forwarder contacts for configuration updates and apps. If the forwarders have the old deployment server’s targetUri in the local directory, they will ignore the updated deploymentclient.conf file that was deployed by the old deployment server, because the local settings have higher precedence than the deployed settings. To fix this issue, the forwarders should either remove the deploymentclient.conf file from the local directory, or update it with the new deployment server’s targetUri. Option C is the correct answer. Option A is incorrect because a version mismatch between the forwarders and the new deployment server would not prevent the forwarders from phoning home to the new deployment server, as long as they are compatible versions. Option B is incorrect because the new deployment server is configured and running, and there is no indication that it is not accepting connections from the forwarders. Option D is incorrect because the pass4SymmKey is the shared secret key that the deployment server and the forwarders use to authenticate each other. It does not affect the forwarders’ ability to phone home to the new deployment server, as long as it is the same on both sides12

1: https://docs.splunk.com/Documentation/Splunk/9.1.2/Updating/Configuredeploymentclients 2: https://docs.splunk.com/Documentation/Splunk/9.1.2/Admin/Wheretofindtheconfigurationfiles

To ensure that high-velocity sources will not have forwarding delays to the indexers, the default limit for maxKBps in limits.conf should be increased. This parameter controls the maximum bandwidth that a forwarder can use to send data to the indexers. By default, it is set to 256 KBps, which may not be sufficient for high-volume data sources. Increasing this limit can reduce the forwarding latency and improve the performance of the forwarders. However, this should be done with caution, as it may affect the network bandwidth and the indexer load. Option B is the correct answer. Option A is incorrect because the sessionTimeout parameter in server.conf controls the duration of a TCP connection between a forwarder and an indexer, not the bandwidth limit. Option C is incorrect because the forceTimebasedAutoLB parameter in outputs.conf controls the frequency of load balancing among the indexers, not the bandwidth limit. Option D is incorrect because the phoneHomelntervallnSecs parameter in deploymentclient.conf controls the interval at which a forwarder contacts the deployment server, not the bandwidth limit12

1: https://docs.splunk.com/Documentation/Splunk/9.1.2/Admin/Limitsconf#limits.conf.spec 2: https://docs.splunk.com/Documentation/Splunk/9.1.2/Forwarding/Routeandfilterdatad#Set_the_maximum_bandwidth_usage_for_a_forwarder

 The following statements are true regarding Splunk Enterprise performance:

Adding search peers increases the search throughput as search load increases. This is because adding more search peers distributes the search workload across more indexers, which reduces the load on each indexer and improves the search speed and concurrency.

Adding search heads provides additional CPU cores to run more concurrent searches. This is because adding more search heads increases the number of search processes that can run in parallel, which improves the search performance and scalability. The following statements are false regarding Splunk Enterprise performance:

Adding search peers does not increase the maximum size of search results. The maximum size of search results is determined by the maxresultrows setting in the limits.conf file, which is independent of the number of search peers.

Adding RAM to an existing search head does not provide additional search capacity. The search capacity of a search head is determined by the number of CPU cores, not the amount of RAM. Adding RAM to a search head may improve the search performance, but not the search capacity. For more information, see Splunk Enterprise performance in the Splunk documentation.

The correct Splunk deployment to have the recommended minimum components for a high-availability search head cluster is 3 search heads, 1 deployer, 3 indexers. This configuration ensures that the search head cluster has at least three members, which is the minimum number required for a quorum and failover1. The deployer is a separate instance that manages the configuration updates for the search head cluster2. The indexers are the nodes that store and index the data, and having at least three of them provides redundancy and load balancing3. The other options are not recommended, as they either have less than three search heads or less than three indexers, which reduces the availability and reliability of the cluster. Therefore, option B is the correct answer, and options A, C, and D are incorrect.

1: About search head clusters 2: Use the deployer to distribute apps and configuration updates 3: About indexer clusters and index replication

Increasing the number of parallel ingestion pipelines in server.conf is most likely to improve indexing performance when indexing is slow and real-time search results are delayed in a Splunk environment with two indexers and one search head. The parallel ingestion pipelines allow Splunk to process multiple data streams simultaneously, which increases the indexing throughput and reduces the indexing latency. Increasing the maximum number of hot buckets in indexes.conf will not improve indexing performance, but rather increase the disk space consumption and the bucket rolling time. Decreasing the maximum size of the search pipelines in limits.conf will not improve indexing performance, but rather reduce the search performance and the search concurrency. Decreasing the maximum concurrent scheduled searches in limits.conf will not improve indexing performance, but rather reduce the search capacity and the search availability. For more information, see Configure parallel ingestion pipelines in the Splunk documentation.

The correct action to take first if a deployment client is not updating apps is to choose a corrective action based on the splunkd.log of the deployment client. This log file contains information about the communication between the deployment server and the deployment client, and it can help identify the root cause of the problem1. The other actions may or may not help, depending on the situation, but they are not the first steps to take. Choosing a longer phone home interval may reduce the load on the deployment server, but it will also delay the updates for the deployment clients2. Increasing the number of CPU cores or the amount of memory for the deployment server may improve its performance, but it will not fix the issue if the problem is on the deployment client side3. Therefore, option C is the correct answer, and options A, B, and D are incorrect.

1: Troubleshoot deployment server issues 2: Configure deployment clients 3: Hardware and software requirements for the deployment server

As per Splunk Enterprise Search Performance documentation, the most significant factor affecting search performance when querying across a large number of buckets is disk I/O throughput. A search that spans “All Time” forces Splunk to inspect all historical buckets (hot, warm, cold, and potentially frozen if thawed), even if only a few events match the query. This dramatically increases the amount of data read from disk, making the search bound by I/O performance rather than CPU or memory.

Increasing the number of indexing pipelines (Option B) only benefits data ingestion, not search performance. Changing to a real-time search (Option D) does not help because real-time searches are optimized for streaming new data, not historical queries. The indexed_realtime_use_by_default setting (Option C) applies only to streaming indexed real-time searches, not historical “All Time” searches.

To improve performance for such searches, Splunk documentation recommends enhancing disk I/O capability — typically through SSD storage, increased disk bandwidth, or optimized storage tiers. Additionally, creating summary indexes or accelerated data models may help for repeated “All Time” queries, but the most direct improvement comes from faster disk performance since Splunk must scan large numbers of buckets for even small result sets.

References (Splunk Enterprise Documentation):

• Search Performance Tuning and Optimization

• Understanding Bucket Search Mechanics and Disk I/O Impact

• limits.conf Parameters for Search Performance

• Storage and Hardware Sizing Guidelines for Indexers and Search Heads

Splunk Enterprise officially requires a minimum of three search heads and one deployer for a supported Search Head Cluster (SHC) configuration. This ensures both high availability and data consistency within the cluster.

The Splunk documentation explains that a search head cluster uses RAFT-based consensus to elect a captain responsible for managing configuration replication, scheduling, and user workload distribution. The RAFT protocol requires a quorum of members to maintain consistency. In practical terms, this means a minimum of three members (search heads) to achieve fault tolerance — allowing one member to fail while maintaining operational stability.

The deployer is a separate Splunk instance responsible for distributing configuration bundles (apps, settings, and user configurations) to all members of the search head cluster. The deployer is not part of the SHC itself but is mandatory for its proper management.

Running with fewer than three search heads or replacing the deployer with a Deployment Server (as in Options B, C, or D) is unsupported and violates Splunk best practices for SHC resiliency and management.

References (Splunk Enterprise Documentation):

• Search Head Clustering Overview – Minimum Supported Architecture

• Deploy and Configure the Deployer for a Search Head Cluster

• High Availability and Fault Tolerance with RAFT in SHC

 The Splunk instance with the given settings in SPLUNK_HOME/etc/system/local/server.conf is missing the master_uri attribute and needs to be restarted. The master_uri attribute is required for the master node to communicate with the peer nodes and the search head cluster. The master_uri attribute specifies the host name and port number of the master node. Without this attribute, the master node cannot function properly. The Splunk instance also needs to be restarted for the changes in the server.conf file to take effect. The replication_factor setting determines how many copies of each bucket are maintained across the peer nodes. The search factor is a separate setting that determines how many searchable copies of each bucket are maintained across the peer nodes. The search factor is not specified in the given settings, so it defaults to the same value as the replication factor, which is 2. This is not a multi-site cluster, because the site attribute is not specified in the clustering stanza. A multi-site cluster is a cluster that spans multiple geographic locations, or sites, and has different replication and search factors for each site.

KV Store is a feature of Splunk Enterprise that allows apps to store and retrieve data within the context of an app1.

KV Store resides on search heads and replicates data across the members of a search head cluster1.

KV Store resiliency refers to the ability of KV Store to maintain data availability and consistency in the event of failures or disruptions2.

One of the factors that affects KV Store resiliency is the network latency between search heads, which can impact the speed and reliability of data replication2.

Decreasing latency between search heads can improve KV Store resiliency by reducing the chances of data loss, inconsistency, or corruption2.

The other options are not directly related to KV Store resiliency. Faster storage, indexer CPU and memory, and Operations Log size may affect other aspects of Splunk performance, but not KV Store345.

The walklex command in Splunk is a specialized administrative search command used to translate and display LISPY (Splunk’s internal representation of search terms). LISPY is the logical search syntax Splunk uses to parse and execute search queries, and examining it helps administrators and developers debug search optimization, field extraction behavior, and index-time search efficiency.

When you run the command | walklex search="your_search_string", Splunk outputs how it tokenizes and interprets that query internally. This is particularly useful for understanding how Splunk’s search language maps to index-time fields and for diagnosing performance issues caused by inefficient search term parsing.

For example:

| walklex search="error OR failure host=server01"

Displays the corresponding LISPY translation used by Splunk’s search subsystem.

Other options are unrelated:

dbinspect provides index bucket metadata.

Monitoring Console shows performance metrics and health status.

Search Job Inspector analyzes search execution phases but doesn’t expose LISPY.

Thus, the correct and Splunk-documented tool for LISPY inspection is the walklex command.

References (Splunk Enterprise Documentation):

• walklex Command Reference – LISPY and Search Debugging

• Understanding Search Language Parsing in Splunk

• Search Internals: How Splunk Interprets Queries

• Splunk Search Performance Troubleshooting Tools

Migration from single-site to multisite index replication only affects new data, not existing data. Multisite policies apply to new data only, meaning that data that is ingested after the migration will follow the multisite replication and search factors. Existing data, or data that was ingested before the migration, will retain the single-site policies, unless they are manually converted to multisite buckets. Single-site buckets do not instantly receive the multisite policies, nor do they automatically convert to multisite buckets. Multisite total values can exceed any single-site factors, as long as they do not exceed the number of peer nodes in the cluster. A master node is not required at each site, only one master node is needed for the entire cluster

When adding or decommissioning a member from a Search Head Cluster (SHC), the proper order of operations is:

Delete Splunk Enterprise, if it exists.

Install and initialize the instance.

Join the SHC.

This order of operations ensures that the member has a clean and consistent Splunk installation before joining the SHC. Deleting Splunk Enterprise removes any existing configurations and data from the instance. Installing and initializing the instance sets up the Splunk software and the required roles and settings for the SHC. Joining the SHC adds the instance to the cluster and synchronizes the configurations and apps with the other members. The other order of operations are not correct, because they either skip a step or perform the steps in the wrong order.

Splunk Enterprise uses the _audit index to log and store all user activity and audit-related information. This includes details such as user logins, searches executed, configuration changes, role modifications, and app management actions.

The _audit index is populated by data collected from the Splunkd audit logger and records actions performed through both Splunk Web and the CLI. Each event in this index typically includes fields like user, action, info, search_id, and timestamp, allowing administrators to track activity across all Splunk users and components for security, compliance, and accountability purposes.

The _internal index, by contrast, contains operational logs such as metrics.log and scheduler.log used for system performance and health monitoring. _kvstore stores internal KV Store metadata, and _telemetry is used for optional usage data reporting to Splunk.

The _audit index is thus the authoritative source for user behavior monitoring within Splunk environments and is a key component of compliance and security auditing.

References (Splunk Enterprise Documentation):

• Audit Logs and the _audit Index – Monitoring User Activity

• Splunk Enterprise Security and Compliance: Tracking User Actions

• Splunk Admin Manual – Overview of Internal Indexes (_internal, _audit, _introspection)

• Splunk Audit Logging and User Access Monitoring

In the deployment planning process, a person should identify who gets to see network data in the data policy definition step. This step involves defining the data access policies and permissions for different users and roles in Splunk. The deployment schedule step involves defining the timeline and milestones for the deployment project. The topology diagramming step involves creating a visual representation of the Splunk architecture and components. The data source inventory step involves identifying and documenting the data sources and types that will be ingested by Splunk

 The following artifacts are included in a Splunk diag file:

Server specs. These are the specifications of the server that Splunk runs on, such as the CPU model, the memory size, the disk space, and the network interface. These specs can help understand the Splunk hardware requirements and performance.

Current open connections. These are the connections that Splunk has established with other Splunk instances or external sources, such as forwarders, indexers, search heads, license masters, deployment servers, and data inputs. These connections can help understand the Splunk network topology and communication.

Internal Splunk log files. These are the log files that Splunk generates to record its own activities, such as splunkd.log, metrics.log, audit.log, and others. These logs can help troubleshoot Splunk issues and monitor Splunk performance.

Index listings. These are the listings of the indexes that Splunk has created and configured, such as the index name, the index location, the index size, and the index attributes. These listings can help understand the Splunk data management and retention. The following artifacts are not included in a Splunk diag file:

Search history. This is the history of the searches that Splunk has executed, such as the search query, the search time, the search results, and the search user. This history is not part of the Splunk diag file, but it can be accessed from the Splunk Web interface or the audit.log file.

Splunk users and their roles. These are the users that Splunk has created and assigned roles to, such as the user name, the user password, the user role, and the user capabilities. These users and roles are not part of the Splunk diag file, but they can be accessed from the Splunk Web interface or the authentication.conf and authorize.conf files.

KV store listings. These are the listings of the KV store collections and documents that Splunk has created and stored, such as the collection name, the collection schema, the document ID, and the document fields. These listings are not part of the Splunk diag file, but they can be accessed from the Splunk Web interface or the mongod.log file.

Indexed data. These are the data that Splunk indexes and makes searchable, such as the rawdata and the tsidx files. These data are not part of the Splunk diag file, as they may contain sensitive or confidential information. For more information, see Generate a diagnostic snapshot of your Splunk Enterprise deployment in the Splunk documentation.