Splunk Configuration file precedence

How does Splunk prioritize and merge the configuration files?

When editing configuration files, it is important to understand how Splunk software evaluates these files and which ones take precedence.

When incorporating changes, Splunk software does the following to your configuration files:

• It merges the settings from all copies of the file, using a location-based prioritization scheme.
• When different copies have conflicting attribute values (that is, when they set the same attribute to different values), it uses the value from the file with the highest priority.
• It determines the priority of configuration files by their location in the directory structure, according to the rules described in this topic.

Index-Time Process:

1. Input phase: Handled at the source (usually a forwarder)
– The data sources are being opened and read
– Data is handled as streams; configuration settings are applied to the entire stream

2. Parsing phase: Handled by indexers (or heavy forwarders)
– Data is broken up into events and advanced processing can be performed

3. Indexing phase: Handled by indexers
– License meter runs as data is initially written to disk, prior to compression – After data is written to disk, it cannot be changed

Index-time processes take place between the point when the data is consumed and the point when it is written to disk.

The following processes occur during index time:

• Default field extraction (such as host, source, sourcetype, and timestamp)
• Static or dynamic host assignment for specific inputs
• Default host assignment overrides
• Source type customization
• Custom index-time field extraction
• Structured data field extraction
• Event timestamping
• Event linebreaking
• Event segmentation (also happens at search time)

Search-time Process:

Search-time processes take place while a search is run, as events are collected by the search. The following processes occur at search time:

• Event segmentation (also happens at index time)
• Event type matching
• Search-time field extraction (automatic and custom field extractions, including multivalue fields and calculated fields)
• Field aliasing
• Addition of fields from lookups
• Source type renaming
• Tagging

What is configuration file context?

To determine the order of directories for evaluating configuration file precedence, Splunk software considers each file’s context. Configuration files operate in either a global context or in the context of the current app and user:

• Global. Activities like indexing take place in a global context. They are independent of any app or user. For example, configuration files that determine monitoring or indexing behavior occur outside of the app and user context and are global in nature.
• App/user. Some activities, like searching, take place in an app or user context. The app and user context is vital to search-time processing, where certain knowledge objects or actions might be valid only for specific users in specific apps.

Default vs. Local Configuration

Splunk software uses configuration files to determine nearly every aspect of its behavior. A Splunk platform deployment can have many copies of the same configuration file. These file copies are usually layered in directories that affect either the users, an app, or the system as a whole.

When editing configuration files, it is important to understand how Splunk software evaluates these files and which ones take precedence.

When incorporating changes, Splunk software does the following to your configuration files:

• It merges the settings from all copies of the file, using a location-based prioritization scheme.
• When different copies have conflicting attribute values (that is, when they set the same attribute to different values), it uses the value from the file with the highest priority.
• It determines the priority of configuration files by their location in the directory structure, according to the rules described in this topic.

Avoid storing configurations in SPLUNK_HOME/etc/system/…

Manage your configurations in the appropriate app under

SPLUNK_HOME/etc/apps/<appname>/local

– If you don’t have an app, create one to manage system settings

Merging of Splunk Configuration Files

Splunk merges configuration files

• –  Generally when Splunk starts, or when searches are run
• –  Into a single run-time model for each file type
• –  As a union of all files if no duplicates/conflicts exist

• In case of conflicts, priority is based on the context:

– Global context (index-time)
– App/User context (search-time)

How does Splunk determine precedence order?

Configuration file precedence order depends on the location of file copies within the directory structure. Splunk software considers the context of each file to determine the precedence order of the directories.

Index-Time Precedence (Global Context)

When the file context is global, directory priority descends in this order:

1. System local directory — highest priority
2. App local directories
3. App default directories
4. System default directory — lowest priority

When consuming a global configuration, such as inputs.conf, Splunk software first uses the attributes from any copy of the file in system/local. Then it looks for any copies of the file located in the app directories, adding any attributes found in them, but ignoring attributes already discovered in system/local. As a last resort, for any attributes not explicitly assigned at either the system or app level, it assigns default values from the file in the system/default directory.

Example of Index-Time Precedence

Search-Time Precedence (App/User Context)

For files with an app/user context, directory priority descends from user to app to system:

1. User directories for current user — highest priority
2. App directories for currently running app (local, followed by default)
3. App directories for all other apps (local, followed by default) — for exported settings only
4. System directories (local, followed by default) — lowest priority

An attribute in savedsearches.conf, for example, might be set at all three levels: the user, the app, and the system. Splunk will always use the value of the user-level attribute, if any, in preference to a value for that same attribute set at the app or system level.

The effect of app directory names varies depending on whether the context is global or local.

App directory names in the global context

When determining priority in the global context, Splunk software uses lexicographical order to determine priority among the collection of apps directories. For example, files in an apps directory named “A” have a higher priority than files in an apps directory named “B”, and so on.

App directory names in the app/user context When determining priority in the app/user context, Splunk software uses reverse-lexicographical order to determine priority among the collection of apps directories, For example, files in an apps directory named “B” have a higher priority than files in an apps directory named “A”, and so on.