Proposal: Reusable File-Based Secret Provider

[TheMeier]

DISCLAIMER this proposal is AI generated

1. Problem Statement

Currently, various notifiers (Email, WeChat, etc.) implement their own logic to retrieve secrets. When *_file configuration options are used (e.g., smtp_auth_password_file), the standard implementation often uses os.ReadFile directly inside the Notify loop.

This leads to:

1. Performance overhead: Disk I/O is performed for every single notification.
2. Code duplication: Similar file-reading logic exists across different packages.
3. Lack of reactivity: There is no centralized way to handle secret rotation (e.g., Kubernetes Secrets updates) without restarting the process or relying on the OS filesystem cache.

2. Proposed Solution

Introduce a SecretFileProvider (or SecretCache) component. This component will be responsible for reading, caching, and watching files containing secrets.

Location

I recommend placing this in a new package pkg/secrets or util/secrets to avoid circular dependencies with the config package, as config is imported by almost everyone.

3. API Design

We define a strategy enum to handle the different use cases requested:

package secrets

type RefreshStrategy int

const (
	// StrategyReadAlways performs an os.ReadFile every time. 
	// Useful for development or filesystems that don't support watches.
	StrategyReadAlways RefreshStrategy = iota

	// StrategyReadOnce reads the file on the first request and caches it 
	// until the process restarts or the provider is reset.
	StrategyReadOnce

	// StrategyFileWatch reads the file and sets up an fsnotify watcher.
	// The cache is updated automatically when the file changes.
	StrategyFileWatch
)

type Provider struct {
    // ... internal fields (mutex, cache map, fsnotify watcher)
}

// NewProvider creates a new secret provider. 
// It initializes the fsnotify watcher if needed.
func NewProvider(logger *slog.Logger) (*Provider, error) { ... }

// GetString retrieves the secret content.
func (p *Provider) GetString(filename string, strategy RefreshStrategy) (string, error) { ... }

// Close cleans up watchers.
func (p *Provider) Close() error { ... }

4. Implementation Details

The Get Logic

The GetString method will function as follows:

1. Check Cache: Acquire a read lock. If the file is in the map and the strategy is ReadOnce or FileWatch, return the cached value.
2. Cache Miss / ReadAlways:
   • Acquire write lock.
   • Read file via os.ReadFile.
   • Store in cache.
3. Setup Watch (if StrategyFileWatch):
   • If the file is not already being watched, add it to the fsnotify watcher.
   • Kubernetes Note: Special care must be taken to watch the directory or handle the ..data symlink swaps common in Kubernetes Secrets, otherwise fsnotify might lose track of the file after an atomic update.

Thread Safety

Since Notify calls happen concurrently, the Provider must use sync.RWMutex to protect the internal cache map.

5. Usage Example

Here is how the implementation in email.go would change.

Before:

func (n *Email) getPassword() (string, error) {
	if len(n.conf.AuthPasswordFile) > 0 {
		content, err := os.ReadFile(n.conf.AuthPasswordFile)
		if err != nil {
			return "", fmt.Errorf("could not read %s: %w", n.conf.AuthPasswordFile, err)
		}
		return strings.TrimSpace(string(content)), nil
	}
	return string(n.conf.AuthPassword), nil
}

After:
Assuming the Email struct now holds a reference to the global or context-injected SecretProvider:

func (n *Email) getPassword() (string, error) {
	if len(n.conf.AuthPasswordFile) > 0 {
        // Use StrategyFileWatch to support rotation without restart
		return n.secretProvider.GetString(n.conf.AuthPasswordFile, secrets.StrategyFileWatch)
	}
	return string(n.conf.AuthPassword), nil
}

6. Lifecycle Management

The SecretProvider should ideally be initialized in main.go or the coordinator and passed down to the pipeline and subsequently to the Notifiers.

When Alertmanager reloads its configuration:

1. The old SecretProvider could be closed (stopping all watches).
2. A new one is created.
3. Alternatively: The provider can be a long-lived singleton that clears its cache on config reload signals.

[siavashs]

Related prometheus/prometheus#13955