Feature: Enforce Generic Workload Restrictions Across Devtron App manifests using CEL-Based Policies

[prakarsh-dt]

🔖 Feature description

In Kubernetes clusters managed by Devtron, manifests generated by the Devtron reference chart (used for application deployments) should adhere to best practices and organizational policies to ensure reliability, security, and performance. We propose a new feature to enforce generic restrictions on these manifests using Common Expression Language (CEL)-based policies, applied in deployment template to validate manifests before they deployed.

1. This feature would enable administrators to define CEL policies that enforce requirements such as:
2. Enabling autoscaling
3. Setting a minimum number of replicas with topology spread across availability zones
4. Configuring liveness and readiness probes
5. Preventing root access for containers.

🎤 Pitch / Usecases

CEL-Based Policy Definition:
Add a new configuration section in Devtron’s dashboard (e.g., under Global Configurations or Application Policies) for defining CEL policies that apply to all manifests generated by the Devtron reference chart.

Example CEL expressions for the requested restrictions:

Autoscaling Enabled:

has(deployment.spec.template.spec.autoscaling) && deployment.spec.template.spec.autoscaling.enabled == true

Minimum 3 Replicas with Topology Spread Across 3 AZs:

deployment.spec.replicas >= 3 && has(deployment.spec.template.spec.topologySpreadConstraints) && deployment.spec.template.spec.topologySpreadConstraints.exists(tsc, tsc.topologyKey == "topology.kubernetes.io/zone" && tsc.maxSkew <= 1)

Liveness and Readiness Probes Configured:

deployment.spec.template.spec.containers.all(c, has(c.livenessProbe) && has(c.readinessProbe))

No Root Access for Containers:

deployment.spec.template.spec.containers.all(c, has(c.securityContext) && c.securityContext.runAsNonRoot == true)

Support CEL variables for manifest metadata (e.g., namespace, labels, annotations) and resource properties (e.g., CPU/memory requests, workload type).

Policy Enforcement:
Integrate CEL policy validation into Devtron’s deployment template as a pre-deployment checklist, evaluating deployment template values/manifests generated by the reference charts.

Use Devtron policy engine to enforce policies, rejecting non-compliant manifests with detailed error messages (e.g., “Deployment not permitted: Autoscaling is not enabled”).
Allow super-admin users and other whitelisted users to override policies for specific applications or environments, with audit logging for accountability.

User Experience:
Provide a user-friendly UI in Devtron’s dashboard to author and manage CEL policies, including syntax validation and drag and drop templates for common restrictions (e.g., autoscaling, probes, topologySpread etc).
Display policy evaluation results in the deployment template edit view and deployment trigger view, clearly indicating which policies passed or failed.

Extensibility:

• Support multiple CEL policies with top to bottom priority (like in security policies) (e.g., global policies, cluster/environment-specific policies, or app-specific overrides).
• Integrate with Devtron’s RBAC to control who can create or modify policies (Or let this be with the superadmins only).
• Allow exemptions for specific manifests via annotations (e.g., devtron.ai/policy-exempt: true), subject to admin approval.
• Enable a dry-run mode to simulate policy enforcement without blocking deployments, aiding policy development.

Use Case:
A Devtron user managing a cluster wants to enforce best practices for all applications deployed via the Devtron. They define CEL policies to ensure autoscaling is enabled, workloads have at least 3 replicas spread across 3 AZs, liveness/readiness probes are configured, and containers do not run as root. When a developer deploys an application missing readiness probes, Devtron rejects the deployment with a clear error, guiding them to fix the configuration.

Benefits:
Deterministic Enforcement: CEL policies ensure manifests comply with organizational standards, preventing misconfigurations from reaching the cluster.
Flexibility: CEL’s expressive syntax supports complex rules across resources, metadata, and configurations.
Improved Reliability and Security: Enforcing autoscaling, replicas, probes, and non-root containers enhances workload resilience and security.
Enhanced Developer Experience: Clear error messages and UI support streamline debugging and compliance.

Additional Context:
This feature supports Devtron’s goal of simplifying Kubernetes management while ensuring robust, secure deployments in various environments.
CEL’s adoption in Kubernetes (e.g., for CRD validation and admission policies) makes it an ideal choice for policy enforcement.
Enforcing restrictions at the manifest level ensures consistency across all Devtron-managed workloads.

🔄️ Alternative

Kyverno or OPA policy based restrictions applied directly on ALL target clusters

👀 Have you spent some time to check if this issue has been raised before?

• I checked and didn't find similar issue

🏢 Have you read the Code of Conduct?

• I have read the Code of Conduct

[prakarsh-dt]

Similar to #6382

[ajaydevtron]

Some more CEL policies that enforce requirements as below :

1. Resource Requests and Limits to define the better QOS
2. Keep Applications Updated (Regularly update your application to the latest chart version available on the Devtron Dashboard to benefit from new features, improvements, and security patches)