What is a Zone

A Zone is used to separate different teams or products that share the same Kubernetes cluster. Improved security through access permissions, policies, network rules, and dedicated compute resources results in far better isolation than using Namespaces alone. The main reasons for using Zones are security and cost savings.

A Zone contains Namespaces. A Zone can create Kubernetes Node Pools (in AWS, NodeGroups). Pods assigned to a Namespace that belongs to a Zone automatically receive the appropriate NodeSelectors and are scheduled on the correct Node Pool. The PSA of the Namespace is controlled by the Zone, preventing any privileged containers from starting. The PSA enforcement level is configurable at the cluster level. Since a Zone can contain multiple Namespaces and Node Pools, it is also possible to specify which Node Pools are used by each Namespace in a Zone. Per-Pod Affinity is also supported. This is convenient for users, as the required affinity rules are applied automatically unless the user wants to control them manually. In the latter case, validations are in place to enforce the rules.

If the granularNamespaceNetworkPolicy is false(default) then the Zone allows network traffic between Namespaces that belong to the same Zone. When Namespaces in different Zones need to communicate, they must always allow incoming connections using standard Kubernetes NetworkPolicy objects. Only NetworkPolicy ingress rules need to be added; egress is open by default.

When Ingress objects are created, the necessary NetworkPolicy for the load balancer to access the Service Pods is created automatically.

Each Zone also creates an ArgoCDProject with permissions to deploy resources to the Zone's Namespaces, enabling the use of "app-of-apps" per team while keeping them within the designated Zone.

Zones can also assign Contributor groups to restrict access to only the Namespaces within the Zone. This access is granted in both ArgoCD and the Kubernetes API.

A Demo video of Zone can be found here https://youtu.be/D1Eg0rE3nSI.

Assigning Permissions

There are two configurations for roleMapping that determine the permissions of user groups: global permissions, assignable through the "platform-api" Helm chart values, and per-Zone permissions, assignable through the Zone object spec. Globally assigned roles apply across all Zones and their Namespaces in the Kubernetes cluster. Per-Zone permissions apply only to that Zone's Namespaces.

The permissions a role has via the Kubernetes API (kubectl) are nearly identical to the permissions granted in the Zone's ArgoCDProject. Due to technical limitations, these permissions cannot be 100% equal.

Detailed Role Descriptions

Available roles and their purpose.

Observer is intended for users who need read-only access to applications without the ability to make changes. It can read objects but not modify them. Can read cluster-scoped objects: namespaces, nodes, persistentvolumes, storageclasses, ingressclasses, ingressclassparams.elbv2.k8s.aws, zones, nodegroups.eks.aws.upbound.io. It can read all namespace-scoped objects in the Zone's Namespaces.

Contributor is intended for users who need to deploy and manage applications within the Zone boundary. It can create new Namespaces in Zones (through ArgoCDApplications, but not kubectl; the Namespace is assigned to the Zone whose ArgoCDProject it was created in). It can read the same cluster-scoped objects as Observer. Can read, create, edit, and delete all namespace-scoped objects in the Zone's Namespaces, including ArgoCDApplication objects, but only within the Zone's ArgoCDProject. Can configure ArgoCDApplications and repositories.

Maintainer is intended for configuring Zone objects and enabling Contributors to deploy applications within the Zone boundary. Maintainers can also perform all Contributor duties. A Maintainer is less powerful than a cluster administrator and has no permissions to manage the "infralib" Zone or its Namespaces. It can read, create, edit, and delete Zones and Namespaces. Can assign user groups to Contributor permissions in a Zone using the spec.roleMapping field in the Zone object. Has all permissions of Contributor. Due to the presence of spec.clusterPermissions in the Zone object, a Maintainer is able to escalate privileges to cluster administrator.

Future state: Currently the capabilities and existence of these roles are provided by the platform-api. In the future it should be possible to tune the permissions and the roles that exist. This is why the roleMapping uses roleRef as a text field.

Infralib Configuration Example for Global/Cluster-wide roleMappings

steps:
  ...
  - name: apps
    type: argocd-apps
    modules:
    ...
    - name: platform-apis
      source: platform-apis
      inputs:
        platform-apis:
          zone:
            environmentConfig:
              roleMapping:
                - roleRef: maintainer
                  groups: ["maintainer-group"]
                - roleRef: observer
                  groups: ["observer-group"]

Infralib Configuration Example for Per-Zone roleMappings

apiVersion: tenancy.entigo.com/v1alpha1
kind: Zone
metadata:
  name: example-zone
spec:
  roleMapping:
    - groups:
        - contributor-group
      roleRef: contributor
    - groups:
        - observer-group
      roleRef: observer

Infralib Configuration Example for Tuning ArgoCDProject Permissions

It is possible to change what resources are whitelisted or blacklisted in the Zone ArgoCDProject or the project "zone" using Helm values in the platform-apis.

        platform-apis:
          zone:
            environmentConfig:
              appProject:
                namespaceResourceBlacklist: #Affects the namespace resource blacklist of the Zone ArgoCDProject. Example contains default values.
                - group: '*.m.upbound.io'
                  kind: '*'
                namespaceResourceWhitelist: []
          clusterResourceWhitelist: #Affects the cluster resource whitelist of the "zone" project. Example contains default values.
          - group: tenancy.entigo.com
            kind: Zone
          - group: eks.aws.upbound.io
            kind: NodeGroup

Example of a Zone

This is an example of how to create a zone. This will create two Node Groups in AWS: "default" and "myspot".

It is a good practice to manage zones using a GitOps methodology, similar to how applications are deployed. For better organization, consider creating a dedicated Git repository and ArgoCDProject specifically for zones.

Use a Git repository with the zone manifest, or create the zone using kubectl.

A Zone cannot be named default. Each Zone gets an ArgoCDProject of the same name, and default is already ArgoCD's own project, so creating it is rejected with 'default' zone cannot be created.

# Example zone
apiVersion: tenancy.entigo.com/v1alpha1
kind: Zone
metadata:
  name: example-zone
  labels:
    tenancy.entigo.com/default-zone: "true"
spec:
  roleMapping:
    - groups:
        - 123456789-1234-1234-1234-123456789
      roleRef: contributor
  pools:
    - name: default
      requirements:
        - key: instance-type
          values:
            - t3.large
        - key: capacity-type
          value: ON_DEMAND
        - key: min-size
          value: 1
        - key: max-size
          value: 2
    - name: myspot
      requirements:
        - key: instance-type
          values:
            - t3.large
        - key: capacity-type
          value: SPOT
        - key: min-size
          value: 2
        - key: max-size
          value: 2

Example for Zonal NodeGroups

In this example we want to create NodeGroups per zone and assign a StatefulSet to be deployed across multiple zones.

# Example zone
apiVersion: tenancy.entigo.com/v1alpha1
kind: Zone
metadata:
  name: example-zonal
spec:
  pools:
    - name: default-a
      requirements:
        - key: instance-type
          values:
            - t3.large
        - key: capacity-type
          value: ON_DEMAND
        - key: min-size
          value: 1
        - key: max-size
          value: 1
        - key: zone
          values:
            - eu-north-1a
    - name: default-b
      requirements:
        - key: instance-type
          values:
            - t3.large
        - key: capacity-type
          value: ON_DEMAND
        - key: min-size
          value: 1
        - key: max-size
          value: 1
        - key: zone
          values:
            - eu-north-1b

Then add a podAntiAffinity rule for the Pods with topology topology.kubernetes.io/zone.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    app: example
  name: example
spec:
  replicas: 2
  selector:
    matchLabels:
      app: example
  serviceName: example
  template:
    metadata:
      labels:
        app: example
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: app
                    operator: In
                    values:
                      - example
              topologyKey: topology.kubernetes.io/zone
      containers:
        - image: nginxinc/nginx-unprivileged
          name: nginx

Create a Zone Using ArgoCD

Please watch the demo video https://youtu.be/D1Eg0rE3nSI

The ArgoCD App of Apps pattern

A Namespace for ArgoCD Application and ApplicationSet objects is automatically created when a Zone is created. The naming convention is <zone-name>-apps. The purpose of this Namespace is to make the app-of-apps pattern easy to use with Zones. Otherwise, this Namespace would have to be created manually. Users of a Zone are encouraged to create their ArgoCD Applications in this Namespace and have those Applications create the subsequent Namespaces for application resources.

Creating Applications in an ArgoCD Project that belongs to the Zone will automatically assign the same Zone to any Namespaces that are created. The ArgoCD Application destination namespace is tracked for this purpose.

An Application can also set the labels and annotations of the Namespace it deploys to, see Namespace Labels and Annotations from an Application.

This apps Namespace is managed by the Zone itself and cannot be moved to another Zone. It is also deleted when a Zone is deleted, which is why no other resources are permitted inside it.

To disable the creation of this Namespace, set createAppsNamespace to false in the platform-api Helm values file.

This pattern is also show in the demo video https://www.youtube.com/watch?v=D1Eg0rE3nSI&t=180

Assign a Namespace to a Zone

$ kubectl create ns example-ns

Since the "example-zone" zone has the label tenancy.entigo.com/default-zone with value true, the Namespace is automatically assigned to that Zone. If no Zone has this label, the first available Zone is chosen.

To change the Zone of the Namespace example-ns to the Zone another-zone, update the tenancy.entigo.com/zone label on the Namespace.

$ kubectl label ns example-ns tenancy.entigo.com/zone=another-zone --overwrite

It is possible to assign a Zone to a Namespace immediately by defining the labels.

apiVersion: v1
kind: Namespace
metadata:
  name: a2
  labels:
    tenancy.entigo.com/zone: example-zone
    tenancy.entigo.com/pool: myspot

To specify the Pool to use for this Namespace, set the tenancy.entigo.com/pool label. This will schedule Pods in this Namespace onto that pool of nodes. The value must name one of the Zone's own Pools, otherwise Pods in the Namespace are rejected as they are created, with an error listing the Pools that are valid.

After the Namespace is created, the permissions are also applied to the Zone's ArgoCDProject, allowing these Namespaces to be used from ArgoCD.

The objects a Zone creates for a new Namespace — its permissions, and the NetworkPolicies that let a load balancer reach the Pods of an Ingress — appear shortly after the Namespace, not at the same moment. A Zone watches a large number of objects and pauses briefly when too many of them change at once, so give it a few minutes before assuming something is wrong. Until the permissions arrive, actions on the new Namespace can be refused.

If the user does not want to create Namespaces using kubectl, they can also create a new Application in ArgoCD. When this Application belongs to a Zone's project, the destination Namespace is automatically created and assigned to the same Zone.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: a3
  namespace: a-apps
spec:
  destination:
    server: https://kubernetes.default.svc
    namespace: a3
  project: example-zone
  source:
    ...

If the "a3" Namespace did not exist, it will be created and assigned to the example-zone Zone.

Namespace Labels and Annotations from an Application

An ArgoCD Application can describe the labels and annotations of its destination Namespace under spec.syncPolicy.managedNamespaceMetadata. ArgoCD is not allowed to manage Namespaces through a Zone's ArgoCDProject, so the Zone applies this metadata to the Namespace on ArgoCD's behalf, and the field works the same way it would in a cluster without Zones.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: a3
  namespace: a-apps
spec:
  destination:
    server: https://kubernetes.default.svc
    namespace: a3
  project: example-zone
  syncPolicy:
    managedNamespaceMetadata:
      labels:
        team: payments
        tenancy.entigo.com/pool: myspot
        pod-security.kubernetes.io/enforce: restricted
      annotations:
        owner: payments@example.com
  source:
    ...

The "a3" Namespace is created, assigned to the example-zone Zone, and then labelled and annotated as the Application declares. Nothing has to be applied with kubectl afterwards.

Metadata of a new Application is applied as soon as the Application is created. Later edits to an existing Application are picked up by a periodic refresh, every 10 minutes by default, so a change to an Application that already exists is not visible on the Namespace immediately.

Which Namespace an Application Describes

An Application only describes the Namespace named in its own spec.destination.namespace, and only while that Namespace belongs to the same Zone as the Application's project. Moving a Namespace to another Zone with kubectl therefore stops the Applications of the old Zone from writing to it.

The <zone-name>-apps Namespace is managed by the Zone itself and is never described this way. If several Applications of a Zone deploy to the same Namespace, the first one found wins, so keep the metadata in a single Application per Namespace.

Keys That Are Ignored

The keys that decide where a Namespace belongs cannot be set from an Application, as either a label or an annotation:

Key	Where it is set instead
tenancy.entigo.com/*	With kubectl on the Namespace, see Assign a Namespace to a Zone. This is what decides which Zone a Namespace belongs to. The Pool label is the one exception, see below.
kubernetes.io/metadata.name	By Kubernetes itself.
app.kubernetes.io/managed-by, generate.kyverno.io/*	By the platform, on Namespaces it creates.

These keys are ignored rather than rejected: an Application that carries one is still accepted and the rest of its labels and annotations are applied as usual.

The Pool Label

tenancy.entigo.com/pool is applied, so an Application can pick which of its Zone's Pools the Namespace runs on, the same choice kubectl on the Namespace gives.

The value has to name a Pool of the Zone that the Namespace belongs to. The Namespace accepts any value, but the Zone validates it on every Pod: with a name that matches no Pool, Pods in the Namespace are rejected as they are created, with an error listing the Pools that are valid. ArgoCD shows that error on the ReplicaSet, StatefulSet or Job that tried to create the Pod, rather than on the Application that set the label.

The Pool applies to Pods created after the label has reached the Namespace. Pods that are already running are not rescheduled.

Pod Security Labels

The pod-security.kubernetes.io/enforce and pod-security.kubernetes.io/warn labels are applied, so a team can run one of its Namespaces stricter than the cluster's podSecurity floor without asking the platform team.

A level looser than the floor is rejected on the Application, when it is created or edited, with an error naming the level the cluster requires. Doing it this way means the person writing the Application sees the error, instead of the Namespace silently keeping its old labels.

Removing a Key

A key removed from managedNamespaceMetadata is not removed from the Namespace, it keeps the value it was last given. Remove it with kubectl when it is no longer wanted.

$ kubectl label ns a3 team-

For the same reason, a key should be managed either from the Application or with kubectl, not both: while the key is still listed in the Application, the next refresh restores the value from there.

Disabling

This is enabled for the whole cluster by default. It is turned off, and the refresh interval changed, in the platform-api Helm values file. Note that these values sit next to environmentConfig, not inside it.

zone:
  namespaceMetadata:
    enabled: true
    syncInterval: 10m

EnvironmentConfig

A Zone object only describes one team's boundary. The rules that apply to every Zone in the cluster are set once in the platform-apis EnvironmentConfig, so the platform team decides how strict the cluster is and developers do not have to configure it per Zone.

Setting	Default	What it does
podSecurity	baseline	The lowest security level a Namespace is allowed to run at.
granularEgress	false	Whether outgoing traffic rules are shared by the whole cluster or kept per Zone.
granularEgressExclude	[]	Zones that keep the shared, unrestricted behaviour.
granularNamespaceNetworkPolicy	false	Whether Namespaces inside the same Zone can talk to each other freely.
createAppsNamespace	true	Creates the <zone-name>-apps Namespace for ArgoCD Applications.
roleMapping	maintainer / observer / contributor	Which user groups get which role in every Zone. See Assigning Permissions.
appProject	see below	What ArgoCD is allowed to deploy into a Zone. See Tuning ArgoCDProject Permissions.
tags	{}	Cloud tags added to Zone resources such as NodeGroups. See Resource Tagging.

podSecurity

Kubernetes Pod Security Admission has three levels: privileged (no restrictions), baseline (blocks the well-known dangerous options) and restricted (also requires containers to run as a non-root user with no extra privileges).

This setting is the floor for the whole cluster. Every Namespace in a Zone is labelled with it, and a Namespace can only be changed to a stricter level, never a looser one:

• baseline (default) — Namespaces run at baseline, and a team may opt into restricted for its own Namespace.
• restricted — every Namespace must run at restricted. Containers that need root or extra Linux capabilities will not start. This is the recommended setting for production clusters, but existing workloads have to be checked first because some off-the-shelf images require root.
• privileged — no floor at all. Not recommended.

granularEgress

This setting decides who an outgoing connection is opened for when a team needs to reach an external address.

• false (default) — outgoing rules are shared by the whole cluster. When one Zone opens an external address, that address is reachable from every other Zone as well.
• true — each Zone gets its own set of outgoing rules. An address opened for Zone A stays closed for Zone B. Teams no longer inherit each other's external dependencies, and it becomes visible which Zone actually needs which external service.

This is not what stops your cluster from reaching the internet in the first place. Outgoing traffic leaving the cluster is normally restricted by a firewall in front of the cluster, and that is true whether granularEgress is on or off. What this setting changes is whether the rules inside Kubernetes are cluster-wide or per Zone.

Turning it on has a real cost: a proxy container is added to every pod in the Zone's Namespaces, so applications need to be restarted, and every external service a Zone uses (payment providers, mail services, package registries) has to be listed for that Zone.

An external host is opened with an Istio ServiceEntry object. Create it in one of the Zone's own Namespaces — that is what keeps it scoped to this Zone:

apiVersion: networking.istio.io/v1
kind: ServiceEntry
metadata:
  name: stripe-api
  namespace: example-zone-team
spec:
  hosts:
    - api.stripe.com
  ports:
    - number: 443
      name: https
      protocol: TLS
  resolution: DNS
  location: MESH_EXTERNAL

With granularEgress disabled there is no per-Zone scoping, so a ServiceEntry created anywhere in the cluster applies to every Zone.

granularEgressExclude

A list of Zone names that keep the shared, unrestricted behaviour even while granularEgress is enabled elsewhere. This is the escape hatch for a Zone that has not been migrated yet, or for a Zone whose workloads legitimately talk to many external services.

granularEgress: true
granularEgressExclude:
  - legacy-zone

granularNamespaceNetworkPolicy

Controls how open the network is inside a Zone.

• false (default) — all Namespaces belonging to the same Zone can reach each other. One team's services can call each other without any extra configuration.
• true — every Namespace is isolated on its own. Even two Namespaces in the same Zone must allow incoming traffic with a standard Kubernetes NetworkPolicy before they can communicate. Traffic within a single Namespace is always allowed.

In both cases, traffic between different Zones always has to be allowed explicitly, and outgoing traffic is unaffected by this setting.

A single Zone can override the cluster-wide value with spec.granularNamespaceNetworkPolicy:

apiVersion: tenancy.entigo.com/v1alpha1
kind: Zone
metadata:
  name: example-zone
spec:
  granularNamespaceNetworkPolicy: true

createAppsNamespace

Creates the <zone-name>-apps Namespace described in The ArgoCD App of Apps pattern. Set it to false if your teams do not use the app-of-apps pattern and you do not want the extra Namespace.

Infralib Configuration Example

steps:
  ...
  - name: apps
    type: argocd-apps
    modules:
    ...
    - name: platform-apis
      source: platform-apis
      inputs:
        platform-apis:
          zone:
            environmentConfig:
              podSecurity: baseline
              granularEgress: false
              granularEgressExclude: []
              granularNamespaceNetworkPolicy: false
              createAppsNamespace: true