Kubernetes 1.23 - What's new? - New features and deprecations

Kubernetes 1.23 is about to be released, and it comes packed with novelties! Where do we begin?

This release brings 45 enhancements, on par with the 56 in Kubernetes 1.22 and the 50 in Kubernetes 1.21. Of those 45 enhancements, 11 are graduating to Stable, a whopping 15 are existing features that keep improving, and 19 are completely new.

The new features included in this version are generally small, but really welcomed, like the kubectl events command, support for OpenAPI v3, and gRPC probes. Also, it’s great to see how long-term projects are evolving, like the steady evolution of CSI Drivers, and Windows support.

However, the icing on the cake of this release are all the great features that are reaching the GA state. Grab onto something, the list is intense: CronJobs, IPv4/IPv6 dual-stack support, Ephemeral volumes, and the HPA API, to name a few. We are really hyped about this release!

Not to mention, watch out for all the deprecations and removals in this version!

There is plenty to talk about, so let’s get started with what’s new in Kubernetes 1.23.

Kubernetes 1.23 – Editor’s pick:

These are the features that look most exciting to us in this release (ymmv):

#2317 Delegate FSGroup to CSI Driver instead of Kubelet

There is a huge effort to move the CSI drivers out of the Kubernetes core, and achieve real modularity. On every new version we see progress in this front, which speaks on how the Kubernetes team has a clear vision and is able to keep building on it over several years.

Delegating more and more features to the CSI drivers will make it easier to add support for new platforms. This includes services like Portworx, Ceph RBD, and AWS EBS; but also new operating systems like Windows.

David de Torres – Manager of engineer at Sysdig

#1440 kubectl events

The new kubectl events respond to the fact that events have their own workflow, and the kubectl get events commands is not enough. This shows how the Kubernetes team is always looking on how to make life easier for cluster administrators.

Miguel Hernández – Security Content Engineer at Sysdig

#1790 Recovering from resize failures

Yet another tool that will make life way easier for all users, especially new adopters of Kubernetes. Having the chance to easily recover from common errors, like having trouble resizing a PVC, reduces frustration when using Kubernetes.

Jesús Ángel Samitier – DevOps Content Engineer at Sysdig

#2896 OpenAPI V3 + #2887 OpenAPI Enum Types

The extended support for OpenAPI v3 will bring fresh air to the tools that depend on the Kubernetes API. It will add small but transcendent details, like supporting Enum types, with proper validation. UIs will be able to display dropdowns for those fields! Small details, true, but those are the ones that will drive adoption by the less technical people.

Víctor Jiménez Cerrada – Content Engineering Manager at Sysdig

#2727 Add gRPC probe to Pod

Kubernetes uses probes to help site reliability engineers to increase the availability of your services. The addition of a probe to support gRPC, a protocol that is growing in popularity, unlocks a powerful tool for your developers to simplify development and improve resiliency of new cloud native applications.

Vicente J. Jiménez Miras – Security Content Engineer at Sysdig

Deprecations in Kubernetes 1.23

API and feature removals

A few beta APIs and features have been removed in Kubernetes 1.23, including:

Added the apiserver_longrunning_requests metric to replace apiserver_longrunning_gauge metric.
Controller-manager: The --port and --address flags have no effect and would be removed in v1.24.
kube-scheduler:
- metricsBindAddress and healthzBindAddress fields from kubescheduler.config.k8s.io/v1beta1 are expected to be empty.
- The --authorization-kubeconfig and --authentication-kubeconfig are expected to be correctly set to get authentication/authorization working.
- Liveness and readiness probes to kube-scheduler MUST use HTTPS now, and the default port has been changed to 10259.
- Applications that fetch metrics from kube-scheduler should use a dedicated service account.
Removed the scheduler_volume_scheduling_duration_seconds metric.
EgressSelectorConfiguration API: master is no longer a valid EgressSelection type.
Removed the --experimental-bootstrap-kubeconfig flag.
Deprecated the VolumeSubpath feature gate, cannot be disabled.
Removed the “update-cluster-status” in “kubeadm reset.”
The kubectl --dry-run option now only supports server, client, or none.
kubeadm: Deprecated the output/v1alpha1 API (replaced with output/v1alpha2).

You can check the full list of changes on the Kubernetes 1.23 release notes. Also, keep the deprecated API migration guide close for the future.

#2845 Deprecate klog specific flags in Kubernetes components

Stage: Deprecated
Feature group: instrumentation
Feature gate: NA

This enhancement is part of a bigger effort to simplify logging in its components. The first stone is removing some flags in klog.

The rationale behind this change comes from the decision of using the glog library for logging on the first days of Kubernetes. Later on, a more flexible solution was needed so this library was forked into klog, the one currently in use. Now, further flexibility is needed, so it’s time to do some cleanup to prepare the library for new features, like:

Improving performance.
Fulfill Kubernetes scalability requirements.
Remove glog features that are no longer needed.

#785 Scheduler component config API to v1beta3 and deprecate v1beta1

Stage: Graduating to Beta
Feature group: scheduling
Feature gate: NA

ComponentConfig is an ongoing effort to make component configuration more dynamic and directly reachable through the Kubernetes API.

Read more in our “Kubernetes 1.19 – What’s new?” article.

In this version, several plugins have been removed:

NodeResourcesLeastAllocated
NodeResourcesMostAllocated
RequestedToCapacityRatio
NodeLabel
ServiceAffinity
NodePreferAvoidPods

Also, scheduling policies are not supported. You should use Scheduler Configuration instead.

Check this documentation PR for more details.

Kubernetes 1.23 API

#2876 CRD Validation Expression Language

Stage: Graduating to Alpha
Feature group: api-machinery
Feature gate: CustomResourceValidationExpressions Default value: false

This enhancement implements a validation mechanism for Custom Resource Definitions (CRDs), as a complement to the existing one based on webhooks.

Two main benefits arise from including the validation instructions in the CRD itself:

All the information of a CRD can be in the same place (it’s self contained).
There is a possibility of significantly simplifying developments by not needing webhooks.

These validation rules use the Common Expression Language (CEL) and are included in CustomResourceDefinition schemas, using the x-kubernetes-validations extension.

…
openAPIV3Schema:
  type: object
  properties:
    spec:
      type: object
      x-kubernetes-validation-rules:
        - rule: "self.replicas <= self.maxReplicas"
          message: "replicas should be smaller than or equal to maxReplicas."
      properties:
        …
        replicas:
          type: integer
        maxReplicas:
          type: integer
      required:
        - replicas
        - maxReplicas

Would reject a request to create this custom resource:

apiVersion: "stable.example.com/v1"
kind: CronTab
metadata:
  name: my-new-cron-object
spec:
  replicas: 20
  maxReplicas: 10

As the value for replicas is bigger than maxReplicas.

Read more about this feature in the Kubernetes documentation.

#2885 Add Server-Side Unknown Field Validation KEP

Stage: Graduating to Alpha
Feature group: api-machinery
Feature gate: ServerSideFieldValidation Default value: false

Currently, you can use kubectl –validate=true to indicate that a request should fail if it specifies unknown fields on an object.

In this case, the validation is done in the client side (kubectl) instead of the server (kube-apiserver), which raises some concerns:

Each client must implement validation.
Rollout of validation bug fixes is slower, as it takes extra effort to update each client.

This enhancement summarizes the work to implement the validation on kube-apiserver, and will be implemented in several phases (first implement validation on the server, update clients to use server-side validation later). Read the plans and implementation details in the KEP.

#2887 OpenAPI Enum Types

Stage: Graduating to Alpha
Feature group: api-machinery
Feature gate: OpenAPIEnum Default value: false

This enhancement improves the Kubernetes OpenAPI generator, so it can recognize types marked with +enum and look for auto-detect possible values for an enum.

Currently, these fields are defined as strings:

type Protocol string
const (
	ProtocolTCP Protocol = "TCP"
	ProtocolUDP Protocol = "UDP"
	ProtocolSCTP Protocol = "SCTP"
)

By adding the +enum marker:

// +enum
type Protocol string

The Kubernetes OpenAPI generator can recognize that the accepted values are only TCP, UDP, and SCTP.

This information is available to other tools that, in the future, could display these fields as dropdowns instead of unrestricted text fields.

#2896 OpenAPI V3

Stage: Graduating to Alpha
Feature group: api-machinery
Feature gate: OpenApiv3 Default value: false

This feature adds support to kube-apiserver to serve Kubernetes and types as OpenAPI v3 objects.

Currently, they are available as OpenAPI v2 objects at the $cluster/openapi/v2 cluster endpoint. However, CRDs can be defined using the v3 of OpenAPI, which allows more complicated definitions, like enum types. When they are translated to be served on the v2 endpoint, some information is missing, like type definitions being transformed into “accept anything.”

When the OpenApiv3 feature gate is enabled, a new /openapi/v3/apis/{group}/{version} endpoint is available. It serves one schema per resource instead of aggregating everything into a single one.

#1040 Priority and Fairness for API Server Requests

Stage: Major Change to Beta
Feature group: api-machinery
Feature gate: APIPriorityAndFairness Default value: true

The APIPriorityAndFairness feature gate enables a new max-in-flight request handler in the API server. By defining different types of requests with FlowSchema objects and assigning them resources with RequestPriority objects, you can ensure the Kubernetes API server will be responsive for admin and maintenance tasks during high loads.

Read more in our “Kubernetes 1.18 – What’s new?” article. Significant changes have been made to this feature, but the team couldn’t finish all they wanted to graduate to Stable. Check more about the plans in the KEP.

Apps in Kubernetes 1.23

#1847 Auto remove PVCs created by StatefulSet

Stage: Graduating to Alpha
Feature group: apps
Feature gate: StatefulSetAutoDeletePVC Default value: false

A new, optional, .spec.persistentVolumeClaimRetentionPolicy field has been added to control if and how Persistent Volume Claims (PVCs) are deleted during the lifecycle of a StatefulSet.

apiVersion: apps/v1
kind: StatefulSet
…
spec:
  persistentVolumeClaimRetentionPolicy:
    whenDeleted: Retain
    whenScaled: Delete
…

You can decide whether to Retain or Delete PVCs, for any deleted Pod (whenDeleted), or just for Pod replicas being scaled down (whenScaled).

#2879 Add count of ready Pods in Job status

Stage: Graduating to Alpha
Feature group: apps
Feature gate: JobReadyPods Default value: false

This enhancement adds the Job.status.ready field that counts the number of Job Pods that have a Ready condition, similar to the existing active, succeeded, and failed fields.

The new ready field differs from the active one in that active includes Job Pods that are both in the Running or Pending phases. As these jobs can stay in the Pending phase for some time, the active field might give a false impression of progress to end users or other controllers.

The new ready field can reduce the need to listen to Pod updates in order to get a more correct picture of the current status.

#19 CronJobs (previously ScheduledJobs)

Stage: Major Change to Stable
Feature group: apps
Feature gate: CronJobControllerV2 Default value: true

Introduced in Kubernetes 1.4 and in beta since 1.8, CronJobs are finally on the road to becoming Stable.

CronJobs runs periodic tasks in a Kubernetes cluster, similar to cron on UNIX-like systems.

Read more in our “Kubernetes 1.20 – What’s new?” article.

#592 TTL after finish

Stage: Graduating to Stable
Feature group: apps
Feature gate: TTLAfterFinished Default value: true

This feature will clean up Finished or Complete jobs automatically. This is helpful, as these Jobs are usually no longer needed in the system and keeping them around will generate more load in the API server.

A ttlSecondsAfterFinished field can now be configured to have a controller clean them.

Read more in our What’s new in Kubernetes 1.12 article, and in the Jobs docs.

#2307 Job tracking without lingering Pods

Stage: Graduating to Beta
Feature group: apps
Feature gate: JobTrackingWithFinalizers Default value: true

With this enhancement, Jobs will be able to remove completed pods earlier, freeing resources in the cluster.

Read more in our “Kubernetes 1.22 – What’s new?” article.

#2599 Add minReadySeconds to Statefulsets

Stage: Graduating to Beta
Feature group: apps
Feature gate: StatefulSetMinReadySeconds Default value: true

This enhancement brings to StatefulSets the optional minReadySeconds field that is already available on Deployments, DaemonSets, ReplicasSets, and Replication Controllers.

Read more in our “Kubernetes 1.22 – What’s new?” article.

Kubernetes 1.23 Instrumentation

#2845 Deprecate klog specific flags in Kubernetes components

Stage: Graduating to Alpha
Feature group: instrumentation
Feature gate: NA

This enhancement is part of a bigger effort to simplify logging in its components. The first stone is removing some flags in klog.

The rationale behind this change comes from the decision of using the glog library for logging on the first days of Kubernetes. Later on, a more flexible solution was needed so this library was forked into klog, the one in use currently. Now, further flexibility is needed, so it’s time to do some cleanup to prepare the library for new features, like:

Improving performance.
Fulfill Kubernetes scalability requirements.
Remove glog features that are no longer needed.

#1602 Structured logging

Stage: Graduating to Beta
Feature group: instrumentation
Feature gate: NA

This enhancement defines a standard structure for Kubernetes log messages, like:

E1025 00:15:15.525108       1 controller_utils.go:114] "Failed to update pod status" err="timeout"

Read more in our “What’s new in Kubernetes 1.19?” article, or in the Kubernetes documentation.

Network in Kubernetes 1.23

#563 Add IPv4/IPv6 dual-stack support

Stage: Graduating to Stable
Feature group: network
Feature gate: IPv6DualStack Default value: true

This feature summarizes the work done to natively support dual-stack mode in your cluster, so you can assign both IPv4 and IPv6 addresses to a given pod.

Read more on the release for 1.20 in the What’s new in Kubernetes series.

#2365 Namespace Scoped Ingress Class Parameters

Stage: Graduating to Stable
Feature group: network
Feature gate: IngressClassNamespacedParams Default value: true

With this enhancement, you can now specify parameters for an IngressClass with a Namespace scope.

Read more in our “Kubernetes 1.21 – What’s new?” article.

#2433 Topology Aware Hints

Stage: Graduating to Beta
Feature group: network
Feature gate: TopologyAwareHints Default value: true

As we mentioned in our article for the Kubernetes 1.21 release, the #2433 Topology Aware Hints enhancement will replace the ServiceTopology feature gate that was introduced in Kubernetes 1.17.

Kubernetes 1.23 Nodes

#2371 cAdvisor-less, CRI-full Container and Pod stats

Stage: Graduating to Alpha
Feature group: node
Feature gate: PodAndContainerStatsFromCRI Default value: false

This enhancement summarizes the efforts to retrieve all the stats about running containers and pods from the Container Runtime Interface (CRI), removing the dependencies from cAdvisor.

Currently, the metrics served by the summary API and the /metrics/cadvisor endpoint are gathered mainly by cAdvidor, and completed by CRI. Having two sources removes clarity and duplicates efforts.

It was decided to improve CRI to provide all the required metrics. Some of the reasons not to use cAdvisor anymore are:

cAdvisor needs to be aware of the underlying container runtime. This invalidates the goal of CRI to achieve a fully-pluggable container runtime.
The container runtime knows better than cAdvisor about the container’s behavior.
cAdvisor does not support some runtimes, like virtual machines.
cAdvisor does not support Windows containers.

To learn further implementation details, head to the KEP.

#2712 PriorityClassValueBasedGracefulShutdown

Stage: Graduating to Alpha
Feature group: node
Feature gate: GracefulNodeShutdownBasedOnPodPriority Default value: false

From now on, you can provide more time to stop for high priority Pods when a note is graceful shutdown.

With the GracefulSutdown (#2000) enhancement, Kubelet tries to gracefully terminate the pods running in the node when shutting down.

This enhancement adds the possibility to provide Pods different times to stop, depending on its Pod priority class value:

shutdownGracePeriodByPodPriority:
  - priority: 100000
    shutdownGracePeriodSeconds: 10
  - priority: 10000
    shutdownGracePeriodSeconds: 180
  - priority: 1000
    shutdownGracePeriodSeconds: 120
  - priority: 0
    shutdownGracePeriodSeconds: 60

A Pod with priority of 100001 will get 10 seconds to stop, while a Pod with a priority of 1001 will get 120 seconds.

#2727 Add gRPC probe to Pod

Stage: Graduating to Alpha
Feature group: node
Feature gate: GRPCContainerProbe Default value: false

This enhancement allows configuring gRPC (HTTP/2 over TLS) liveness probes to Pods.

The liveness probes added in Kubernetes 1.16 allow to periodically check if an application is still alive.

Now, you’ll be able to configure probes that use the gRPC protocol, in addition to the existing HTTP and TCP options.

 readinessProbe:
      grpc:
        port: 9090
        service: my-service
      initialDelaySeconds: 5
      periodSeconds: 10

#2902 CPUManager policy option to distribute CPUs across NUMA nodes

Stage: Graduating to Alpha
Feature group: node
Feature gate: CPUManagerPolicyExperimentalOptions Default value: false

This enhancement adds distribute-cpus-across-numa as a new CPUManager policy option to ensure that CPU allocations are evenly distributed across NUMA nodes.

It builds on top of the “#2625 New CPU Manager Policies” enhancement.

#2625 Add options to reject non SMT-aligned workload

Stage: Graduating to Beta
Feature group: node
Feature gate: CPUManagerPolicyOptions Default value: false

This enhancement allows the CPU Manager to assign exclusive workloads to specific CPU cores. Isolating workloads per CPU core avoids context and cache switching, which is crucial for latency-sensitive applications.

Read more in our “Kubernetes 1.22 – What’s new?” article.

#277 Ephemeral Containers

Stage: Graduating to Beta
Feature group: node
Feature gate: EphemeralContainers Default value: true

Ephemeral containers are a great way to debug running pods. Although you can’t add regular containers to a pod after creation, you can run ephemeral containers with kubectl debug.

Read more in our “Kubernetes 1.16 – What’s new?” article.

#2040 Kubelet CRI Support

Stage: Graduating to Beta
Feature group: node
Feature gate: NA

The Container Runtime Interface (CRI) is a plugin interface that enables Kubelet to use a wide variety of container runtimes, without the need to recompile. This includes alternatives to Docker, like CRI-O or containerd.

Read more in our “Kubernetes 1.20 – What’s new?” article.

#2403 Extend podresources API to report allocatable resources

Stage: Graduating to Beta
Feature group: node
Feature gate: KubeletPodResourcesGetAllocatable Default value: true

This addition to the kubelet pod resources endpoint (pod resources API) will allow third-party consumers to learn about the compute resources allocated to a Pod. As a result, it will be easier to evaluate a node capacity.

Read more in our “Kubernetes 1.21 – What’s new?” article.

Scheduling in Kubernetes 1.23

#785 Scheduler component config API to v1beta3 and deprecate v1beta1

Stage: Graduating to Beta
Feature group: scheduling
Feature gate: NA

ComponentConfig is an ongoing effort to make component configuration more dynamic and directly reachable through the Kubernetes API.

Read more in our “Kubernetes 1.19 – What’s new?” article.

In this version, several plugins have been removed:

NodeResourcesLeastAllocated
NodeResourcesMostAllocated
RequestedToCapacityRatio
NodeLabel
ServiceAffinity
NodePreferAvoidPods

And Scheduling policies are not supported, you should use Scheduler Configuration instead.

Check this documentation PR for more details.

#2891 Simplified Multi-point plugin configuration for scheduler

Stage: Graduating to Beta
Feature group: scheduling
Feature gate: NA

This enhancement simplifies the configuration of scheduler plugins, allowing plugin registration for multiple extension points simultaneously.

So, instead of writing:

apiVersion: kubescheduler.config.k8s.io/v1beta3
kind: KubeSchedulerConfiguration
profiles:
  - schedulerName: non-multipoint-scheduler
    plugins:
      preScore:
        enabled:
        - name: MyPlugin
      score:
        enabled:
        - name: MyPlugin
      preFilter:
        enabled:
        - name: MyPlugin
      filter:
        enabled:
        - name: MyPlugin

You could replace preScore, score, preFilter and filter with multiPoint:

apiVersion: kubescheduler.config.k8s.io/v1beta3
kind: KubeSchedulerConfiguration
profiles:
  - schedulerName: multipoint-scheduler
    plugins:
      multiPoint:
        enabled:
        - name: MyPlugin

#2926 Allow updating node affinity constraints of suspended jobs

Stage: Graduating to Beta
Feature group: scheduling
Feature gate: JobMutableNodeSchedulingDirectives Default value: true

This enhancement builds up on “#2232 Stopped Job” from Kubernetes 1.21, which added the .spec.suspend field to pause Jobs.

In Kubernetes 1.23, it will be possible to update the node affinity, node selector, tolerations, labels, and annotations fields in a Job’s pod template before it starts. That way, you can influence where the Pods will run, like all in the same zone, or in nodes with the same GPU model.

Kubernetes 1.23 Storage

#1790 Recovering from resize failures

Stage: Graduating to Alpha
Feature group: storage
Feature gate: RecoverVolumeExpansionFailure Default value: false

Currently, if you expand a Persistent Volume Claim (PVC) to a size not supported by the underlying storage provider, you’ll get an error, and it’s not trivial to fix.

To fix this, this enhancement enables users to reduce a PVC’s size. And, to protect the system from abuse, a few measures were implemented:

The size reduction (reflected in pvc.Status.AllocatedResources) only takes place if a previous expansion request has failed.
The new requested size must be bigger than pvc.Status.Capacity.
The effective quota is the biggest of pvc.Status.Capacity or pvc.Status.AllocatedResources.

#2644 Always honor reclaim policy

Stage: Graduating to Alpha
Feature group: storage
Feature gate: HonorPVReclaimPolicy Default value: false

This enhancement fixes an issue where some Bound Persistent Volume (PV) – Persistent Volume Claim (PVC) pairs, the ordering of PV-PVC deletion determines whether the PV delete reclaim policy is honored.

Grosso modo, what happens is that deleting a Bound PV:

A deletionTimestamp is added to the PV, triggering an update.
If this update is processed before the PVC is deleted, the deletion is ignored.

Check the KEP for a detailed explanation on how this process works, and the implementation details on the fix.

#2317 Delegate FSGroup to CSI Driver instead of Kubelet

Stage: Graduating to Alpha
Feature group: storage
Feature gate: DelegateFSGroupToCSIDriver Default value: false

This enhancement proposes providing the CSI driver with the fsgroup of the pods as an explicit field, so the CSI driver can be the one applying this natively on mount time.

Read more in our “Kubernetes 1.22 – What’s new?” article.

#2589 Portworx file in-tree to CSI driver migration

Stage: Graduating to Alpha
Feature group: storage
Feature gate: CSIMigrationPortworx Default value: false

All plugin operations for the Portworx Container Storage Interface (CSI) driver are now redirected to the out-of-tree ‘pxd.portworx.com’ driver.

This enhancement is part of a bigger effort of migrating CSI drivers outside the Kubernetes (from in-tree to out-of-tree). See #178, #625, #1490, #1491.

#2923 Ceph RBD in-tree provisioner to CSI driver migration

Stage: Graduating to Alpha
Feature group: storage
Feature gate: CSIMigrationRBD Default value: false

All plugin operations for the RBD Container Storage Interface (CSI) driver are now redirected to the out-of-tree ‘rbd.csi.ceph.com’ driver.

This enhancement is part of a bigger effort of migrating CSI drivers outside the Kubernetes (from in-tree to out-of-tree). See #178, #625, #1490, #1491.

#1487 AWS EBS in-tree to CSI driver migration

Stage: Graduating to Stable
Feature group: storage
Feature gate: CSIMigrationAWS Default value: false

All plugin operations for the AWS EBS Container Storage Interface (CSI) driver are now redirected to the out-of-tree ‘ebs.csi.aws.com’ driver.

This enhancement is part of a bigger effort of migrating CSI drivers outside the Kubernetes (from in-tree to out-of-tree). See #178, #625, #1490, #1491.

#695 Non-recursive volume ownership (FSGroup)

Stage: Graduating to Stable
Feature group: storage
Feature gate: ConfigurableFSGroupPolicy Default value: true

Before a volume is bind-mounted inside a container, all of its file permissions are changed to the provided fsGroup value. This ends up being a slow process on very large volumes, and also breaks some permission sensitive applications, like databases.

The FSGroupChangePolicy field, introduced in Kubernetes 1.18, can alter this behavior. If set to Always, it will maintain the current behavior. However, when set to OnRootMismatch, it will only change the volume permissions if the top level directory does not match the expected fsGroup value.

#1682 Config FSGroup Policy in CSI Driver object

Stage: Graduating to Stable
Feature group: storage
Feature gate: CSIVolumeFSGroupPolicy Default value: true

Not all volumes support the fsGroup operations mentioned in the previous enhancement (#695), like NFS. This can result in errors reported to the user.

This enhancement adds a new field called CSIDriver.Spec.SupportsFSGroup that allows the driver to define if it supports volume ownership modifications via fsGroup.

Read more in our “Kubernetes 1.19 – What’s new?” article.

#1698 Generic inline ephemeral volumes

Stage: Graduating to Stable
Feature group: storage
Feature gate: GenericEphemeralVolume Default value: true

This enhancement provides a simple API to define inline ephemeral volumes that will work with any storage driver that supports dynamic provisioning.

Read more in our “Kubernetes 1.19 – What’s new?” article.

Windows support in Kubernetes 1.23

#2802 Identify Pod’s OS during API Server admission

Stage: Graduating to Alpha
Feature group: windows
Feature gate: IdentifyPodOS Default value: false

This enhancement adds the OS field to PodSpec, so you can define what operating system a pod should be running on.

That way, Kubernetes have better context to:

Reject Pods that shouldn’t run in a node, implemented in kubelet.
Schedule the pods in the appropriate nodes, to be implemented on the scheduler.
Apply the appropriate security constraints, to be implemented by admission plugins.

#1981 Support for Windows privileged containers

Stage: Graduating to Beta
Feature group: windows
Feature gate: WindowsHostProcessContainers Default value: true

This enhancement brings the privileged containers feature available in Linux to Windows hosts.

Privileged containers have access to the host, as if they were running directly on it. Although they are not recommended for most of the workloads, they are quite useful for administration, security, and monitoring purposes.

Read more in our “Kubernetes 1.22 – What’s new?” article.

Other

#1440 kubectl events

Stage: Graduating to Alpha
Feature group: cli
Feature gate: NA

A new kubectl events command is available that will enhance the current functionality of kubectl get events.

There is a need to add functionality to the current kubectl get events command. However, expanding it presents some challenges. After all, any change affects the entire kubectl get command.

The new kubectl events command will present an experience tailored to work with events. In addition to supporting all the current functionality of the kubectl get events command, this first version solves an issue where kubectl get events --watch wouldn’t sort events properly.

#2915 kubeadm: replace the legacy kubelet-config-x.y naming

Stage: Graduating to Alpha
Feature group: cluster-lifecycle
Feature gate: UnversionedKubeletConfigMap Default value: false

This enhancement changes how the kubelet-config-x.y ConfigMaps and RBAC rules are named, removing the -x.y versioning reference (example: -1.23).

The reasons behind this change:

The x.y versioning reference is taken from the control plane, but it would make more sense to take it from kubelet.
During upgrades, new ConfigMap and RBAC rules are created with the new versioning, but the old ones are not removed.

It is a goal to remove the complication that this versioning adds, keeping kube-system/kubelet-config as the source of truth.

#2702 Graduate HPA API to GA

Stage: Graduating to Stable
Feature group: autoscaling
Feature gate: NA

The v2 of the Horizontal Pod Autoscaler (HPA) API has been around since November, 2016. Five years ago! Although it is pretty much stable, there are some minor issues before it can be officially considered as so.

This enhancement summarizes the work to take this API to the finish line, including:

Complete the end-to-end tests.
Deprecate old versions of the API.
Container Resource Targets: Rename Resource to PodResource.
Rename the value Disabled with ScalingDisabled.
Behavior Rename: The values for behavior select policy values changes from Min to MinChange and Max to MaxChange.

If you are interested in HPA, check out this “How to configure Keda to deploy a Kubernetes HPA that uses Prometheus metrics” article.

#1933 Defend against logging secrets via static analysis

Stage: Graduating to Stable
Feature group: security
Feature gate: NA

Static analysis performed while preparing #1753 to provide insight on where sensitive data is used.

Read more in our “Kubernetes 1.20 – What’s new?” article.

#2420 Reduce Kubernetes Build Maintenance

Stage: Graduating to Stable
Feature group: testing
Feature gate: NA

This enhancement summarizes the work done to move all build scripts to make build, and remove bazel build.

Read more in our “Kubernetes 1.21 – What’s new?” article.

#2464 kubetest2 CI migration

Stage: Graduating to Beta
Feature group: testing
Feature gate: NA

This enhancement summarizes all the work done to update all the usages of kubetest, the interface for launching and running end-to-end tests, to kubetest2. This also includes updating a series of scripts, called scenarios, that are used for running common use cases.

#2579 PodSecurity admission (PodSecurityPolicy replacement)

Stage: Graduating to Beta
Feature group: auth
Feature gate: PodSecurity Default value: true

A new PodSecurity admission controller is now enabled by default to replace the Pod Security Policies deprecated in Kubernetes 1.21.

Read more in our “Kubernetes 1.22 – What’s new?” article.

That’s all for Kubernetes 1.23, folks! Exciting as always; get ready to upgrade your clusters if you are intending to use any of these features.

If you liked this, you might want to check out our previous ‘What’s new in Kubernetes’ editions:

Get involved in the Kubernetes community:

Visit the project homepage.
Check out the Kubernetes project on GitHub.
Get involved with the Kubernetes community.
Meet the maintainers on the Kubernetes Slack.
Follow @KubernetesIO on Twitter.

And if you enjoy keeping up to date with the Kubernetes ecosystem, subscribe to our container newsletter, a monthly email with the coolest stuff happening in the cloud-native ecosystem.

Kubernetes 1.23 – What’s new?

Kubernetes 1.23 – Editor’s pick:

#2317 Delegate FSGroup to CSI Driver instead of Kubelet

#1440 kubectl events

#1790 Recovering from resize failures

#2896 OpenAPI V3 + #2887 OpenAPI Enum Types

#2727 Add gRPC probe to Pod

Deprecations in Kubernetes 1.23

API and feature removals

#2845 Deprecate klog specific flags in Kubernetes components

#785 Scheduler component config API to v1beta3 and deprecate v1beta1

Kubernetes 1.23 API

#2876 CRD Validation Expression Language

#2885 Add Server-Side Unknown Field Validation KEP

#2887 OpenAPI Enum Types

#2896 OpenAPI V3

#1040 Priority and Fairness for API Server Requests

Apps in Kubernetes 1.23

#1847 Auto remove PVCs created by StatefulSet

#2879 Add count of ready Pods in Job status

#19 CronJobs (previously ScheduledJobs)

#592 TTL after finish

#2307 Job tracking without lingering Pods

#2599 Add minReadySeconds to Statefulsets

Kubernetes 1.23 Instrumentation

#2845 Deprecate klog specific flags in Kubernetes components

#1602 Structured logging

Network in Kubernetes 1.23

#563 Add IPv4/IPv6 dual-stack support

#2365 Namespace Scoped Ingress Class Parameters

#2433 Topology Aware Hints

Kubernetes 1.23 Nodes

#2371 cAdvisor-less, CRI-full Container and Pod stats

#2712 PriorityClassValueBasedGracefulShutdown

#2727 Add gRPC probe to Pod

#2902 CPUManager policy option to distribute CPUs across NUMA nodes

#2625 Add options to reject non SMT-aligned workload

#277 Ephemeral Containers

#2040 Kubelet CRI Support

#2403 Extend podresources API to report allocatable resources

Scheduling in Kubernetes 1.23

#785 Scheduler component config API to v1beta3 and deprecate v1beta1

#2891 Simplified Multi-point plugin configuration for scheduler

#2926 Allow updating node affinity constraints of suspended jobs

Kubernetes 1.23 Storage

#1790 Recovering from resize failures

#2644 Always honor reclaim policy

#2317 Delegate FSGroup to CSI Driver instead of Kubelet

#2589 Portworx file in-tree to CSI driver migration

#2923 Ceph RBD in-tree provisioner to CSI driver migration

#1487 AWS EBS in-tree to CSI driver migration

#695 Non-recursive volume ownership (FSGroup)

#1682 Config FSGroup Policy in CSI Driver object

#1698 Generic inline ephemeral volumes

Windows support in Kubernetes 1.23

#2802 Identify Pod’s OS during API Server admission

#1981 Support for Windows privileged containers

Other

#1440 kubectl events

#2915 kubeadm: replace the legacy kubelet-config-x.y naming

#2702 Graduate HPA API to GA

#1933 Defend against logging secrets via static analysis

#2420 Reduce Kubernetes Build Maintenance

#2464 kubetest2 CI migration

#2579 PodSecurity admission (PodSecurityPolicy replacement)