Kubernetes ErrImagePull and ImagePullBackOff in detail

Pod statuses like ImagePullBackOff or ErrImagePull are common when working with containers.

ErrImagePull is an error happening when the image specified for a container can’t be retrieved or pulled.

ImagePullBackOff is the waiting grace period while the image pull is fixed.

In this article, we will take a look at:

• Container Images
• Pulling Images
• Image Pull Policy
• ErrImagePull
• Debugging ErrImagePull
• Monitoring Image Pull Errors
• Other Image Errors

Container Images

One of the greatest strengths of containerization is the ability to run any particular image in seconds. A container is a group of processes executing in isolation from the underlying system. A container image contains all the resources needed to run those processes: the binaries, libraries, and any necessary configuration.

A container registry is a repository for container images, where there are two basic actions:

• Push: upload an image so it’s available in the repo
• Pull: download an image to use it in a container

The docker CLI will be used in the examples for this article, but you can use any tool that implements the Open Container Initiative Distribution specs for all the container registry interactions.

Pulling images

Images can be defined by name. In addition, a particular version of an image can be labeled with a specific name or tag. It can also be identified by its digest, a hash of the content.

The tag latest refers to the most recent version of a given image.

Pull images by name

By only providing the name for the image, the image with tag latest will be pulled

docker pull nginx
kubectl run mypod nginx

Pull images by name + tag

If you don’t want to pull the latest image, you can provide a specific release tag:

docker pull nginx:1.23.1-alpine
kubectl run mypod nginx:1.23.1-alpine
Code language: JavaScript (javascript)

For more information, you can check this article about tag mutability.

Pull images by digest

A digest is sha256 hash of the actual image. You can pull images using this digest, then verify its authenticity and integrity with the downloaded file.

docker pull nginx@sha256:d164f755e525e8baee113987bdc70298da4c6f48fdc0bbd395817edf17cf7c2b
kubectl run mypod --image=nginx@sha256:d164f755e525e8baee113987bdc70298da4c6f48fdc0bbd395817edf17cf7c2b
Code language: JavaScript (javascript)

Image Pull Policy

Kubernetes features the ability to set an Image Pull Policy (imagePullPolicy field) for each container. Based on this, the way the kubelet retrieves the container image will differ.

There are three different values for imagePullPolicy:

• Always
• IfNotPresent
• Never

Always

With imagePullPolicy set to Always, kubelet will check the repository every time when pulling images for this container.

IfNotPresent

With imagePullPolicy set to IfNotPresent, kubelet will only pull images from the repository if it doesn’t exist in the node locally.

Never

With imagePullPolicy set to Never, kubelet will never try to pull images from the image registry. If there’s an image cached locally (pre-pulled), it will be used to start the container.

If the image is not present locally, Pod creation will fail with ErrImageNeverPull error.

Note that you can modify the entire image pull policy of your cluster by using the AlwaysPullImages admission controller.

Default Image Pull Policy

• If you omit the imagePullPolicy and the tag is latest, imagePullPolicy is set to Always.
• If you omit the imagePullPolicy and the tag for the image, imagePullPolicy is set to Always.
• If you omit the imagePullPolicy and the tag is set to a value different than latest, imagePullPolicy is set to IfNotPresent.

ErrImagePull

When Kubernetes tries to pull an image for a container in a Pod, things might go wrong. The status ErrImagePull is displayed when kubelet tried to start a container in the Pod, but something was wrong with the image specified in your Pod, Deployment, or ReplicaSet manifest.

Imagine that you are using kubectl to retrieve information about the Pods in your cluster:

$ kubectl get pods
NAME    READY   STATUS             RESTARTS   AGE
goodpod 1/1     Running            0          21h
mypod   0/1     ErrImagePull       0          4s
Code language: JavaScript (javascript)

Which means:

• Pod is not in READY status
• Status is ErrImagePull

Additionally, you can check the logs for containers in your Pod:

$ kubectl logs mypod --all-containers
Error from server (BadRequest): container "mycontainer" in pod "mypod" is waiting to start: trying and failing to pull image
Code language: JavaScript (javascript)

In this case, this is pointing to a 400 Error (BadRequest), since probably the image indicated is not available or doesn’t exist.

ImagePullBackOff

ImagePullBackOff is a Kubernetes waiting status, a grace period with an increased back-off between retries. After the back-off period expires, kubelet will try to pull the image again.

This is similar to the CrashLoopBackOff status, which is also a grace period between retries after an error in a container. Back-off time is increased each retry, up to a maximum of five minutes.

Note that ImagePullBackOff is not an error. As mentioned, it’s just a status reason that is caused by a problem when pulling the image.

$ kubectl get pods
NAME    READY   STATUS             RESTARTS   AGE
goodpod 1/1     Running            0          21h
mypod   0/1     ImagePullBackOff   0          84s
Code language: JavaScript (javascript)

Which means:

• Pod is not in READY status
• Status is ImagePullBackOff
• Unlike CrashLoopBackOff, there are no restarts (technically Pod hasn’t even started)

$ kubectl describe pod mypod
State:          Waiting
Reason:       ImagePullBackOff
...
Warning  Failed     3m57s (x4 over 5m28s)  kubelet            Error: ErrImagePull
Warning  Failed     3m42s (x6 over 5m28s)  kubelet            Error: ImagePullBackOff
Normal   BackOff    18s (x20 over 5m28s)   kubelet            Back-off pulling image "failed-image"
Code language: JavaScript (javascript)

Debugging ErrImagePull and ImagePullBackOff

There are several potential causes of why you might encounter an Image Pull Error. Here are some examples:

• Wrong image name
• Wrong image tag
• Wrong image digest
• Network problem or image repo not available
• Pulling from a private registry but not imagePullSecret was provided

This is just a list of possible causes, but it’s important to note that there might be many others based on your solution. The best course of action would be to check:

$ kubectl describe pod podname

$ kubect logs podname –all-containers

$ kubectl get events --field-selector involvedObject.name=podname
Code language: JavaScript (javascript)

In the following example you can see how to dig into the logs, where an image error is found.

Other image errors

ErrImageNeverPull

This error appears when kubelet fails to pull an image in the node and the imagePullPolicy is set to Never. In order to fix it, either change the Pull Policy to allow images to be pulled externally or add the correct image locally.

Pending

Remember that an ErrImagePull and the associated ImagePullBackOff may be different from a Pending status on your Pod.

Pending status, most likely, is the result of kube-scheduler not being able to assign your Pod to a working or eligible Node.

Monitoring Image Pull Errors in Prometheus

Using Prometheus and Kube State Metrics (KSM), we can easily track our Pods with containers in ImagePullBackOff or ErrImagePull statuses.

kube_pod_container_status_waiting_reason{reason="ErrImagePull"}
kube_pod_container_status_waiting_reason{reason="ImagePullBackOff"}
Code language: JavaScript (javascript)

In fact, these two metrics are complementary, as we can see in the following Prometheus queries:

The Pod is shifting between the waiting period in ImagePullBackOff and the image pull retrial returning an ErrImagePull.

Also, if you’re using containers with ImagePullPolicy set to Never, remember that you need to track the error as ErrImageNeverPull.

kube_pod_container_status_waiting_reason{reason="ErrImageNeverPull"}
Code language: JavaScript (javascript)

Conclusion

Container images are a great way to kickstart your cloud application needs. Thanks to them, you have access to thousands of curated applications that are ready to be started and scaled.

However, due to misconfiguration, misalignments, or repository problems, image errors might start appearing. A container can’t start properly if the image definition is malformed or there are errors on the setup.

Kubernetes provides a graceful period in case of an image pull error. This Image Pull Backoff is quite useful, as it gives you time to fix the problem in the image definition. But you need to be aware when this happens in your cluster and what does it mean each time.

---