Kubernetes ErrImagePull and ImagePullBackOff in detail
content:Container Images Pulling Images Image Pull Policy ErrImagePull ImagePullBackOff Debugging ErrImagePull Monitoring Image Pull Errors in Prometheus Other Image Errors Conclusion
Pod statuses like ImagePullBackOff or ErrImagePull are common when working with containers.
ErrImagePull is an error happening when the image specified for a container can’t be retrieved or pulled.
ImagePullBackOff is the waiting grace period while the image pull is fixed.
In this article, we will take a look at:
- Container Images
- Pulling Images
- Image Pull Policy
- Debugging ErrImagePull
- Monitoring Image Pull Errors
- Other Image Errors
One of the greatest strengths of containerization is the ability to run any particular image in seconds. A container is a group of processes executing in isolation from the underlying system. A container image contains all the resources needed to run those processes: the binaries, libraries, and any necessary configuration.
A container registry is a repository for container images, where there are two basic actions:
- Push: upload an image so it’s available in the repo
- Pull: download an image to use it in a container
Images can be defined by name. In addition, a particular version of an image can be labeled with a specific name or tag. It can also be identified by its digest, a hash of the content.
latest refers to the most recent version of a given image.
Pull images by name
By only providing the name for the image, the image with tag
latest will be pulled
docker pull nginx kubectl run mypod nginx
Pull images by name + tag
If you don’t want to pull the
latest image, you can provide a specific release tag:
docker pull nginx:1.23.1-alpine kubectl run mypod nginx:1.23.1-alpine
For more information, you can check this article about tag mutability.
Pull images by digest
A digest is sha256 hash of the actual image. You can pull images using this digest, then verify its authenticity and integrity with the downloaded file.
docker pull [email protected]:d164f755e525e8baee113987bdc70298da4c6f48fdc0bbd395817edf17cf7c2b kubectl run mypod [email protected]:d164f755e525e8baee113987bdc70298da4c6f48fdc0bbd395817edf17cf7c2b
Image Pull Policy
Kubernetes features the ability to set an Image Pull Policy (imagePullPolicy field) for each container. Based on this, the way the
kubelet retrieves the container image will differ.
There are three different values for imagePullPolicy:
With imagePullPolicy set to Always, kubelet will check the repository every time when pulling images for this container.
With imagePullPolicy set to IfNotPresent, kubelet will only pull images from the repository if it doesn’t exist in the node locally.
With imagePullPolicy set to Never, kubelet will never try to pull images from the image registry. If there’s an image cached locally (pre-pulled), it will be used to start the container.
If the image is not present locally, Pod creation will fail with
Note that you can modify the entire image pull policy of your cluster by using the AlwaysPullImages admission controller.
Default Image Pull Policy
- If you omit the imagePullPolicy and the tag is
latest, imagePullPolicy is set to
- If you omit the imagePullPolicy and the tag for the image, imagePullPolicy is set to
- If you omit the imagePullPolicy and the tag is set to a value different than
latest, imagePullPolicy is set to
When Kubernetes tries to pull an image for a container in a Pod, things might go wrong. The status ErrImagePull is displayed when
kubelet tried to start a container in the Pod, but something was wrong with the image specified in your Pod, Deployment, or ReplicaSet manifest.
Imagine that you are using kubectl to retrieve information about the Pods in your cluster:
$ kubectl get pods NAME READY STATUS RESTARTS AGE goodpod 1/1 Running 0 21h mypod 0/1 ErrImagePull 0 4s
- Pod is not in
- Status is
Additionally, you can check the logs for containers in your Pod:
$ kubectl logs mypod --all-containers Error from server (BadRequest): container "mycontainer" in pod "mypod" is waiting to start: trying and failing to pull image
In this case, this is pointing to a 400 Error (BadRequest), since probably the image indicated is not available or doesn’t exist.
ImagePullBackOff is a Kubernetes waiting status, a grace period with an increased back-off between retries. After the back-off period expires, kubelet will try to pull the image again.
This is similar to the CrashLoopBackOff status, which is also a grace period between retries after an error in a container. Back-off time is increased each retry, up to a maximum of five minutes.
Note that ImagePullBackOff is not an error. As mentioned, it’s just a status reason that is caused by a problem when pulling the image.
$ kubectl get pods NAME READY STATUS RESTARTS AGE goodpod 1/1 Running 0 21h mypod 0/1 ImagePullBackOff 0 84s
- Pod is not in
- Status is
- Unlike CrashLoopBackOff, there are no restarts (technically Pod hasn’t even started)
$ kubectl describe pod mypod State: Waiting Reason: ImagePullBackOff ... Warning Failed 3m57s (x4 over 5m28s) kubelet Error: ErrImagePull Warning Failed 3m42s (x6 over 5m28s) kubelet Error: ImagePullBackOff Normal BackOff 18s (x20 over 5m28s) kubelet Back-off pulling image "failed-image"
Debugging ErrImagePull and ImagePullBackOff
There are several potential causes of why you might encounter an Image Pull Error. Here are some examples:
- Wrong image name
- Wrong image tag
- Wrong image digest
- Network problem or image repo not available
- Pulling from a private registry but not imagePullSecret was provided
This is just a list of possible causes, but it’s important to note that there might be many others based on your solution. The best course of action would be to check:
$ kubectl describe pod podname $ kubect logs podname –all-containers $ kubectl get events --field-selector involvedObject.name=podname
In the following example you can see how to dig into the logs, where an image error is found.
Other image errors
This error appears when kubelet fails to pull an image in the node and the imagePullPolicy is set to Never. In order to fix it, either change the Pull Policy to allow images to be pulled externally or add the correct image locally.
Remember that an ErrImagePull and the associated ImagePullBackOff may be different from a Pending status on your Pod.
Pending status, most likely, is the result of kube-scheduler not being able to assign your Pod to a working or eligible Node.
Monitoring Image Pull Errors in Prometheus
Using Prometheus and Kube State Metrics (KSM), we can easily track our Pods with containers in ImagePullBackOff or ErrImagePull statuses.
In fact, these two metrics are complementary, as we can see in the following Prometheus queries:
The Pod is shifting between the waiting period in ImagePullBackOff and the image pull retrial returning an ErrImagePull.
Also, if you’re using containers with
ImagePullPolicy set to
Never, remember that you need to track the error as
Container images are a great way to kickstart your cloud application needs. Thanks to them, you have access to thousands of curated applications that are ready to be started and scaled.
However, due to misconfiguration, misalignments, or repository problems, image errors might start appearing. A container can’t start properly if the image definition is malformed or there are errors on the setup.
Kubernetes provides a graceful period in case of an image pull error. This Image Pull Backoff is quite useful, as it gives you time to fix the problem in the image definition. But you need to be aware when this happens in your cluster and what does it mean each time.
Troubleshoot Image Pull Errors with Sysdig Monitor
With the Advisor feature of Sysdig Monitor, you can easily review Image Pull Errors happening in your Kubernetes cluster.
Sysdig Advisor accelerates mean time to resolution (MTTR) with live logs, performance data, and suggested remediation steps. It’s the easy button for Kubernetes troubleshooting!