Kubernetes CreateContainerConfigError and CreateContainerError

CreateContainerConfigError and CreateContainerError are two of the most prevalent Kubernetes errors found in cloud-native applications.

CreateContainerConfigError is an error happening when the configuration specified for a container in a Pod is not correct or is missing a vital part.

CreateContainerError is a problem happening at a later stage in the container creation flow. Kubernetes displays this error when it attempts to create the container in the Pod.

In this article, you will learn:

• What is Kubernetes CreateContainerConfigError?
• What is Kubernetes CreateContainerError?
• Kubernetes container creation flow
• Common causes for CreateContainerError and CreateConfigError
• How to troubleshoot both errors
• How to detect both errors in Prometheus

What is CreateContainerConfigError?

During the process to start a new container, Kubernetes first tries to generate the configuration for it. In fact, this is handled internally by calling a method called generateContainerConfig, which will try to retrieve:

• Container command and arguments
• Relevant persistent volumes for the container
• Relevant ConfigMaps for the container
• Relevant secrets for the container

Any problem in the elements above will result in a CreateContainerConfigError.

What is CreateContainerError?

Kubernetes throws a CreateContainerError when there’s a problem in the creation of the container, but unrelated with configuration, like a referenced volume not being accessible or a container name already being used.

Similar to other problems like CrashLoopBackOff, this article only covers the most common causes, but there are many others depending on your current application.

How you can detect CreateContainerConfigError and CreateContainerError

You can detect both errors by running kubectl get pods:

NAME  READY STATUS                     RESTARTS AGE

mypod 0/1   CreateContainerConfigError 0        11mCode language: YAML (yaml)

As you can see from this output:

• Pod is not ready: container has an error.
• There are no restarts: these two errors are not like CrashLoopBackOff, where automatic retrials are in place.

Kubernetes container creation flow

In order to understand CreateContainerError and CreateContainerConfligError, we need to first know the exact flow for container creation.

Kubernetes follows the next steps every time a new container needs to be started:

1. Pull the image.
2. Generate container configuration.
3. Precreate container.
4. Create container.
5. Pre-start container.
6. Start container.

As you can see, steps 2 and 4 are where a CreateContainerConfig and CreateContainerErorr might appear, respectively.

Common causes for CreateContainerError and CreateContainerConfigError

Not found ConfigMap

Kubernetes ConfigMaps are a key element to store non-confidential information to be used by Pods as key-value pairs.

When adding a ConfigMap reference in a Pod, you are effectively indicating that it should retrieve specific data from it. But, if a Pod references a non-existent ConfigMap, Kubernetes will return a CreateContainerConfigError.

Not found Secret

Secrets are a more secure manner to store sensitive information in Kubernetes. Remember, though, this is just raw data encoded in base64, so it’s not really encrypted, just obfuscated.

In case a Pod contains a reference to a non-existent secret, Kubelet will throw a CreateContainerConfigError, indicating that necessary data couldn’t be retrieved in order to form container config.

Container name already in use

While an unusual situation, in some cases a conflict might occur because a particular container name is already being used. Since every docker container should have a unique name, you will need to either delete the original or rename the new one being created.

How to troubleshoot CreateContainerError and CreateContainerConfigError

While the causes for an error in container creation might vary, you can always rely on the following methods to troubleshoot the problem that’s preventing the container from starting.

Describe Pods

With kubectl describe pod, you can retrieve the detailed information for the affected Pod and its containers:

Containers:
  mycontainer:
    Container ID:
    Image:          nginx
    Image ID:
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       CreateContainerConfigError
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:  3
---
Volumes:
  config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      myconfigmap
    Optional:  falseCode language: YAML (yaml)

Get logs from containers

Use kubectl logs to retrieve the log information from containers in the Pod. Note that for Pods with multiple containers, you need to use the –all-containers parameter:

Error from server (BadRequest): container "mycontainer" in pod "mypod" is waiting to start: CreateContainerConfigErrorCode language: YAML (yaml)

Check the events

You can also run kubectl get events to retrieve all the recent events happening in your Pods. Remember that the describe pods command also displays the particular events at the end.

How to detect CreateContainerConfigError and CreateContainerError in Prometheus

When using Prometheus + kube-state-metrics, you can quickly retrieve Pods that have containers with errors at creation or config steps:

kube_pod_container_status_waiting_reason{reason="CreateContainerConfigError"} > 0
kube_pod_container_status_waiting_reason{reason="CreateContainerError"} > 0Code language: YAML (yaml)

Other similar errors

Pending

Pending is a Pod status that appears when the Pod couldn’t even be started. Note that this happens at schedule time, so Kube-scheduler couldn’t find a node because of not enough resources or not proper taints/tolerations config.

ContainerCreating

ContainerCreating is another waiting status reason that can happen when the container could not be started because of a problem in the execution (e.g: No command specified)

Error from server (BadRequest): container "mycontainer" in pod "mypod" is waiting to start: ContainerCreating   Code language: YAML (yaml)

RunContainerError

This might be a similar situation to CreateContainerError, but note that this happens during the run step and not the container creation step.

A RunContainerError most likely points to problems happening at runtime, like attempts to write on a read-only volume.

CrashLoopBackOff

Remember that CrashLoopBackOff is not technically an error, but the waiting time grace period that is added between retrials.

Unlike CrashLoopBackOff events, CreateContainerError and CreateContainerConfigError won’t be retried automatically.

Conclusion

In this article, you have seen how both CreateContainerConfigError and CreateContainerError are important messages in the Kubernetes container creation process. Being able to detect them and understand at which stage they are happening is crucial for the day-to-day debugging of cloud-native services.

Also, it’s important to know the internal behavior of the Kubernetes container creation flow and what is errors might appear at each step.

Finally, CreateContainerConfigError and CreateContainerError might be mistaken with other different Kubernetes errors, but these two happen at container creation stage and they are not automatically retried.

---