Troubleshooting OpenEBS Install
General guidelines for troubleshooting
- Contact OpenEBS Community for support.
- Search for similar issues added in this troubleshootiung section.
- Search for any reported issues on StackOverflow under OpenEBS tag
Installation failed because insufficient user rights
iSCSI client is not setup on Nodes. Application Pod is in ContainerCreating state.
Why does OpenEBS provisioner pod restart continuously?
OpenEBS installation fails on Azure.
A multipath.conf file claims all SCSI devices in OpenShift
Installation failed because of insufficient user rights
OpenEBS installation can fail in some cloud platform with the following errors.
namespace "openebs" created
serviceaccount "openebs-maya-operator" created
clusterrolebinding.rbac.authorization.k8s.io "openebs-maya-operator" created
deployment.apps "maya-apiserver" created
service "maya-apiserver-service" created
deployment.apps "openebs-provisioner" created
deployment.apps "openebs-snapshot-operator" created
configmap "openebs-ndm-config" created
daemonset.extensions "openebs-ndm" created
Error from server (Forbidden): error when creating "https://raw.githubusercontent.com/openebs/openebs/v0.8.x/k8s/openebs-operator.yaml": clusterroles.rbac.authorization.k8s.io "openebs-maya-operator" is forbidden: attempt to grant extra privileges: [{[*] [*] [nodes] [] []} {[*] [*] [nodes/proxy] [] []} {[*] [*] [namespaces] [] []} {[*] [*] [services] [] []} {[*] [*] [pods] [] []} {[*] [*] [deployments] [] []} {[*] [*] [events] [] []} {[*] [*] [endpoints] [] []} {[*] [*] [configmaps] [] []} {[*] [*] [jobs] [] []} {[*] [*] [storageclasses] [] []} {[*] [*] [persistentvolumeclaims] [] []} {[*] [*] [persistentvolumes] [] []} {[get] [volumesnapshot.external-storage.k8s.io] [volumesnapshots] [] []} {[list] [volumesnapshot.external-storage.k8s.io] [volumesnapshots] [] []} {[watch] [volumesnapshot.external-storage.k8s.io] [volumesnapshots] [] []} {[create] [volumesnapshot.external-storage.k8s.io] [volumesnapshots] [] []} {[update] [volumesnapshot.external-storage.k8s.io] [volumesnapshots] [] []} {[patch] [volumesnapshot.external-storage.k8s.io] [volumesnapshots] [] []} {[delete] [volumesnapshot.external-storage.k8s.io] [volumesnapshots] [] []} {[get] [volumesnapshot.external-storage.k8s.io] [volumesnapshotdatas] [] []} {[list] [volumesnapshot.external-storage.k8s.io] [volumesnapshotdatas] [] []} {[watch] [volumesnapshot.external-storage.k8s.io] [volumesnapshotdatas] [] []} {[create] [volumesnapshot.external-storage.k8s.io] [volumesnapshotdatas] [] []} {[update] [volumesnapshot.external-storage.k8s.io] [volumesnapshotdatas] [] []} {[patch] [volumesnapshot.external-storage.k8s.io] [volumesnapshotdatas] [] []} {[delete] [volumesnapshot.external-storage.k8s.io] [volumesnapshotdatas] [] []} {[get] [apiextensions.k8s.io] [customresourcedefinitions] [] []} {[list] [apiextensions.k8s.io] [customresourcedefinitions] [] []} {[create] [apiextensions.k8s.io] [customresourcedefinitions] [] []} {[update] [apiextensions.k8s.io] [customresourcedefinitions] [] []} {[delete] [apiextensions.k8s.io] [customresourcedefinitions] [] []} {[*] [*] [disks] [] []} {[*] [*] [storagepoolclaims] [] []} {[*] [*] [storagepools] [] []} {[*] [*] [castemplates] [] []} {[*] [*] [runtasks] [] []} {[*] [*] [cstorpools] [] []} {[*] [*] [cstorvolumereplicas] [] []} {[*] [*] [cstorvolumes] [] []} {[get] [] [] [] [/metrics]}] user=&{user.name@mayadata.io [system:authenticated] map[user-assertion.cloud.google.com:[AKUJVpmzjjLCED3Vk2Q7wSjXV1gJs/pA3V9ZW53TOjO5bHOExEps6b2IZRjnru9YBKvaj3pgVu+34A0fKIlmLXLHOQdL/uFA4WbKbKfMdi1XC52CcL8gGTXn0/G509L844+OiM+mDJUftls7uIgOIRFAyk2QBixnYv22ybLtO2n8kcpou+ZcNFEVAD6z8Xy3ZLEp9pMd9WdQuttS506x5HIQSpDggWFf9T96yPc0CYmVEmkJm+O7uw==]]} ownerrules=[{[create] [authorization.k8s.io] [selfsubjectaccessreviews selfsubjectrulesreviews] [] []} {[get] [] [] [] [/api /api/* /apis /apis/* /healthz /openapi /openapi/* /swagger-2.0.0.pb-v1 /swagger.json /swaggerapi /swaggerapi/* /version /version/]}] ruleResolutionErrors=[]
Troubleshooting
You must enable RBAC before OpenEBS installation. This can be done from the kubernetes master console by executing the following command.
kubectl create clusterrolebinding <cluster_name>-admin-binding --clusterrole=cluster-admin --user=<user-registered-email-with-the-provider>
iSCSI client is not setup on Nodes. Pod is in ContainerCreating state.
After OpenEBS installation, you may proceed with application deployment which will provision OpenEBS volume. This may fail due to the following error. This can be found by describing the application pod.
MountVolume.WaitForAttach failed for volume “pvc-ea5b871b-32d3-11e9-9bf5-0a8e969eb15a” : open /sys/class/iscsi_host: no such file or directory -
Troubleshooting
This logs points that iscsid.service may not be enabled and running on your Nodes. You need to check if the service iscsid.service is running. If it is not running, you have to enable and start the service. You can refer prerequisites section and choose your platform to get the steps for enabling it.
Why does OpenEBS provisioner pod restart continuously?
The following output displays the pod status of all namespaces in which the OpenEBS provisioner is restarting continuously.
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE
default percona 0/1 Pending 0 36m <none> <none>
kube-system calico-etcd-tl4td 1/1 Running 0 1h 192.168.56.65 master
kube-system calico-kube-controllers-84fd4db7cd-jz9wt 1/1 Running 0 1h 192.168.56.65 master
kube-system calico-node-5rqdl 2/2 Running 0 1h 192.168.56.65 master
kube-system calico-node-zt95x 2/2 Running 0 1h 192.168.56.66 node
kube-system coredns-78fcdf6894-2plxb 1/1 Running 0 1h 192.168.219.65 master
kube-system coredns-78fcdf6894-gcjj7 1/1 Running 0 1h 192.168.219.66 master
kube-system etcd-master 1/1 Running 0 1h 192.168.56.65 master
kube-system kube-apiserver-master 1/1 Running 0 1h 192.168.56.65 master
kube-system kube-controller-manager-master 1/1 Running 0 1h 192.168.56.65 master
kube-system kube-proxy-9t98s 1/1 Running 0 1h 192.168.56.65 master
kube-system kube-proxy-mwk9f 1/1 Running 0 1h 192.168.56.66 node
kube-system kube-scheduler-master 1/1 Running 0 1h 192.168.56.65 master
openebs maya-apiserver-5598cf68ff-tndgm 1/1 Running 0 1h 192.168.167.131 node
openebs openebs-provisioner-776846bbff-rqfzr 0/1 CrashLoopBackOff 16 1h 192.168.167.129 node
openebs openebs-snapshot-operator-5b5f97dd7f-np79k 0/2 CrashLoopBackOff 32 1h 192.168.167.130 node
Troubleshooting
Perform the following steps to verify if the issue is due to misconfiguration while installing the network component.
- Check if your network related pods are running fine.
- Check if OpenEBS provisioner HTTPS requests are reaching the apiserver
- Use the latest version of network provider images.
- Try other network components such as Calico, kube-router etc. if you are not using any of these.
OpenEBS installation fails on Azure
On AKS, while installing OpenEBS using Helm, you may see the following error.
$ helm installstable/openebs --name openebs --namespace openebs
Error: release openebsfailed: clusterroles.rbac.authorization.k8s.io "openebs" isforbidden: attempt to grant extra privileges:[PolicyRule{Resources:["nodes"], APIGroups:["*"],Verbs:["get"]} PolicyRule{Resources:["nodes"],APIGroups:["*"], Verbs:["list"]}PolicyRule{Resources:["nodes"], APIGroups:["*"],Verbs:["watch"]} PolicyRule{Resources:["nodes/proxy"],APIGroups:["*"], Verbs:["get"]}PolicyRule{Resources:["nodes/proxy"], APIGroups:["*"],Verbs:["list"]} PolicyRule{Resources:["nodes/proxy"],APIGroups:["*"], Verbs:["watch"]}PolicyRule{Resources:["namespaces"], APIGroups:["*"],Verbs:["*"]} PolicyRule{Resources:["services"],APIGroups:["*"], Verbs:["*"]} PolicyRule{Resources:["pods"],APIGroups:["*"], Verbs:["*"]}PolicyRule{Resources:["deployments"], APIGroups:["*"],Verbs:["*"]} PolicyRule{Resources:["events"],APIGroups:["*"], Verbs:["*"]}PolicyRule{Resources:["endpoints"], APIGroups:["*"],Verbs:["*"]} PolicyRule{Resources:["persistentvolumes"],APIGroups:["*"], Verbs:["*"]} PolicyRule{Resources:["persistentvolumeclaims"],APIGroups:["*"], Verbs:["*"]}PolicyRule{Resources:["storageclasses"],APIGroups:["storage.k8s.io"], Verbs:["*"]}PolicyRule{Resources:["storagepools"], APIGroups:["*"],Verbs:["get"]} PolicyRule{Resources:["storagepools"], APIGroups:["*"],Verbs:["list"]} PolicyRule{NonResourceURLs:["/metrics"],Verbs:["get"]}] user=&{system:serviceaccount:kube-system:tiller6f3172cc-4a08-11e8-9af5-0a58ac1f1729 [system:serviceaccounts system:serviceaccounts:kube-systemsystem:authenticated] map[]} ownerrules=[]ruleResolutionErrors=[clusterroles.rbac.authorization.k8s.io"cluster-admin" not found]
Troubleshooting
You must enable RBAC on Azure before OpenEBS installation. For more details, see Prerequisites.
A multipath.conf file claims all SCSI devices in OpenShift
A multipath.conf file without either find_multipaths or a manual blacklist claims all SCSI devices.
Workaround:
Add the find_multipaths line to /etc/multipath.conf file similar to the following snippet.
defaults { user_friendly_names yes find_multipaths yes }Run
multipath -w /dev/sdccommand (replace the devname with your persistent devname).
See Also:
FAQs
Seek support or help
Latest release notes
Feedback
Was this page helpful?
Thanks for the feedback. Open an issue in the GitHub repo if you want to report a problem or suggest an improvement. Engage and get additional help on https://kubernetes.slack.com/messages/openebs/.