Ubuntu Pastebin

Paste from Zic at Tue, 31 Jan 2017 13:06:43 +0000

Download as text 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
Note: I upgraded all the charms of the canonical-kubernetes to the latest version.

What I was doing (don't know if it's tied to the problem):
1) Deploying an etcd-operator cluster
2) After a short period, I notice that the /var/log/syslog of the current CDK kubernetes-master weight about 40Go, flooding of error about the new Kubetnete's "kind: EtcdCluster" added by the etcd-operator thirdpartyresource (kubectl get thirdpartyresource)
3) I deleted the etcd cluster deployed by etcd-operator, deleted the etcd-operator deployment, deleted the etcd-operator thirdpartyressource to stop this huge logs, it didn't change anything.
4) I restarted the kube-controller-manager (via systemctl restart kube-controller-manager) to see if it's make any changes. Nope.
5) I poweroff-ed the current flooding kubernetes-master to see if it switch to another master (I'm running 3 of it) will do the same kind of error.
6) Logs stopped to flood, but now I can't do anything with the cluster and I have this in my logs instead of etcd-operator flooding:

Jan 31 12:13:18 mth-k8smaster-01 kube-apiserver[1177]: I0131 12:13:18.007960    1177 jwt.go:239] Signature error (key 0): crypto/rsa: verification error
Jan 31 12:13:18 mth-k8smaster-01 kube-apiserver[1177]: E0131 12:13:18.008034    1177 handlers.go:58] Unable to authenticate the request due to an error: crypto/rsa: verification error
Jan 31 12:13:18 mth-k8smaster-01 kube-apiserver[1177]: I0131 12:13:18.008089    1177 panics.go:76] GET /api/v1/services?resourceVersion=0: (666.203µs) 401 [[kube-dns/v0.0.0 (linux/amd64) kubernetes/$Format] 10.128.74.217:46202]
Jan 31 12:13:18 mth-k8smaster-01 kube-apiserver[1177]: I0131 12:13:18.008739    1177 jwt.go:239] Signature error (key 0): crypto/rsa: verification error
Jan 31 12:13:18 mth-k8smaster-01 kube-apiserver[1177]: E0131 12:13:18.008768    1177 handlers.go:58] Unable to authenticate the request due to an error: crypto/rsa: verification error
Jan 31 12:13:18 mth-k8smaster-01 kube-apiserver[1177]: I0131 12:13:18.008811    1177 panics.go:76] GET /api/v1/configmaps?resourceVersion=0: (491.936µs) 401 [[nginx-ingress-controller/v0.0.0 (linux/amd64) kubernetes/$Format] 10.128.74.217:53348]
Jan 31 12:13:18 mth-k8smaster-01 kube-apiserver[1177]: I0131 12:13:18.008808    1177 jwt.go:239] Signature error (key 0): crypto/rsa: verification error
Jan 31 12:13:18 mth-k8smaster-01 kube-apiserver[1177]: E0131 12:13:18.008838    1177 handlers.go:58] Unable to authenticate the request due to an error: crypto/rsa: verification error
Jan 31 12:13:18 mth-k8smaster-01 kube-apiserver[1177]: I0131 12:13:18.008882    1177 panics.go:76] GET /api/v1/endpoints?resourceVersion=0: (657.736µs) 401 [[kube-dns/v0.0.0 (linux/amd64) kubernetes/$Format] 10.128.74.217:46202]
Jan 31 12:13:18 mth-k8smaster-01 kube-apiserver[1177]: I0131 12:13:18.009786    1177 jwt.go:239] Signature error (key 0): crypto/rsa: verification error
Jan 31 12:13:18 mth-k8smaster-01 kube-apiserver[1177]: E0131 12:13:18.009814    1177 handlers.go:58] Unable to authenticate the request due to an error: crypto/rsa: verification error
Jan 31 12:13:18 mth-k8smaster-01 kube-apiserver[1177]: I0131 12:13:18.009891    1177 panics.go:76] GET /apis/extensions/v1beta1/ingresses?resourceVersion=0: (487.465µs) 401 [[nginx-ingress-controller/v0.0.0 (linux/amd64) kubernetes/$Format] 10.128.74.217:53348]
Jan 31 12:13:18 mth-k8smaster-01 kube-apiserver[1177]: I0131 12:13:18.033513    1177 jwt.go:239] Signature error (key 0): crypto/rsa: verification error
Jan 31 12:13:18 mth-k8smaster-01 kube-apiserver[1177]: E0131 12:13:18.033565    1177 handlers.go:58] Unable to authenticate the request due to an error: crypto/rsa: verification error
Jan 31 12:13:18 mth-k8smaster-01 kube-apiserver[1177]: I0131 12:13:18.033618    1177 panics.go:76] GET /api/v1/services?resourceVersion=0: (531.08µs) 401 [[nginx-ingress-controller/v0.0.0 (linux/amd64) kubernetes/$Format] 10.128.74.216:38493]
Jan 31 12:13:18 mth-k8smaster-01 kube-apiserver[1177]: I0131 12:13:18.033676    1177 jwt.go:239] Signature error (key 0): crypto/rsa: verification error
Jan 31 12:13:18 mth-k8smaster-01 kube-apiserver[1177]: E0131 12:13:18.033706    1177 handlers.go:58] Unable to authenticate the request due to an error: crypto/rsa: verification error
Jan 31 12:13:18 mth-k8smaster-01 kube-apiserver[1177]: I0131 12:13:18.033964    1177 panics.go:76] GET /api/v1/services?resourceVersion=0: (499.67µs) 401 [[nginx-ingress-controller/v0.0.0 (linux/amd64) kubernetes/$Format] 10.128.74.217:53348]
Jan 31 12:13:18 mth-k8smaster-01 kube-apiserver[1177]: I0131 12:13:18.034698    1177 jwt.go:239] Signature error (key 0): crypto/rsa: verification error
Jan 31 12:13:18 mth-k8smaster-01 kube-apiserver[1177]: E0131 12:13:18.034726    1177 handlers.go:58] Unable to authenticate the request due to an error: crypto/rsa: verification error
Jan 31 12:13:18 mth-k8smaster-01 kube-apiserver[1177]: I0131 12:13:18.034769    1177 panics.go:76] GET /api/v1/secrets?resourceVersion=0: (519.723µs) 401 [[nginx-ingress-controller/v0.0.0 (linux/amd64) kubernetes/$Format] 10.128.74.216:38493]
Jan 31 12:13:18 mth-k8smaster-01 kube-apiserver[1177]: I0131 12:13:18.034988    1177 jwt.go:239] Signature error (key 0): crypto/rsa: verification error
Jan 31 12:13:18 mth-k8smaster-01 kube-apiserver[1177]: E0131 12:13:18.035025    1177 handlers.go:58] Unable to authenticate the request due to an error: crypto/rsa: verification error
Jan 31 12:13:18 mth-k8smaster-01 kube-apiserver[1177]: I0131 12:13:18.035069    1177 panics.go:76] GET /api/v1/endpoints?resourceVersion=0: (532.627µs) 401 [[nginx-ingress-controller/v0.0.0 (linux/amd64) kubernetes/$Format] 10.128.74.217:53348]
Jan 31 12:13:18 mth-k8smaster-01 kube-apiserver[1177]: I0131 12:13:18.036340    1177 panics.go:76] PUT /api/v1/nodes/mth-k8svitess-02/status: (19.85497ms) 200 [[kubelet/v1.5.2 (linux/amd64) kubernetes/08e0995] 10.128.74.201:48438]
Jan 31 12:13:18 mth-k8smaster-01 kube-apiserver[1177]: I0131 12:13:18.036634    1177 jwt.go:239] Signature error (key 0): crypto/rsa: verification error
Jan 31 12:13:18 mth-k8smaster-01 kube-apiserver[1177]: E0131 12:13:18.036664    1177 handlers.go:58] Unable to authenticate the request due to an error: crypto/rsa: verification error
Jan 31 12:13:18 mth-k8smaster-01 kube-apiserver[1177]: I0131 12:13:18.036711    1177 panics.go:76] GET /apis/extensions/v1beta1/ingresses?resourceVersion=0: (821.068µs) 401 [[nginx-ingress-controller/v0.0.0 (linux/amd64) kubernetes/$Format] 10.128.74.216:38493]
Jan 31 12:13:18 mth-k8smaster-01 kube-apiserver[1177]: I0131 12:13:18.037418    1177 jwt.go:239] Signature error (key 0): crypto/rsa: verification error
Jan 31 12:13:18 mth-k8smaster-01 kube-apiserver[1177]: E0131 12:13:18.037449    1177 handlers.go:58] Unable to authenticate the request due to an error: crypto/rsa: verification error
Jan 31 12:13:18 mth-k8smaster-01 kube-apiserver[1177]: I0131 12:13:18.037494    1177 panics.go:76] GET /api/v1/namespaces/kube-system/configmaps?fieldSelector=metadata.name%3Dkube-dns&resourceVersion=0: (507.226µs) 401 [[kube-dns/v0.0.0 (linux/amd64) kubernetes/$Format] 10.128.74.217:46202]
Jan 31 12:13:18 mth-k8smaster-01 kube-apiserver[1177]: I0131 12:13:18.040882    1177 jwt.go:239] Signature error (key 0): crypto/rsa: verification error
Jan 31 12:13:18 mth-k8smaster-01 kube-apiserver[1177]: E0131 12:13:18.040916    1177 handlers.go:58] Unable to authenticate the request due to an error: crypto/rsa: verification error
Download as text