1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168 | unit-rabbitmq-server-6: 23:25:26 DEBUG unit.rabbitmq-server/6.juju-log cluster:110: Hardening function 'config_changed'
unit-rabbitmq-server-6: 23:25:26 DEBUG unit.rabbitmq-server/6.juju-log cluster:110: Hardening function 'update_status'
unit-rabbitmq-server-6: 23:25:26 INFO unit.rabbitmq-server/6.juju-log cluster:110: Unknown hook cluster-relation-broken - skipping.
unit-rabbitmq-server-6: 23:25:26 INFO unit.rabbitmq-server/6.cluster-relation-broken error: no unit id specified
unit-rabbitmq-server-6: 23:25:27 INFO unit.rabbitmq-server/6.juju-log cluster:110: getting local nodename for ip address: 10.5.100.55
unit-rabbitmq-server-6: 23:25:27 INFO unit.rabbitmq-server/6.juju-log cluster:110: local nodename: juju-d020f9-8-lxd-2
unit-rabbitmq-server-6: 23:25:27 DEBUG unit.rabbitmq-server/6.juju-log cluster:110: Waiting for rabbitmq app to start: /var/lib/rabbitmq/mnesia/rabbit@juju-d020f9-8-lxd-2.pid
unit-rabbitmq-server-6: 23:25:27 DEBUG unit.rabbitmq-server/6.juju-log cluster:110: Running ['timeout', '180', '/usr/sbin/rabbitmqctl', 'wait', '/var/lib/rabbitmq/mnesia/rabbit@juju-d020f9-8-lxd-2.pid']
unit-rabbitmq-server-6: 23:25:27 INFO unit.rabbitmq-server/6.cluster-relation-broken Waiting for 'rabbit@juju-d020f9-8-lxd-2' ...
unit-rabbitmq-server-6: 23:25:27 INFO unit.rabbitmq-server/6.cluster-relation-broken pid is 14488 ...
unit-rabbitmq-server-6: 23:24:55 INFO juju.worker.leadership rabbitmq-server/6 will renew rabbitmq-server leadership at 2017-01-24 23:25:25.345076751 +0000 UTC
unit-rabbitmq-server-6: 23:25:25 INFO juju.worker.leadership rabbitmq-server/6 will renew rabbitmq-server leadership at 2017-01-24 23:25:55.345289278 +0000 UTC
unit-rabbitmq-server-6: 23:25:25 INFO juju.worker.uniter awaiting error resolution for "relation-broken" hook
unit-rabbitmq-server-6: 23:25:25 DEBUG unit.rabbitmq-server/6.juju-log cluster:110: Hardening function 'install'
unit-rabbitmq-server-6: 23:25:25 DEBUG unit.rabbitmq-server/6.juju-log cluster:110: Hardening function 'upgrade_charm'
unit-rabbitmq-server-6: 23:25:55 INFO juju.worker.leadership rabbitmq-server/6 will renew rabbitmq-server leadership at 2017-01-24 23:26:25.345448902 +0000 UTC
unit-rabbitmq-server-6: 23:26:25 INFO juju.worker.leadership rabbitmq-server/6 will renew rabbitmq-server leadership at 2017-01-24 23:26:55.345689938 +0000 UTC
unit-rabbitmq-server-6: 23:26:55 INFO juju.worker.leadership rabbitmq-server/6 will renew rabbitmq-server leadership at 2017-01-24 23:27:25.34581926 +0000 UTC
unit-rabbitmq-server-6: 23:27:25 INFO juju.worker.leadership rabbitmq-server/6 will renew rabbitmq-server leadership at 2017-01-24 23:27:55.34596583 +0000 UTC
unit-rabbitmq-server-6: 23:27:55 INFO juju.worker.leadership rabbitmq-server/6 will renew rabbitmq-server leadership at 2017-01-24 23:28:25.346130343 +0000 UTC
unit-rabbitmq-server-7: 23:28:05 INFO juju.worker.uniter no operations in progress; waiting for changes
unit-rabbitmq-server-8: 23:28:05 INFO juju.worker.uniter no operations in progress; waiting for changes
unit-rabbitmq-server-7: 23:28:06 DEBUG unit.rabbitmq-server/7.juju-log Hardening function 'install'
unit-rabbitmq-server-7: 23:28:06 DEBUG unit.rabbitmq-server/7.juju-log Hardening function 'upgrade_charm'
unit-rabbitmq-server-8: 23:28:06 DEBUG unit.rabbitmq-server/8.juju-log Hardening function 'install'
unit-rabbitmq-server-7: 23:28:06 DEBUG unit.rabbitmq-server/7.juju-log Hardening function 'config_changed'
unit-rabbitmq-server-8: 23:28:06 DEBUG unit.rabbitmq-server/8.juju-log Hardening function 'upgrade_charm'
unit-rabbitmq-server-7: 23:28:06 DEBUG unit.rabbitmq-server/7.juju-log Hardening function 'update_status'
unit-rabbitmq-server-8: 23:28:06 DEBUG unit.rabbitmq-server/8.juju-log Hardening function 'config_changed'
unit-rabbitmq-server-8: 23:28:06 DEBUG unit.rabbitmq-server/8.juju-log Hardening function 'update_status'
unit-rabbitmq-server-7: 23:28:06 DEBUG unit.rabbitmq-server/7.juju-log No hardening applied to 'update_status'
unit-rabbitmq-server-7: 23:28:06 INFO unit.rabbitmq-server/7.juju-log Updating status.
unit-rabbitmq-server-8: 23:28:06 DEBUG unit.rabbitmq-server/8.juju-log No hardening applied to 'update_status'
unit-rabbitmq-server-8: 23:28:06 INFO unit.rabbitmq-server/8.juju-log Updating status.
unit-rabbitmq-server-8: 23:28:11 INFO juju.worker.uniter.operation ran "update-status" hook
unit-rabbitmq-server-8: 23:28:11 INFO juju.worker.uniter no operations in progress; waiting for changes
unit-rabbitmq-server-6: 23:28:25 INFO juju.worker.leadership rabbitmq-server/6 will renew rabbitmq-server leadership at 2017-01-24 23:28:55.346356395 +0000 UTC
unit-rabbitmq-server-6: 23:28:27 INFO unit.rabbitmq-server/6.cluster-relation-broken
unit-rabbitmq-server-6: 23:28:27 INFO unit.rabbitmq-server/6.cluster-relation-broken Session terminated, terminating shell... ...terminated.
unit-rabbitmq-server-6: 23:28:27 INFO unit.rabbitmq-server/6.cluster-relation-broken Error: unable to connect to node 'rabbit@juju-d020f9-8-lxd-2': nodedown
unit-rabbitmq-server-6: 23:28:27 INFO unit.rabbitmq-server/6.cluster-relation-broken
unit-rabbitmq-server-6: 23:28:27 INFO unit.rabbitmq-server/6.cluster-relation-broken DIAGNOSTICS
unit-rabbitmq-server-6: 23:28:27 INFO unit.rabbitmq-server/6.cluster-relation-broken ===========
unit-rabbitmq-server-6: 23:28:27 INFO unit.rabbitmq-server/6.cluster-relation-broken
unit-rabbitmq-server-6: 23:28:27 INFO unit.rabbitmq-server/6.cluster-relation-broken attempted to contact: ['rabbit@juju-d020f9-8-lxd-2']
unit-rabbitmq-server-6: 23:28:27 INFO unit.rabbitmq-server/6.cluster-relation-broken
unit-rabbitmq-server-6: 23:28:27 INFO unit.rabbitmq-server/6.cluster-relation-broken rabbit@juju-d020f9-8-lxd-2:
unit-rabbitmq-server-6: 23:28:27 INFO unit.rabbitmq-server/6.cluster-relation-broken * connected to epmd (port 4369) on juju-d020f9-8-lxd-2
unit-rabbitmq-server-6: 23:28:27 INFO unit.rabbitmq-server/6.cluster-relation-broken * epmd reports: node 'rabbit' not running at all
unit-rabbitmq-server-6: 23:28:27 INFO unit.rabbitmq-server/6.cluster-relation-broken no other nodes on juju-d020f9-8-lxd-2
unit-rabbitmq-server-6: 23:28:27 INFO unit.rabbitmq-server/6.cluster-relation-broken * suggestion: start the node
unit-rabbitmq-server-6: 23:28:27 INFO unit.rabbitmq-server/6.cluster-relation-broken
unit-rabbitmq-server-6: 23:28:27 INFO unit.rabbitmq-server/6.cluster-relation-broken current node details:
unit-rabbitmq-server-6: 23:28:27 INFO unit.rabbitmq-server/6.cluster-relation-broken - node name: 'rabbitmq-cli-29565@juju-d020f9-8-lxd-2'
unit-rabbitmq-server-6: 23:28:27 INFO unit.rabbitmq-server/6.cluster-relation-broken - home dir: /var/lib/rabbitmq
unit-rabbitmq-server-6: 23:28:27 INFO unit.rabbitmq-server/6.cluster-relation-broken - cookie hash: dXfgzBrUuEMWBGavfsKDzw==
unit-rabbitmq-server-6: 23:28:27 INFO unit.rabbitmq-server/6.cluster-relation-broken
unit-rabbitmq-server-6: 23:28:28 INFO unit.rabbitmq-server/6.cluster-relation-broken Traceback (most recent call last):
unit-rabbitmq-server-6: 23:28:28 INFO unit.rabbitmq-server/6.cluster-relation-broken File "/var/lib/juju/agents/unit-rabbitmq-server-6/charm/hooks/cluster-relation-broken", line 768, in <module>
unit-rabbitmq-server-6: 23:28:28 INFO unit.rabbitmq-server/6.cluster-relation-broken rabbit.assess_status(rabbit.ConfigRenderer(rabbit.CONFIG_FILES))
unit-rabbitmq-server-6: 23:28:28 INFO unit.rabbitmq-server/6.cluster-relation-broken File "/var/lib/juju/agents/unit-rabbitmq-server-6/charm/hooks/rabbit_utils.py", line 796, in assess_status
unit-rabbitmq-server-6: 23:28:28 INFO unit.rabbitmq-server/6.cluster-relation-broken assess_status_func(configs)()
unit-rabbitmq-server-6: 23:28:28 INFO unit.rabbitmq-server/6.cluster-relation-broken File "/var/lib/juju/agents/unit-rabbitmq-server-6/charm/hooks/rabbit_utils.py", line 820, in _assess_status_func
unit-rabbitmq-server-6: 23:28:28 INFO unit.rabbitmq-server/6.cluster-relation-broken services=services(), ports=None)
unit-rabbitmq-server-6: 23:28:28 INFO unit.rabbitmq-server/6.cluster-relation-broken File "/var/lib/juju/agents/unit-rabbitmq-server-6/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1178, in _determine_os_workload_status
unit-rabbitmq-server-6: 23:28:28 INFO unit.rabbitmq-server/6.cluster-relation-broken state, message, lambda: charm_func(configs))
unit-rabbitmq-server-6: 23:28:28 INFO unit.rabbitmq-server/6.cluster-relation-broken File "/var/lib/juju/agents/unit-rabbitmq-server-6/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1306, in _ows_check_charm_func
unit-rabbitmq-server-6: 23:28:28 INFO unit.rabbitmq-server/6.cluster-relation-broken charm_state, charm_message = charm_func_with_configs()
unit-rabbitmq-server-6: 23:28:28 INFO unit.rabbitmq-server/6.cluster-relation-broken File "/var/lib/juju/agents/unit-rabbitmq-server-6/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1178, in <lambda>
unit-rabbitmq-server-6: 23:28:28 INFO unit.rabbitmq-server/6.cluster-relation-broken state, message, lambda: charm_func(configs))
unit-rabbitmq-server-6: 23:28:28 INFO unit.rabbitmq-server/6.cluster-relation-broken File "/var/lib/juju/agents/unit-rabbitmq-server-6/charm/hooks/rabbit_utils.py", line 731, in assess_cluster_status
unit-rabbitmq-server-6: 23:28:28 INFO unit.rabbitmq-server/6.cluster-relation-broken ret = wait_app()
unit-rabbitmq-server-6: 23:28:28 INFO unit.rabbitmq-server/6.cluster-relation-broken File "/var/lib/juju/agents/unit-rabbitmq-server-6/charm/hooks/rabbit_utils.py", line 353, in wait_app
unit-rabbitmq-server-6: 23:28:28 INFO unit.rabbitmq-server/6.cluster-relation-broken raise ex
unit-rabbitmq-server-6: 23:28:28 INFO unit.rabbitmq-server/6.cluster-relation-broken subprocess.CalledProcessError: Command '['timeout', '180', '/usr/sbin/rabbitmqctl', 'wait', '/var/lib/rabbitmq/mnesia/rabbit@juju-d020f9-8-lxd-2.pid']' returned non-zero exit status 124
unit-rabbitmq-server-6: 23:28:28 ERROR juju.worker.uniter.operation hook "cluster-relation-broken" failed: exit status 1
unit-rabbitmq-server-6: 23:28:28 INFO juju.worker.uniter awaiting error resolution for "relation-broken" hook
unit-rabbitmq-server-7: 23:28:36 INFO juju.worker.uniter.operation ran "update-status" hook
unit-rabbitmq-server-7: 23:28:36 INFO juju.worker.uniter no operations in progress; waiting for changes
Every 0.1s: juju status --color Tue Jan 24 23:31:25 2017
Model Controller Cloud/Region Version
default maas-prod-controller maas-prod 2.0.2
App Version Status Scale Charm Store Rev OS Notes
ceph-mon 10.2.3 active 3 ceph-mon jujucharms 6 ubuntu
ceph-osd 10.2.3 active 3 ceph-osd jujucharms 238 ubuntu
ceph-radosgw 10.2.3 active 1 ceph-radosgw jujucharms 245 ubuntu
cinder 9.0.0 active 1 cinder jujucharms 257 ubuntu
cinder-ceph 9.0.0 active 1 cinder-ceph jujucharms 221 ubuntu
glance 13.0.0 active 1 glance jujucharms 253 ubuntu
keystone 10.0.0 active 1 keystone jujucharms 258 ubuntu
neutron-api 9.0.0 active 1 neutron-api jujucharms 246 ubuntu
neutron-gateway 9.0.0 active 1 neutron-gateway jujucharms 232 ubuntu
neutron-openvswitch 9.0.0 active 3 neutron-openvswitch jujucharms 238 ubuntu
nova-cloud-controller 14.0.1 waiting 1 nova-cloud-controller jujucharms 292 ubuntu
nova-compute 14.0.1 active 3 nova-compute jujucharms 259 ubuntu
ntp unknown 4 ntp jujucharms 0 ubuntu
openstack-dashboard 10.0.0 active 1 openstack-dashboard jujucharms 243 ubuntu
percona-cluster 5.6.21-25.8 active 3 percona-cluster jujucharms 246 ubuntu
rabbitmq-server 3.5.7 error 3 rabbitmq-server jujucharms 54 ubuntu
Unit Workload Agent Machine Public address Ports Message
ceph-mon/6 active idle 9/lxd/0 10.5.100.62 Unit is ready and clustered
ceph-mon/7* active idle 10/lxd/0 10.5.100.58 Unit is ready and clustered
ceph-mon/8 active idle 11/lxd/0 10.5.100.66 Unit is ready and clustered
ceph-osd/6 active idle 9 10.5.100.153 Unit is ready (1 OSD)
ceph-osd/7* active idle 10 10.5.100.54 Unit is ready (1 OSD)
ceph-osd/8 active idle 11 10.5.100.154 Unit is ready (1 OSD)
ceph-radosgw/2* active idle 8/lxd/0 10.5.100.56 80/tcp Unit is ready
cinder/2* active idle 9/lxd/1 10.5.100.63 8776/tcp Unit is ready
cinder-ceph/1* active idle 10.5.100.63 Unit is ready
glance/2* active idle 10/lxd/1 10.5.100.59 9292/tcp Unit is ready
keystone/2* active idle 11/lxd/1 10.5.100.67 5000/tcp Unit is ready
neutron-api/2* active idle 9/lxd/2 10.5.100.64 9696/tcp Unit is ready
neutron-gateway/2* active idle 8 10.5.100.152 Unit is ready
ntp/4* unknown idle 10.5.100.152
nova-cloud-controller/3* waiting idle 12/lxd/0 10.5.100.71 8774/tcp Incomplete relations: messaging
nova-compute/6 active idle 9 10.5.100.153 Unit is ready
neutron-openvswitch/4 active idle 10.5.100.153 Unit is ready
ntp/6 unknown idle 10.5.100.153
nova-compute/7* active idle 10 10.5.100.54 Unit is ready
neutron-openvswitch/3* active idle 10.5.100.54 Unit is ready
ntp/5 unknown idle 10.5.100.54
nova-compute/8 active idle 11 10.5.100.154 Unit is ready
neutron-openvswitch/5 active idle 10.5.100.154 Unit is ready
ntp/7 unknown idle 10.5.100.154
openstack-dashboard/2* active idle 11/lxd/2 10.5.100.68 80/tcp,443/tcp Unit is ready
percona-cluster/4* active idle 8/lxd/1 10.5.100.57 Unit is ready
percona-cluster/5 active idle 9/lxd/4 10.5.100.69 Unit is ready
percona-cluster/6 active idle 10/lxd/4 10.5.100.70 Unit is ready
rabbitmq-server/6* error idle 8/lxd/2 10.5.100.55 5672/tcp hook failed: "cluster-relation-broken"
rabbitmq-server/7 waiting idle 9/lxd/3 10.5.100.65 5671/tcp,5672/tcp Unit has peers, but RabbitMQ not clustered
rabbitmq-server/8 waiting idle 10/lxd/3 10.5.100.61 5671/tcp,5672/tcp Unit has peers, but RabbitMQ not clustered
Machine State DNS Inst id Series AZ
8 started 10.5.100.152 8tp8kk xenial VLAN100
8/lxd/0 started 10.5.100.56 juju-d020f9-8-lxd-0 xenial
8/lxd/1 started 10.5.100.57 juju-d020f9-8-lxd-1 xenial
8/lxd/2 started 10.5.100.55 juju-d020f9-8-lxd-2 xenial
9 started 10.5.100.153 8qcnt7 xenial VLAN100
9/lxd/0 started 10.5.100.62 juju-d020f9-9-lxd-0 xenial
9/lxd/1 started 10.5.100.63 juju-d020f9-9-lxd-1 xenial
9/lxd/2 started 10.5.100.64 juju-d020f9-9-lxd-2 xenial
9/lxd/3 started 10.5.100.65 juju-d020f9-9-lxd-3 xenial
9/lxd/4 started 10.5.100.69 juju-d020f9-9-lxd-4 xenial
10 started 10.5.100.54 shhw36 xenial VLAN100
10/lxd/0 started 10.5.100.58 juju-d020f9-10-lxd-0 xenial
10/lxd/1 started 10.5.100.59 juju-d020f9-10-lxd-1 xenial
10/lxd/3 started 10.5.100.61 juju-d020f9-10-lxd-3 xenial
10/lxd/4 started 10.5.100.70 juju-d020f9-10-lxd-4 xenial
11 started 10.5.100.154 enh8de xenial VLAN100
11/lxd/0 started 10.5.100.66 juju-d020f9-11-lxd-0 xenial
11/lxd/1 started 10.5.100.67 juju-d020f9-11-lxd-1 xenial
11/lxd/2 started 10.5.100.68 juju-d020f9-11-lxd-2 xenial
12 started 10.5.100.151 f6376g xenial VLAN100
12/lxd/0 started 10.5.100.71 juju-d020f9-12-lxd-0 xenial
|