2017-03-23 22:35:11 DEBUG health:57:health: Health check: busy
2017-03-23 22:35:16 DEBUG health:57:health: Health check: busy
2017-03-23 22:35:21 DEBUG health:57:health: Health check: busy
2017-03-23 22:35:26 DEBUG health:57:health: Health check: busy
2017-03-23 22:35:31 DEBUG health:57:health: Health check: busy
2017-03-23 22:35:36 DEBUG health:57:health: Health check: settling
2017-03-23 22:35:41 DEBUG health:57:health: Health check: settling
2017-03-23 22:35:46 DEBUG health:57:health: Health check: settling
2017-03-23 22:35:51 DEBUG health:57:health: Health check: settling
2017-03-23 22:35:56 DEBUG health:57:health: Health check: settling
2017-03-23 22:36:01 DEBUG health:57:health: Health check: healthy
2017-03-23 22:36:01 DEBUG end_to_end:46:end_to_end: SKIPPING: No end-to-end provided by bundle
2017-03-23 22:36:01 DEBUG chaos:111:chaos: Starting chaos
2017-03-23 22:36:02 DEBUG chaos:142:chaos: Writing chaos plan to /srv/artifacts/cwr_bundle_hadoop_processing/9/chaos_plan_job-9-matrix-decent-ram.yaml
2017-03-23 22:36:02 DEBUG chaos:84:perform_action: Creating CHAOS remove_unit: [<Unit entity_id="rsyslog/0">]
2017-03-23 22:36:02 DEBUG chaos:111:remove_unit: Skipping remove unit for <Unit entity_id="rsyslog/0">, as it is the last unit in rsyslog
2017-03-23 22:36:04 DEBUG chaos:84:perform_action: Creating CHAOS add_unit: [<Application entity_id="rsyslog">]
2017-03-23 22:36:06 DEBUG health:57:health: Health check: healthy
2017-03-23 22:36:10 DEBUG chaos:152:chaos: Could not run reboot. No objects for selectors [{'application': 'rsyslog-forwarder-ha', 'selector': 'units'}, {'value': False, 'selector': 'leader'}, {'selector': 'one'}]
2017-03-23 22:36:11 DEBUG chaos:84:perform_action: Creating CHAOS remove_unit: [<Unit entity_id="slave/0">]
2017-03-23 22:36:11 DEBUG health:57:health: Health check: healthy
2017-03-23 22:36:14 DEBUG chaos:84:perform_action: Creating CHAOS reboot: [<Unit entity_id="slave/0">]
2017-03-23 22:36:16 DEBUG health:57:health: Health check: healthy
2017-03-23 22:36:21 DEBUG health:57:health: Health check: healthy
2017-03-23 22:36:26 DEBUG health:57:health: Health check: healthy
2017-03-23 22:36:31 DEBUG health:57:health: Health check: healthy
2017-03-23 22:36:36 DEBUG health:57:health: Health check: healthy
2017-03-23 22:36:41 DEBUG health:57:health: Health check: healthy
2017-03-23 22:36:44 DEBUG chaos:89:perform_action: Timeout running reboot
2017-03-23 22:36:44 DEBUG matrix:180:execute: Error in Rule(task=Task(command='matrix.tasks.chaos', args={}, gating='ha_only'), conditions=[Condition(mode='after', statement='health.status.healthy')])'s task Task(command='matrix.tasks.chaos', args={}, gating='ha_only')
2017-03-23 22:36:44 DEBUG Traceback (most recent call last):
2017-03-23 22:36:44 DEBUG File "/usr/local/lib/python3.5/dist-packages/matrix/model.py", line 174, in execute
2017-03-23 22:36:44 DEBUG result = await self.execute_plugin(context, cmd, rule)
2017-03-23 22:36:44 DEBUG File "/usr/local/lib/python3.5/dist-packages/matrix/model.py", line 191, in execute_plugin
2017-03-23 22:36:44 DEBUG result = await cmd(context, rule, self, event)
2017-03-23 22:36:44 DEBUG File "/usr/local/lib/python3.5/dist-packages/matrix/tasks/chaos/main.py", line 155, in chaos
2017-03-23 22:36:44 DEBUG if errors and utils.should_gate(context=context, task=task):
2017-03-23 22:36:44 DEBUG File "/usr/local/lib/python3.5/dist-packages/matrix/utils.py", line 252, in should_gate
2017-03-23 22:36:44 DEBUG if task.gating == 'ha_only' and context.ha is True:
2017-03-23 22:36:44 DEBUG AttributeError: 'Context' object has no attribute 'ha'
2017-03-23 22:36:44 DEBUG matrix:322:run_once: Pending tasks remain, aborting due to failure or timeout
2017-03-23 22:36:44 DEBUG matrix:340:run_once: Exception processing test: end_to_end
2017-03-23 22:36:44 DEBUG Traceback for <Task finished coro=<RuleEngine.rule_runner() done, defined at /usr/local/lib/python3.5/dist-packages/matrix/rules.py:177> exception=AttributeError("'Context' object has no attribute 'ha'",)> (most recent call last):
2017-03-23 22:36:44 DEBUG File "/usr/lib/python3.5/asyncio/tasks.py", line 292, in _step
2017-03-23 22:36:44 DEBUG self = None # Needed to break cycles when an exception occurs.
2017-03-23 22:36:44 DEBUG File "/usr/local/lib/python3.5/dist-packages/matrix/rules.py", line 270, in rule_runner
2017-03-23 22:36:44 DEBUG break
2017-03-23 22:36:44 DEBUG File "/usr/local/lib/python3.5/dist-packages/matrix/model.py", line 356, in execute
2017-03-23 22:36:44 DEBUG result = await self.task.execute(context, self)
2017-03-23 22:36:44 DEBUG File "/usr/local/lib/python3.5/dist-packages/matrix/model.py", line 181, in execute
2017-03-23 22:36:44 DEBUG raise
2017-03-23 22:36:44 DEBUG File "/usr/local/lib/python3.5/dist-packages/matrix/model.py", line 191, in execute_plugin
2017-03-23 22:36:44 DEBUG result = await cmd(context, rule, self, event)
2017-03-23 22:36:44 DEBUG File "/usr/local/lib/python3.5/dist-packages/matrix/tasks/chaos/main.py", line 155, in chaos
2017-03-23 22:36:44 DEBUG if errors and utils.should_gate(context=context, task=task):
2017-03-23 22:36:44 DEBUG File "/usr/local/lib/python3.5/dist-packages/matrix/utils.py", line 252, in should_gate
2017-03-23 22:36:44 DEBUG if task.gating == 'ha_only' and context.ha is True:
2017-03-23 22:36:44 DEBUG AttributeError: 'Context' object has no attribute 'ha'
2017-03-23 22:36:44 DEBUG
2017-03-23 22:36:44 DEBUG matrix:342:run_once: Setting exit code to 1.
2017-03-23 22:36:44 DEBUG ------------------------------------------------------------------------------