CORTX-34147: Add time stamp for consul process state updated in bq

[pavankrishnat]

Problem:
HARE bq messages in Support Bundle do not contain timestamps.
This is needed for effective offline debug using logs from Support bundle.

Solution:
Add timestamps to bq publisher; also add the consul kv dump so generated to
Support Bundle.

Signed-off-by: pavankrishnat [email protected]

[pavankrishnat]

Deployed 3N setup with PR changes.
Build Images: https://eos-jenkins.colo.seagate.com/job/GitHub-custom-ci-builds/job/generic/job/custom-ci/7846/console
3N deployment: https://eos-jenkins.colo.seagate.com/job/Cortx-Automation/job/RGW/job/setup-cortx-rgw-cluster/12727/console

Timestamp o/p:

bq-delivered/cortx-server-2.cortx-server-headless.cortx.svc.cluster.local:76
bq/10:{"timestamp": "2022-09-15 09:32:06.802311", "message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0xc", "state": "M0_CONF_HA_PROCESS_STARTING", "type": "M0_CONF_HA_PROCESS_HA"}}
bq/11:{"timestamp": "2022-09-15 09:32:07.030644", "message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0xc", "state": "M0_CONF_HA_PROCESS_DTM_RECOVERED", "type": "M0_CONF_HA_PROCESS_HA"}}
bq/12:{"timestamp": "2022-09-15 09:32:09.407068", "message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x9", "state": "M0_CONF_HA_PROCESS_STARTING", "type": "M0_CONF_HA_PROCESS_HA"}}
bq/13:{"timestamp": "2022-09-15 09:32:09.906653", "message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x9", "state": "M0_CONF_HA_PROCESS_DTM_RECOVERED", "type": "M0_CONF_HA_PROCESS_HA"}}
bq/14:{"timestamp": "2022-09-15 09:32:12.663020", "message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x0", "state": "M0_CONF_HA_PROCESS_STOPPED", "type": "M0_CONF_HA_PROCESS_HA"}}
bq/15:{"timestamp": "2022-09-15 09:32:33.500649", "message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x9", "state": "M0_CONF_HA_PROCESS_STARTING", "type": "M0_CONF_HA_PROCESS_HA"}}
bq/16:{"timestamp": "2022-09-15 09:32:33.774727", "message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x3", "state": "M0_CONF_HA_PROCESS_STARTING", "type": "M0_CONF_HA_PROCESS_HA"}}
bq/17:{"timestamp": "2022-09-15 09:32:33.800894", "message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x0", "state": "M0_CONF_HA_PROCESS_STARTING", "type": "M0_CONF_HA_PROCESS_HA"}}
bq/18:{"timestamp": "2022-09-15 09:32:34.885731", "message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x4", "state": "M0_CONF_HA_PROCESS_STARTING", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/19:{"timestamp": "2022-09-15 09:32:34.981265", "message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0xc", "state": "M0_CONF_HA_PROCESS_STARTING", "type": "M0_CONF_HA_PROCESS_HA"}}
bq/2:{"timestamp": "2022-09-15 09:32:04.270972", "message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x0", "state": "M0_CONF_HA_PROCESS_STARTING", "type": "M0_CONF_HA_PROCESS_HA"}}
bq/20:{"timestamp": "2022-09-15 09:32:35.313785", "message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x2", "state": "M0_CONF_HA_PROCESS_STARTING", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/21:{"timestamp": "2022-09-15 09:32:36.003414", "message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x6", "state": "M0_CONF_HA_PROCESS_STARTING", "type": "M0_CONF_HA_PROCESS_HA"}}
bq/22:{"timestamp": "2022-09-15 09:32:36.369147", "message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x5", "state": "M0_CONF_HA_PROCESS_STARTING", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/23:{"timestamp": "2022-09-15 09:32:36.403062", "message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x0", "state": "M0_CONF_HA_PROCESS_DTM_RECOVERED", "type": "M0_CONF_HA_PROCESS_HA"}}
bq/24:{"timestamp": "2022-09-15 09:32:36.370746", "message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0xc", "state": "M0_CONF_HA_PROCESS_DTM_RECOVERED", "type": "M0_CONF_HA_PROCESS_HA"}}
bq/25:{"timestamp": "2022-09-15 09:32:36.618191", "message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x1", "state": "M0_CONF_HA_PROCESS_STARTING", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/26:{"timestamp": "2022-09-15 09:32:36.874636", "message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0xe", "state": "M0_CONF_HA_PROCESS_STARTING", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/27:{"timestamp": "2022-09-15 09:32:36.783515", "message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0xd", "state": "M0_CONF_HA_PROCESS_STARTING", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/28:{"timestamp": "2022-09-15 09:32:36.382924", "message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x3", "state": "M0_CONF_HA_PROCESS_DTM_RECOVERED", "type": "M0_CONF_HA_PROCESS_HA"}}
bq/29:{"timestamp": "2022-09-15 09:32:37.098214", "message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x9", "state": "M0_CONF_HA_PROCESS_DTM_RECOVERED", "type": "M0_CONF_HA_PROCESS_HA"}}
bq/3:{"timestamp": "2022-09-15 09:32:04.520076", "message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x0", "state": "M0_CONF_HA_PROCESS_DTM_RECOVERED", "type": "M0_CONF_HA_PROCESS_HA"}}
bq/30:{"timestamp": "2022-09-15 09:32:37.403330", "message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x7", "state": "M0_CONF_HA_PROCESS_STARTING", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/31:{"timestamp": "2022-09-15 09:32:37.199989", "message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0xa", "state": "M0_CONF_HA_PROCESS_STARTING", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/32:{"timestamp": "2022-09-15 09:32:37.812660", "message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x13", "state": "M0_CONF_HA_PROCESS_STARTING", "type": "M0_CONF_HA_PROCESS_OTHER"}}
bq/33:{"timestamp": "2022-09-15 09:32:37.806868", "message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x6", "state": "M0_CONF_HA_PROCESS_DTM_RECOVERED", "type": "M0_CONF_HA_PROCESS_HA"}}
bq/34:{"timestamp": "2022-09-15 09:32:38.195043", "message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x8", "state": "M0_CONF_HA_PROCESS_STARTING", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/35:{"timestamp": "2022-09-15 09:32:38.090514", "message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0xf", "state": "M0_CONF_HA_PROCESS_STARTING", "type": "M0_CONF_HA_PROCESS_HA"}}
bq/36:{"timestamp": "2022-09-15 09:32:37.902176", "message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0xb", "state": "M0_CONF_HA_PROCESS_STARTING", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/37:{"timestamp": "2022-09-15 09:32:39.992530", "message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x11", "state": "M0_CONF_HA_PROCESS_STARTING", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/38:{"timestamp": "2022-09-15 09:32:40.194858", "message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x10", "state": "M0_CONF_HA_PROCESS_STARTING", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/39:{"timestamp": "2022-09-15 09:32:41.405994", "message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0xf", "state": "M0_CONF_HA_PROCESS_DTM_RECOVERED", "type": "M0_CONF_HA_PROCESS_HA"}}
bq/4:{"timestamp": "2022-09-15 09:32:05.309703", "message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x6", "state": "M0_CONF_HA_PROCESS_STARTING", "type": "M0_CONF_HA_PROCESS_HA"}}
bq/40:{"timestamp": "2022-09-15 09:32:43.182650", "message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0xe", "state": "M0_CONF_HA_PROCESS_DTM_RECOVERED", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/41:{"timestamp": "2022-09-15 09:32:44.907891", "message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x2", "state": "M0_CONF_HA_PROCESS_DTM_RECOVERED", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/42:{"timestamp": "2022-09-15 09:32:45.611697", "message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0xd", "state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/43:{"timestamp": "2022-09-15 09:32:45.993907", "message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x5", "state": "M0_CONF_HA_PROCESS_DTM_RECOVERED", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/44:{"timestamp": "2022-09-15 09:32:46.495629", "message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0xd", "state": "M0_CONF_HA_PROCESS_DTM_RECOVERED", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/45:{"timestamp": "2022-09-15 09:32:47.096427", "message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x4", "state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/46:{"timestamp": "2022-09-15 09:32:48.178364", "message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x4", "state": "M0_CONF_HA_PROCESS_DTM_RECOVERED", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/47:{"timestamp": "2022-09-15 09:32:48.714165", "message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x1", "state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/48:{"timestamp": "2022-09-15 09:32:49.802098", "message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x1", "state": "M0_CONF_HA_PROCESS_DTM_RECOVERED", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/49:{"timestamp": "2022-09-15 09:32:50.006210", "message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0xb", "state": "M0_CONF_HA_PROCESS_DTM_RECOVERED", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/5:{"timestamp": "2022-09-15 09:32:05.549774", "message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x6", "state": "M0_CONF_HA_PROCESS_DTM_RECOVERED", "type": "M0_CONF_HA_PROCESS_HA"}}
bq/50:{"timestamp": "2022-09-15 09:32:50.006224", "message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x11", "state": "M0_CONF_HA_PROCESS_DTM_RECOVERED", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/51:{"timestamp": "2022-09-15 09:32:52.322129", "message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x10", "state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/52:{"timestamp": "2022-09-15 09:32:53.097295", "message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0xa", "state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/53:{"timestamp": "2022-09-15 09:32:53.733022", "message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x10", "state": "M0_CONF_HA_PROCESS_DTM_RECOVERED", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/54:{"timestamp": "2022-09-15 09:32:53.993844", "message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x8", "state": "M0_CONF_HA_PROCESS_DTM_RECOVERED", "type": "M0_CONF_HA_PROCESS_M0D"}}

cmd: /opt/seagate/cortx/hare/bin/hare_setup support_bundle -c 'consul://cortx-consul-server:8500/conf' -s all

Support Bundle log and terminal log

[d-nayak]

@pavankrishnat: Please edit commit message to adhere to our format as follows:

Problem:
HARE bq messages in Support Bundle do not contain timestamps.
This is needed for effective offline debug using logs from Support bundle.

Solution:
Add timestamps to bq publisher; also add the consul kv dump so generated to Support Bundle.