CORTX-33879: Fixed errors while fetching resource limit conf from consul confstore

[rohitkolapkar]

Pull Request

Problem Statement

• Handle Resource Limit configuration during mini-provisioning to work with consul
• Problem here is -
  CSM is trying to read Limits conf dict from gconf, and consul based gconf doesnt support this.
• Solution,
  Iterate over the list and create key structure and fetch the config.

Coding

Checklist for Author

• Coding conventions are followed and code is consistent

Testing

Checklist for Author

• Unit and System Tests are added
• Test Cases cover Happy Path, Non-Happy Path and Scalability
• Testing was performed with RPM

Impact Analysis

Checklist for Author/Reviewer/GateKeeper

• Interface change (if any) are documented
• Side effects on other features (deployment/upgrade)
• Dependencies on other component(s)

Review Checklist

Checklist for Author

• JIRA number/GitHub Issue added to PR
• PR is self reviewed
• Jira and state/status is updated and JIRA is updated with PR link
• Check if the description is clear and explained

Documentation

Checklist for Author

• Changes done to WIKI / Confluence page / Quick Start Guide

Dev Testing

[root@cortx-control /]# /opt/seagate/cortx/csm/bin/csm_setup post_install --config consul://cortx-consul-server:8500/conf --services agent 2022-08-08 14:40:54 csm_setup [28]: INFO [execute] Setup: Initiating Post Install phase. 2022-08-08 14:40:54 csm_setup [28]: INFO [load_csm_config_indices] Loading CSM configuration 2022-08-08 14:40:54 csm_setup [28]: INFO [get_consul_config] Fetching consul endpoint : http://cortx-consul-server:8500 2022-08-08 14:40:54 csm_setup [28]: INFO [copy_base_configs] Copying Csm base configurations to destination indices 2022-08-08 14:40:55 csm_setup [28]: INFO [_prepare_and_validate_confstore_keys] Post Install: Validating required configuration. 2022-08-08 14:40:55 csm_setup [28]: INFO [_validate_conf_store_keys] Validating confstore keys: ['cortx>common>security>ssl_certificate'] 2022-08-08 14:40:55 csm_setup [28]: INFO [set_ssl_certificate] Post Install: Setting SSL certificate path: /etc/cortx/solution/ssl/s3.seagate.com.pem 2022-08-08 14:40:55 csm_setup [28]: INFO [set_logpath] Post Install: Setting log path: /etc/cortx/log/csm 2022-08-08 14:40:55 csm_setup [28]: INFO [create] Post Install: Creating CSM configuration file on required location. 2022-08-08 14:40:55 csm_setup [28]: INFO [execute] Setup: Successfully passed Post Install phase. :PASS [root@cortx-control /]# /opt/seagate/cortx/csm/bin/csm_setup prepare --config consul://cortx-consul-server:8500/conf --services agent 2022-08-08 14:41:07 csm_setup [32]: INFO [execute] Setup: Initiating Prepare phase. 2022-08-08 14:41:07 csm_setup [32]: INFO [load_csm_config_indices] Loading CSM configuration 2022-08-08 14:41:07 csm_setup [32]: INFO [get_consul_config] Fetching consul endpoint : http://cortx-consul-server:8500 2022-08-08 14:41:07 csm_setup [32]: INFO [_prepare_and_validate_confstore_keys] Prepare: Validating required configuration. 2022-08-08 14:41:07 csm_setup [32]: INFO [_validate_conf_store_keys] Validating confstore keys: ['node>ecabccb4bced858e282f568d6bf04558>hostname', 'node>ecabccb4bced858e282f568d6bf04558>cluster_id', 'cortx>external>consul>secret'] 2022-08-08 14:41:07 csm_setup [32]: INFO [_set_secret_string_for_decryption] Prepare: Setting decryption keys for CSM and S3 2022-08-08 14:41:07 csm_setup [32]: INFO [_set_cluster_id] Prepare: Setting up cluster id 2022-08-08 14:41:07 csm_setup [32]: INFO [_set_db_host_addr] Prepare: Setting database host address 2022-08-08 14:41:07 csm_setup [32]: INFO [get_consul_config] Fetching consul endpoint : http://cortx-consul-server:8500 2022-08-08 14:41:07 csm_setup [32]: INFO [create] Prepare: Creating configuration file on required location. 2022-08-08 14:41:07 csm_setup [32]: INFO [execute] Setup: Successfully passed Prepare phase. :PASS [root@cortx-control /]# /opt/seagate/cortx/csm/bin/csm_setup config --config consul://cortx-consul-server:8500/conf --services agent 2022-08-08 14:41:24 csm_setup [36]: INFO [execute] Setup: Initiating Config phase. 2022-08-08 14:41:24 csm_setup [36]: INFO [load_csm_config_indices] Loading CSM configuration 2022-08-08 14:41:24 csm_setup [36]: INFO [get_consul_config] Fetching consul endpoint : http://cortx-consul-server:8500 2022-08-08 14:41:24 csm_setup [36]: INFO [execute] Config: Force flag: False 2022-08-08 14:41:24 csm_setup [36]: INFO [_prepare_and_validate_confstore_keys] Config: Validating required configuration. 2022-08-08 14:41:24 csm_setup [36]: INFO [_validate_conf_store_keys] Validating confstore keys: ['node>ecabccb4bced858e282f568d6bf04558>cluster_id', 'cortx>rgw>auth_user', 'cortx>rgw>auth_admin', 'cortx>rgw>auth_secret'] 2022-08-08 14:41:24 csm_setup [36]: INFO [set_resource_limits] Config: Fetching CSM services cpu and memory limits 2022-08-08 14:41:24 csm_setup [36]: INFO [_calculate_request_quota] CPU boundaries are currently not included in calculation: 250:500 2022-08-08 14:41:24 csm_setup [36]: INFO [_set_csm_endpoint] Config: Setting CSM endpoint in configuration. 2022-08-08 14:41:24 csm_setup [36]: INFO [_set_s3_info] Config: Setting S3 information in configuration 2022-08-08 14:41:24 csm_setup [36]: INFO [_set_s3_endpoints] Config: Setting S3 endpoints in configuration 2022-08-08 14:41:24 csm_setup [36]: INFO [set_hax_endpoint] Config: Setting HAX endpoints in configuration 2022-08-08 14:41:24 csm_setup [36]: INFO [execute] Config: Creating topics for various CSM functionalities. 2022-08-08 14:41:24 csm_setup [36]: INFO [_create_topics] Config: Creating topics for communication channel. 2022-08-08 14:41:24 csm_setup [36]: INFO [_create_topics] Config: Connecting to message bus using endpoint :['tcp://cortx-kafka:9092'] 2022-08-08 14:41:24 csm_setup [36]: INFO [init] MessageBus initialized as kafka 2022-08-08 14:41:24 csm_setup [36]: INFO [_create_perf_stat_topic] Config: Creating performance statistics topic 2022-08-08 14:41:24 csm_setup [36]: INFO [_create_perf_stat_topic] Config: message_type:perf_stat already exists. 2022-08-08 14:41:24 csm_setup [36]: INFO [_create_cluster_stop_topic] Config: Creating cluster stop topic 2022-08-08 14:41:24 csm_setup [36]: INFO [_create_cluster_admin] Creating cluster admin account 2022-08-08 14:41:24 csm_setup [36]: INFO [_create_cluster_admin] Set Cortx admin credentials in config [_create_cluster_admin] WARNING: Cortx cluster admin already created. Please use '-f' option to create admin user forcefully. 2022-08-08 14:41:25 csm_setup [36]: INFO [create] Config: Creating CSM configuration file on required location. 2022-08-08 14:41:25 csm_setup [36]: INFO [execute] Setup: Successfully passed Config phase. :PASS /usr/local/lib/python3.6/site-packages/consul/aio.py:58: RuntimeWarning: coroutine 'ClientSession.close' was never awaited Unclosed client session client_session: <aiohttp.client.ClientSession object at 0x7efe909a4668> Unclosed connector connections: ['[(<aiohttp.client_proto.ResponseHandler object at 0x7efe909383f0>, 16693244.30384383)]'] connector: <aiohttp.connector.TCPConnector object at 0x7efe909a4a20> [root@cortx-control /]# /opt/seagate/cortx/csm/bin/csm_setup init --config consul://cortx-consul-server:8500/conf --services agent 2022-08-08 14:41:50 csm_setup [47]: INFO [execute] Setup: Initiating Init phase. 2022-08-08 14:41:50 csm_setup [47]: INFO [execute] Setup: Successfully passed Init phase. :PASS [root@cortx-control /]#