ServiceUnavailable exception on aws.iam.load_policy

[achantavy]

Description:

What issue is being seen? Describe what should be happening instead of the bug, for example: Cartography should not crash, the expected value isn't returned, the data schema is wrong, etc.

Expectation:
AWS IAM sync should not crash with neo4j database errors.

Actual:
Graph write operations can crash with a ServiceUnavailable exception from Neo4j. This is difficult to reproduce. We have seen this issue also in #522, but that case was different because that involved loading a lot of data at once in an UNWIND.

In this case, we are performing multiple auto-commit transactions via MERGE, which likely involves many connections being spun up and down over and over. To solve, batching these writes would likely help. The right solution involves implementing a standardized graph writer for all operations in this project.

To Reproduce:

Steps to reproduce the behavior. Provide all data and inputs required to reproduce the issue.

Difficult to repro; it comes up intermittently.

Logs:

If applicable, copy and paste your console log with the failing stack trace.

File "{PATH}/cartography/intel/aws/iam.py", line 721, in sync
    sync_roles(neo4j_session, boto3_session, current_aws_account_id, update_tag, common_job_parameters)
  File "{PATH}/cartography/util.py", line 59, in timed
    result = method(*args, **kwargs)
  File "{PATH}/cartography/intel/aws/iam.py", line 645, in sync_roles
    sync_role_inline_policies(current_aws_account_id, boto3_session, data, neo4j_session, aws_update_tag)
  File "{PATH}/cartography/intel/aws/iam.py", line 669, in sync_role_inline_policies
    load_policy_data(neo4j_session, inline_policy_data, PolicyType.inline.value, aws_update_tag)
  File "{PATH}/cartography/util.py", line 59, in timed
    result = method(*args, **kwargs)
  File "{PATH}/cartography/intel/aws/iam.py", line 562, in load_policy_data
    load_policy(neo4j_session, policy_id, policy_name, policy_type, principal_arn, aws_update_tag)
  File "{PATH}/cartography/util.py", line 59, in timed
    result = method(*args, **kwargs)
  File "{PATH}/cartography/intel/aws/iam.py", line 520, in load_policy
    aws_update_tag=aws_update_tag,
  File "{PATH}/neo4j/__init__.py", line 972, in consume
    for _ in self:
  File "{PATH}/neo4j/__init__.py", line 952, in records
    self._session.fetch()
  File "{PATH}/neo4j/__init__.py", line 527, in fetch
    detail_count, _ = self._connection.fetch()
  File "{PATH}/neobolt/direct.py", line 419, in fetch
    return self._fetch()
  File "{PATH}/neobolt/direct.py", line 440, in _fetch
    self._receive()
  File "{PATH}/neobolt/direct.py", line 487, in _receive
    raise self.Error(message)
neobolt.exceptions.ServiceUnavailable: Failed to read from defunct connection Address(host='{URL}', port=7687) (Address(host='{IP}', port=7687))

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[stale]

This issue has been automatically closed for inactivity. If you still wish to make these changes, please open a new change or reopen this one.