[extension/awslogsencoding] Add support for AWS RDS log

[constanca-m]

Component(s)

extension/encoding/awslogsencoding

Is your feature request related to a problem? Please describe.

Add support for AWS RDS logs coming through cloudwatch. The log structure depends on the engine or log type, so we need to add separation for it (PostgreSQL for example).

Describe the solution you'd like

extension:
  awslogs_encoding:
    format: cloudwatch_logs_subscription_filter
    message_format:
      engine: postgresql # mariadb, ...
      log_type: # general_log, slow_query_log, ...

Describe alternatives you've considered

N/A

Additional context

N/A

[github-actions]

Pinging code owners:

• extension/encoding/awslogsencoding: @axw @constanca-m

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[axw]

@constanca-m IIANM, RDS logs are published through CloudWatch: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_LogAccess.Procedural.UploadtoCloudWatch.html

Add support for AWS RDS logs in S3 bucket. It is possible to transfer these log files between RDS and S3 (see documentation).

That link appears to be related to transferring Oracle data files to/from S3.

I do think we should have support for parsing RDS logs, but I don't know that the proposed configuration would be right. Presumably it would need to compose somehow with the cloudwatch_logs_subscription_filter format.

[constanca-m]

IIANM, RDS logs are published through CloudWatch [...] That link appears to be related to transferring Oracle data files to/from S3.

Indeed, but our subscription filter does not parse the message field (at the moment). So I looked into the documentation, and thought that for now we could transfer the RDS log files to S3, and then parse from there (documentation on postgresql logs exported to S3 here, I think all engines have documentation on exporting the log files to S3).

[axw]

documentation on postgresql logs exported to S3 here

@constanca-m maybe I'm blind, but similar to the Oracle link, that documentation appears to be about transferring Postgres data files to/from S3, and not logs. That may be used for importing/exporting database snapshots.

[constanca-m]

You are right @axw , for some reason I was interpreting the data file as a log file, but you just made me realize that was wrong. So in that case it is not straightforward to send to S3, so you were right that we have to think about this case on the cloudwatch level.

[axw]

OK, well regardless of that I agree that we should have this capability. Could you please clarify the issue description?

I expect we'll need to somehow support a nested encoding for CloudWatch logs. Ideally that would work with both the subscription filter format, and with the pull-based awscloudwatchceiver.

[constanca-m]

@axw I have updated it, but I feel we should open a separate issue to add a feature of message_format to indicate how to parse a CW message field. WDYT?