Add support for VPC flow log in plain text format #38896
Labels
Comments
constanca-m
added
enhancement
|
Pinging code owners:
See Adding Labels via Comments if you do not have permissions to add labels yourself. |
|
Issue filed by code owner, removing |
dmathieu
pushed a commit
to dmathieu/opentelemetry-collector-contrib
that referenced
this issue
Apr 8, 2025
…ext file format (open-telemetry#38897) <!--Ex. Fixing a bug - Describe the bug and how this fixes the issue. Ex. Adding a feature - Explain what this achieves.--> #### Description Add support for VPC flow logs sent to S3 in plain text format. <!-- Issue number (e.g. open-telemetry#1234) or full URL to issue, if applicable. --> #### Link to tracking issue Fixes open-telemetry#38896. <!--Describe what testing was performed and which tests were added.--> #### Testing There are new unit tests added. <!--Describe the documentation added.--> #### Documentation Comments in the code and unit tests should be enough. Fields are mapped this way: | Flow log field | Attribute | Available? | Currently supported? | |------------------------------|------------------------------------|----------------|----------------------| | `version` | `aws.vpc.flow.log.version` | 🔴 | | | `account-id` | `cloud.account.id` | 🟢 | | | `interface-id` | `aws.eni.id` | 🔴 | | | `srcaddr` | `source.address` or `network.peer.address` | 🔴 | | | `pkt-srcaddr` | `source.address` | 🟢 | | | `dstaddr` | `destination.address` or `network.peer.address` | 🔴 | | | `pkt-dstaddr` | `destination.address` | 🟢 | | | `srcport` | `source.port` | 🟢 | | | `dstport` | `destination.port` | 🟢 | | | `protocol` | `network.protocol.name` | 🟢 | | | `packets` | `aws.vpc.flow.packets` | 🔴 | | | `bytes` | `aws.vpc.flow.bytes` | 🔴 | | | `start` | `aws.vpc.flow.start` | 🔴 | | | `end` | log timestamp | N/A | | | `action` | `aws.vpc.flow.action` | 🔴 | | | `log-status` | `aws.vpc.flow.status` | 🔴 | | | `vpc-id` | `aws.vpc.id` | 🔴 | | | `subnet-id` | `aws.vpc.subnet.id` | 🔴 | | | `instance-id` | `host.id` | 🔴 | | | `tcp-flags` | `network.tcp.flags` | 🔴 | | | `type` | `network.type` | 🟢 | | | `region` | `cloud.region` | 🟢 | | | `az-id` | `aws.az.id` | 🔴 | | | `sublocation-type` | `aws.sublocation.type` | 🔴 | | | `sublocation-id` | `aws.sublocation.id` | 🔴 | | | `pkt-src-aws-service` | `aws.vpc.flow.source.service` | 🔴 | | | `pkt-dst-aws-service` | `aws.vpc.flow.destination.service` | 🔴 | | | `flow-direction` | `network.io.direction` | 🟢 | | | `traffic-path` | `aws.vpc.flow.traffic_path` | 🔴 | | | `ecs-cluster-arn` | `aws.ecs.cluster.arn` | 🟢 | 🔴 | | `ecs-cluster-name` | `aws.ecs.cluster.name` | 🔴 | 🔴 | | `ecs-container-instance-arn` | `aws.ecs.container.instance.arn` | 🔴 | 🔴 | | `ecs-container-instance-id` | `aws.ecs.container.instance.id` | 🔴 | 🔴 | | `ecs-container-id` | `aws.ecs.container.id` | 🔴 | 🔴 | | `ecs-second-container-id` | `aws.ecs.second.container.arn` | 🔴 | 🔴 | | `ecs-service-name` | `aws.ecs.service.name` | 🔴 | 🔴 | | `ecs-task-definition-arn` | `aws.ecs.task.definition.arn` | 🔴 | 🔴 | | `ecs-task-arn` | `aws.ecs.task.arn` | 🟢 | 🔴 | | `ecs-task-id` | `aws.ecs.task.id` | 🟢 | 🔴 | | `reject-reason` | `aws.vpc.flow.reject_reason` | 🔴 | | --------- Co-authored-by: Andrew Wilkins <[email protected]>
Fiery-Fenix
pushed a commit
to Fiery-Fenix/opentelemetry-collector-contrib
that referenced
this issue
Apr 24, 2025
…ext file format (open-telemetry#38897) <!--Ex. Fixing a bug - Describe the bug and how this fixes the issue. Ex. Adding a feature - Explain what this achieves.--> #### Description Add support for VPC flow logs sent to S3 in plain text format. <!-- Issue number (e.g. open-telemetry#1234) or full URL to issue, if applicable. --> #### Link to tracking issue Fixes open-telemetry#38896. <!--Describe what testing was performed and which tests were added.--> #### Testing There are new unit tests added. <!--Describe the documentation added.--> #### Documentation Comments in the code and unit tests should be enough. Fields are mapped this way: | Flow log field | Attribute | Available? | Currently supported? | |------------------------------|------------------------------------|----------------|----------------------| | `version` | `aws.vpc.flow.log.version` | 🔴 | | | `account-id` | `cloud.account.id` | 🟢 | | | `interface-id` | `aws.eni.id` | 🔴 | | | `srcaddr` | `source.address` or `network.peer.address` | 🔴 | | | `pkt-srcaddr` | `source.address` | 🟢 | | | `dstaddr` | `destination.address` or `network.peer.address` | 🔴 | | | `pkt-dstaddr` | `destination.address` | 🟢 | | | `srcport` | `source.port` | 🟢 | | | `dstport` | `destination.port` | 🟢 | | | `protocol` | `network.protocol.name` | 🟢 | | | `packets` | `aws.vpc.flow.packets` | 🔴 | | | `bytes` | `aws.vpc.flow.bytes` | 🔴 | | | `start` | `aws.vpc.flow.start` | 🔴 | | | `end` | log timestamp | N/A | | | `action` | `aws.vpc.flow.action` | 🔴 | | | `log-status` | `aws.vpc.flow.status` | 🔴 | | | `vpc-id` | `aws.vpc.id` | 🔴 | | | `subnet-id` | `aws.vpc.subnet.id` | 🔴 | | | `instance-id` | `host.id` | 🔴 | | | `tcp-flags` | `network.tcp.flags` | 🔴 | | | `type` | `network.type` | 🟢 | | | `region` | `cloud.region` | 🟢 | | | `az-id` | `aws.az.id` | 🔴 | | | `sublocation-type` | `aws.sublocation.type` | 🔴 | | | `sublocation-id` | `aws.sublocation.id` | 🔴 | | | `pkt-src-aws-service` | `aws.vpc.flow.source.service` | 🔴 | | | `pkt-dst-aws-service` | `aws.vpc.flow.destination.service` | 🔴 | | | `flow-direction` | `network.io.direction` | 🟢 | | | `traffic-path` | `aws.vpc.flow.traffic_path` | 🔴 | | | `ecs-cluster-arn` | `aws.ecs.cluster.arn` | 🟢 | 🔴 | | `ecs-cluster-name` | `aws.ecs.cluster.name` | 🔴 | 🔴 | | `ecs-container-instance-arn` | `aws.ecs.container.instance.arn` | 🔴 | 🔴 | | `ecs-container-instance-id` | `aws.ecs.container.instance.id` | 🔴 | 🔴 | | `ecs-container-id` | `aws.ecs.container.id` | 🔴 | 🔴 | | `ecs-second-container-id` | `aws.ecs.second.container.arn` | 🔴 | 🔴 | | `ecs-service-name` | `aws.ecs.service.name` | 🔴 | 🔴 | | `ecs-task-definition-arn` | `aws.ecs.task.definition.arn` | 🔴 | 🔴 | | `ecs-task-arn` | `aws.ecs.task.arn` | 🟢 | 🔴 | | `ecs-task-id` | `aws.ecs.task.id` | 🟢 | 🔴 | | `reject-reason` | `aws.vpc.flow.reject_reason` | 🔴 | | --------- Co-authored-by: Andrew Wilkins <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Component(s)
extension/encoding/awslogsencoding
Is your feature request related to a problem? Please describe.
VPC flow logs can be sent to S3 either in plain text or parquet format.
Describe the solution you'd like
Add support for the logs sent in plain text format.
Describe alternatives you've considered
No response
Additional context
See main issue #38861.
The text was updated successfully, but these errors were encountered: