feat: replace standard logging with structured logging #122

therealvio · 2025-01-15T07:47:12Z

Purpose 🎯

These changes replace the standard library logging with structured logging using structlog. Structured logging provides easier-to-parse capabilities for toolings and humans when going through logs.

Context 🧠

Part 1 to satisfy Suggestion: Add Structured logging #76
- I intend to contribute a follow-up PR that adds logging all connection test events for the purpose of fostering greater support for SLIs, as mentioned in this comment. For those unfamiliar, calculating the SLI is a matter of bad events/all valid events, and right now aws-alternat exercises a kind of "no news is good news" reporting.

Notes 📓

I have performed an in-place replacement of the default logger
No changes to the message contents have been, with some exceptions that have been explicitly pulled out into standalone commits for ease of reviewing.
Example screenshot of debug log messages through structlog

therealvio · 2025-01-15T07:49:08Z

functions/replace-route/app.py

 logger = logging.getLogger()
-logger.setLevel(logging.INFO)
 logging.getLogger('boto3').setLevel(logging.CRITICAL)
 logging.getLogger('botocore').setLevel(logging.CRITICAL)


I found this guide with regards to working with the standard library, but I wasn't sure how I could make this applicable here. Otherwise, third party deps will continue using the standard logger unless someone can figure out if wrapping the logger is possible.

My colleague suggests that we could use https://nhairs.github.io/python-json-logger/latest/ as the formatter for these handlers. e.g. something like:

handler = logging.StreamHandler() handler.setFormatter(JsonFormatter()) logging.getLogger('boto3').addHandler(handler) logging.getLogger('botocore').addHandler(handler)

Do you think this would work?

I think so! This would align with that guide, the section specifically being: "Rendering using logging-based formatters".

I will give it a go and push out a separate commit for this. May take me a bit because it's going to involve tweaking the configs based on what I have now, vs what the guide I am looking at is suggesting.

An FYI: In the change, handler is a loaded term, since that's the entry point of the function. For now, I have opted for loghandler and we can come up with a better nameas we iterate and review.

I have applied the suggestion regarding python-json-logger for stdlib calls within third party deps :)

Thanks for your patience!

functions/replace-route/app.py

bwhaley · 2025-01-21T16:47:09Z

Thank you for the PR! I am a bit busy atm, but I hope to review & discuss in the coming days. Appreciate the patience!

bwhaley

Left a couple comments/ideas to consider.

Mind taking a peek at test_replace_route.py and adding a unit test?

bwhaley · 2025-01-24T23:49:51Z

functions/replace-route/app.py

    }
    try:
-        logger.info("Replacing existing route %s for route table %s", route_table_id, new_route_table)
+        slogger.info("Replacing existing route %s for route table %s", route_table_id, new_route_table)


This line prints a dictionary (new_route_table), which would end up nested in json. The message here is sorta broken - "existing route {route_table_id}" is wrong, and then it prints the whole route table.

Let's update the message so it's more accurate and doesn't include a nested dictionary. It will print more cleanly in the output. It does break backward compatibility for folks monitoring for this specific phrasing, which is too bad, but worthwhile to fix. May as well do it now.

Suggested change

slogger.info("Replacing existing route %s for route table %s", route_table_id, new_route_table)

slogger.info("Updating route table %s to use NAT Gateway ID %s", route_table_id, nat_gateway_id)

Also, most folks should be monitoring for ERROR status or the "Failed connectivity tests! Replacing route" message. This one is INFO.

No worries. I will break this out into a separate commit.

Also, most folks should be monitoring for ERROR status or the "Failed connectivity tests! Replacing route" message. This one is INFO.

You'd think it would go without saying. Perhaps in the release we could include that as a sub dot-point? It feels weird having to tell people "how" to monitor things, but mistakes happen.

I have made this into a standalone commit including a breaking change footer for the commit.

I had to make some tweaks to your suggested log line because NAT Gateway ID assumes only NAT gateway IDs are being fed in. When the replace_route() function is called, it can use either a NAT gateway or EC2 instance ID. So instead, I reworded it as: Updating route table %s to use NAT target %s".

functions/replace-route/app.py

bwhaley · 2025-01-25T00:08:57Z

functions/replace-route/app.py

+)
+
+# logger is still needed to set the level for dependencies
 logger = logging.getLogger()


Is this line still needed?

The snippet is selecting 4 lines, I gather this may be for the comment.

This relates to this comment regarding third party dependencies. structlog doesn't appear to completely drop-in to replace logger in this specific area. If it's not valuable, I am happy to drop it. My intention here was to describe why logger is still being used while slogger is in place.

I don't believe any of my commits, or this PR actually describes why they're staying, so a git log wouldn't help anyone in that area :(. I don't know if the commit will be a squash or not, but if it's a merge commit, then I can amend the commit message where the surrounding change happened if it helps?

What I'm confused about is that logger = logging.getLogger() is still set here but is never used throughout the file. The levels are set using logging.getLogger below. But is the logger object still used in some way? I may be misunderstanding.

Looking back on this, and recollecting:

What's going on here is described in the structlog docs regarding configuring the stdlib logger being used by third party dependencies like boto and botocore that use stdlib logger. In this case, we are configuring the log level for these dependencies.

I believe we solved the JSON formatting problem for a similar problem in this commentary thread. This is similar in nature.

Yep, that makes sense.

bwhaley · 2025-01-25T00:11:55Z

functions/replace-route/app.py

 logger = logging.getLogger()
-logger.setLevel(logging.INFO)
 logging.getLogger('boto3').setLevel(logging.CRITICAL)
 logging.getLogger('botocore').setLevel(logging.CRITICAL)


My colleague suggests that we could use https://nhairs.github.io/python-json-logger/latest/ as the formatter for these handlers. e.g. something like:

handler = logging.StreamHandler() handler.setFormatter(JsonFormatter()) logging.getLogger('boto3').addHandler(handler) logging.getLogger('botocore').addHandler(handler)

Do you think this would work?

therealvio · 2025-01-28T01:02:54Z

Thanks for the feedback thus far, I will go through these as I get time, worse comes to worse it may not be until next week if that's cool?

therealvio · 2025-01-30T06:59:52Z

Mind taking a peek at test_replace_route.py and adding a unit test?

I may be missing something, was there something in particular you were looking for me to add?

Full transparency here: I never got the test suite to run, mostly because I never did testing in the Python ecosystem before. I am more than keen to check out any material that would be of help here. As long as I can get it running, I can extend the test suite without a fuss :)

For what it is worth: I mostly tested through running the SAM app locally, which served me well enough for this PR.

bwhaley · 2025-02-04T22:04:11Z

The two things to add would be tests and the potential use of python-json-logger.

You can look at the GitHub workflow to see how the tests run. It's basically this

          pip install pip --upgrade
          pip install pyopenssl --upgrade
          pip install -r functions/replace-route/requirements.txt
          pip install -r functions/replace-route/tests/test_requirements.txt
          python -m pytest

therealvio · 2025-02-10T23:21:21Z

To keep you in the loop: I am going to be slammed next week, and I am not sure if I will get time to take a look this week.

This is still on my mind, it just may be a while before I get back to it.

This changes removes the addition of the eventPayload key that was imposed in the previous commit. I would rather leave the decision to the maintainers. Whether it be reverting *this* commit, or accepting this one, or leaving it to the future PR.

It's been a while since this branch and related item of work has been picked up. So we may as well update the dependencies and do a soft-restart here.

therealvio · 2025-11-11T04:23:47Z

Hey @bwhaley, I am picking this back up after some time. I have made changes based on commentary though what are you looking for in the tests specifically that you would like added? I am happy to put something in, though I am not sure what we are looking to test for specifically.

Are you looking for something like what is described in this stack overflow thread about testing logging output, testing the structlog configuration and asserting against it using a basic test with some snippets, or something else?

Thanks!

bwhaley

Left a couple more comments. Kicking off tests.

bwhaley · 2025-12-17T23:21:42Z

functions/replace-route/app.py

+)
+
+# logger is still needed to set the level for dependencies
 logger = logging.getLogger()


Yep, that makes sense.

.gitignore

bwhaley · 2025-12-17T23:30:02Z

functions/replace-route/app.py

-        logger.error(f"Unable to handle unknown event type: {json.dumps(event)}")
+        slogger.error("Unable to handle unknown event type: %s", json.dumps(event))


How does this appear in the logs? Is the JSON value of event legible in the output? This seems like an unlikely error, and I'm not sure how to easily test it, but I would be curious if it's readable.

bwhaley

The addition of structlog and orjson and python-json-logger actually adds a fair amount of bloat to the package. Currently, only boto3 is needed, which is included by default in the Lambda runtime. The others are not, so they need to be added as a layer to the Lambda function.

How did you test it?

While updating the logging to switch to structured logging, feedback was provided that identified a case of inaccurate and broken logging when `replace_route()` is invoked. Not only is the log message wrong by using a route table id instead of the route id, but also the use of the `new_route_table` variable would produce a dictionary in the log. The log line has now been updated to accurately state what is going on, and remove the use of a nested dictionary in the structured log. BREAKING CHANGE: Users monitoring this message with the original specific phrasing will experience broken monitors. It is recommended that users should be monitoring generally for ERROR level logs, or the "Failed connectivity tests! Replacing route" logs specifically.

therealvio commented Jan 15, 2025

View reviewed changes

functions/replace-route/app.py Outdated Show resolved Hide resolved

therealvio commented Jan 15, 2025

View reviewed changes

functions/replace-route/app.py Show resolved Hide resolved

therealvio commented Jan 15, 2025

View reviewed changes

functions/replace-route/app.py Outdated Show resolved Hide resolved

therealvio marked this pull request as ready for review January 15, 2025 07:59

therealvio requested a review from a team as a code owner January 15, 2025 07:59

therealvio mentioned this pull request Jan 15, 2025

Suggestion: Add Structured logging #76

Closed

therealvio changed the title ~~feat: use structured logging with structlog~~ feat: replace standard logging with structured logging Jan 15, 2025

therealvio mentioned this pull request Jan 16, 2025

feat: opt-in to expose successful connection logs #123

Open

bwhaley mentioned this pull request Jan 24, 2025

Triggers workflows on pull_request #124

Closed

bwhaley reviewed Jan 25, 2025

View reviewed changes

therealvio force-pushed the feat/use-structured-logging branch from 53c2e62 to 79ef09b Compare November 11, 2025 00:06

therealvio and others added 4 commits November 11, 2025 14:36

feat: use structured logging with structlog

cc48c99

fix: use slogger as drop-in replacement

8630a08

This changes removes the addition of the eventPayload key that was imposed in the previous commit. I would rather leave the decision to the maintainers. Whether it be reverting *this* commit, or accepting this one, or leaving it to the future PR.

Trigger tests

73e68cb

chore: update structlog dependenices

c527700

It's been a while since this branch and related item of work has been picked up. So we may as well update the dependencies and do a soft-restart here.

therealvio force-pushed the feat/use-structured-logging branch from 2bf4e2a to 670c84b Compare November 11, 2025 04:24

therealvio requested a review from bwhaley November 11, 2025 04:24

bwhaley reviewed Dec 17, 2025

View reviewed changes

bwhaley reviewed Dec 18, 2025

View reviewed changes

therealvio added 4 commits December 18, 2025 19:07

fix: use json-logger to for stdlib logging in deps

1e85e79

style: capital letters for start of commentary

ac86244

docs: clarify intention of slogger variable

c7a9226

therealvio force-pushed the feat/use-structured-logging branch from 670c84b to 258b403 Compare December 18, 2025 08:08

	slogger.info("Replacing existing route %s for route table %s", route_table_id, new_route_table)
	slogger.info("Updating route table %s to use NAT Gateway ID %s", route_table_id, nat_gateway_id)

		logger.error(f"Unable to handle unknown event type: {json.dumps(event)}")
		slogger.error("Unable to handle unknown event type: %s", json.dumps(event))

feat: replace standard logging with structured logging #122

Are you sure you want to change the base?

feat: replace standard logging with structured logging #122

Uh oh!

Conversation

therealvio commented Jan 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose 🎯

Context 🧠

Notes 📓

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

therealvio Jan 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

therealvio Jan 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bwhaley commented Jan 21, 2025

Uh oh!

bwhaley left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bwhaley Jan 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

therealvio Jan 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bwhaley Feb 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

therealvio commented Jan 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

therealvio commented Jan 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bwhaley commented Feb 4, 2025

Uh oh!

therealvio commented Feb 10, 2025

Uh oh!

therealvio commented Nov 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bwhaley left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

therealvio commented Jan 15, 2025 •

edited

Loading

therealvio Jan 30, 2025 •

edited

Loading

therealvio Jan 30, 2025 •

edited

Loading

bwhaley Jan 25, 2025 •

edited

Loading

therealvio Jan 30, 2025 •

edited

Loading

bwhaley Feb 4, 2025 •

edited

Loading

therealvio commented Jan 28, 2025 •

edited

Loading

therealvio commented Jan 30, 2025 •

edited

Loading

therealvio commented Nov 11, 2025 •

edited

Loading

bwhaley left a comment •

edited

Loading

bwhaley left a comment •

edited

Loading