Repository navigation
Bug: MDC / fluent key-value entries named message, level, timestamp or error produce duplicate JSON members (LambdaJsonEncoder) #2724
Description
Expected Behaviour
The JSON emitted by the Logback LambdaJsonEncoder should contain each of the four core fields (level, message, timestamp, error) at most once, with the value produced by the logger. MDC entries or SLF4J fluent addKeyValue(...) entries that use one of these four names should be skipped, consistent with how StructuredArguments already treats them (RESERVED_KEYS, from #1759 / #1822).
Context: I read the #1822 discussion where MDC was intentionally left unfiltered because Powertools itself writes its context keys (function_name, service, …) into the MDC and cannot tell its own MDC.put from the user's. That constraint does not apply to these four keys: Powertools never puts level, message, timestamp or error into the MDC, so skipping them there is unambiguous. This report is limited to those four names; it does not propose blocking other user keys.
Current Behaviour
Location (aws-powertools/powertools-lambda-java @ default branch 6455f45):
LambdaJsonEncoder.java:75-93—encodewriteslevelandmessageitself, then merges MDC entries and fluent key-values into one map.JsonUtils.java:57-63—serializeMDCEntriesonly skipsPowertoolsLoggedFields(Lambda context keys).
Current code:
// LambdaJsonEncoder.encode
serializer.writeStringField(LEVEL_ATTR_NAME, event.getLevel().toString());
serializer.writeRaw(',');
serializer.writeStringField(FORMATTED_MESSAGE_ATTR_NAME, event.getFormattedMessage());
serializeException(event, serializer);
TreeMap<String, String> sortedMap = new TreeMap<>(event.getMDCPropertyMap());
sortedMap.putAll(getKeyValuePairs(event));
serializePowertools(sortedMap, serializer);
serializeMDCEntries(sortedMap, serializer);
// JsonUtils.serializeMDCEntries
for (Map.Entry<String, String> entry : sortedMap.entrySet()) {
if (!PowertoolsLoggedFields.stringValues().contains(entry.getKey())) {
serializeMDCEntry(entry, serializer);
}
}Problem: PowertoolsLoggedFields does not include the four core names, so an MDC or fluent key-value entry named level, message, timestamp or error is written as a second member with the same name:
{"level":"INFO","message":"test message","level":"HACKED","timestamp":"2026-10-29T15:35:00.799Z"}
{"level":"INFO","message":"original message","message":"overwritten value","timestamp":"2026-10-29T15:35:00.801Z"}
{"level":"INFO","message":"test message","timestamp":"one year ago","timestamp":"2026-10-29T15:35:00.809Z"}RFC 8259 §4 says object member names SHOULD be unique and that behaviour with duplicates is unpredictable; in practice parsers differ on whether the first or the last value wins, so the effective level/message depends on the consumer. MDC is also commonly populated by third-party libraries, not just application code.
The docs warn against reserved keys in MDC and say such keys "will overwrite the context set by the Logger"; for these four keys the actual result is a duplicate member rather than an overwrite.
Code snippet
MDC.put("level", "HACKED");
LambdaJsonEncoder encoder = new LambdaJsonEncoder();
Logger logger = (Logger) LoggerFactory.getLogger("demo");
LoggingEvent event = new LoggingEvent("fqcn", logger, Level.INFO, "test message", null, null);
String json = new String(encoder.encode(event), StandardCharsets.UTF_8);
assertThat(json).contains("\"level\":\"INFO\"").doesNotContain("HACKED"); // fails
// same for MDC.put("message", ...), MDC.put("timestamp", ...), MDC.put("error", ...)Steps to Reproduce
Reproduction
- Check out
mainatd767ff5. - Put the snippet above in a JUnit test in
powertools-logging/powertools-logging-logback(callMDC.clear()before/after). mvn -pl powertools-logging/powertools-logging-logback -am test -Dtest=<TestClass> -Dsurefire.failIfNoSpecifiedTests=false
Failing output:
Expecting actual:
"{"level":"INFO","message":"test message","error":"fake error","timestamp":"2026-10-29T15:35:00.784Z"}
...
Expecting actual:
"{"level":"INFO","message":"test message","level":"HACKED","timestamp":"2026-10-29T15:35:00.799Z"}
...
Expecting actual:
"{"level":"INFO","message":"original message","message":"overwritten value","timestamp":"2026-10-29T15:35:00.801Z"}
...
Expecting actual:
"{"level":"INFO","message":"test message","timestamp":"one year ago","timestamp":"2026-10-29T15:35:00.809Z"}Environment:
- Commit:
d767ff5(main); currentmain6455f45differs only by docs dependency bumps and apowertools-batchMultiThreadMDCchange, so the code paths above are unchanged. - Toolchain: OpenJDK 25.0.5, Apache Maven 3.9.16 (unit test, not deployed to Lambda)
I only exercised the Logback encoder; I have not checked whether the Log4j PowertoolsResolver has the same gap for these four keys.
Possible Solution
Already fixed in aws-powertools/powertools-lambda-typescript
PR: aws-powertools/powertools-lambda-typescript#3553 — "fix(logger): prevent overwriting standard keys" (merged 2025-02-05T15:59:41Z, merge commit f0bdf3cdb1f).
The TypeScript Logger had the same gap: user-supplied attributes could clash with the standard keys. The fix defines a ReservedKeys list and drops (with a warning) any user attribute that uses one of those names, whether it comes from appended keys, the main input object or extra input objects. Its list is broader (sampling_rate, service); this report only asks for the four core keys.
How powertools-lambda-typescript fixed it:
--- a/packages/logger/src/constants.ts
+++ b/packages/logger/src/constants.ts
@@ -37,4 +37,17 @@ const LogLevelThreshold = {
SILENT: 28,
} as const;
-export { LogJsonIndent, LogLevel, LogLevelThreshold };
+/**
+ * Reserved keys that are included in every log item when using the default log formatter.
+ *
+ * These keys are reserved and cannot be overwritten by custom log attributes.
+ */
+const ReservedKeys = [
+ 'level',
+ 'message',
+ 'sampling_rate',
+ 'service',
+ 'timestamp',
+];
+
+export { LogJsonIndent, LogLevel, LogLevelThreshold, ReservedKeys };--- a/packages/logger/src/Logger.ts
+++ b/packages/logger/src/Logger.ts
@@ -688,70 +700,118 @@ class Logger extends Utility implements LoggerInterface {
...
+ #processExtraObject(
+ item: Record<string, unknown>,
+ additionalAttributes: LogAttributes
+ ): void {
+ for (const [key, value] of Object.entries(item)) {
+ if (!this.#checkReservedKeyAndWarn(key)) {
+ additionalAttributes[key] = value;
+ }
+ }
+ }
+
+ /**
+ * Check if a given key is reserved and warn the user if it is.
+ *
+ * @param key - The key to check
+ */
+ #checkReservedKeyAndWarn(key: string): boolean {
+ if (ReservedKeys.includes(key)) {
+ this.warn(`The key "${key}" is a reserved key and will be dropped.`);
+ return true;
+ }
+ return false;
}Suggested fix
LambdaJsonEncoder now drops MDC and SLF4J fluent key-value entries named level, message, timestamp or error before writing them, like StructuredArguments does. The filter lives in the encoder rather than in the shared serializeMDCEntries, so the ECS encoder output is unchanged, and the MDC warning in the docs now mentions it:
diff --git a/docs/core/logging.md b/docs/core/logging.md
--- a/docs/core/logging.md
+++ b/docs/core/logging.md
@@ -744,7 +744,7 @@ Mapped Diagnostic Context (MDC) is essentially a Key-Value store. It is supporte
Always set additional keys as part of your handler method to ensure they have the latest value, or explicitly clear them with [`clearState=true`](#clearing-state).
???+ warning "Do not add reserved keys to MDC"
- Avoid adding any of the keys listed in [standard structured keys](#standard-structured-keys) and [additional structured keys](#additional-structured-keys) to your MDC. This may cause unindented behavior and will overwrite the context set by the Logger. Unlike with `StructuredArguments`, the Logger will **not** ignore reserved keys set via MDC.
+ Avoid adding any of the keys listed in [standard structured keys](#standard-structured-keys) and [additional structured keys](#additional-structured-keys) to your MDC. This may cause unindented behavior and will overwrite the context set by the Logger. Unlike with `StructuredArguments`, the Logger will **not** ignore reserved keys set via MDC, except `level`, `message`, `timestamp` and `error`, which the Logback `LambdaJsonEncoder` ignores in MDC and SLF4J fluent key-value entries.
### Removing additional keys
diff --git a/powertools-logging/powertools-logging-logback/src/main/java/software/amazon/lambda/powertools/logging/logback/LambdaJsonEncoder.java b/powertools-logging/powertools-logging-logback/src/main/java/software/amazon/lambda/powertools/logging/logback/LambdaJsonEncoder.java
--- a/powertools-logging/powertools-logging-logback/src/main/java/software/amazon/lambda/powertools/logging/logback/LambdaJsonEncoder.java
+++ b/powertools-logging/powertools-logging-logback/src/main/java/software/amazon/lambda/powertools/logging/logback/LambdaJsonEncoder.java
@@ -44,6 +44,8 @@ public class LambdaJsonEncoder extends EncoderBase<ILoggingEvent> {
protected static final String EXCEPTION_CLASS_ATTR_NAME = "name";
protected static final String EXCEPTION_STACK_ATTR_NAME = "stack";
protected static final String EXCEPTION_ATTR_NAME = "error";
+ private static final Set<String> RESERVED_ATTR_NAMES = Set.of(LEVEL_ATTR_NAME, FORMATTED_MESSAGE_ATTR_NAME,
+ TIMESTAMP_ATTR_NAME, EXCEPTION_ATTR_NAME);
private final ThrowableProxyConverter throwableProxyConverter = new ThrowableProxyConverter();
protected ThrowableHandlingConverter throwableConverter = null;
@@ -80,6 +82,8 @@ public class LambdaJsonEncoder extends EncoderBase<ILoggingEvent> {
TreeMap<String, String> sortedMap = new TreeMap<>(event.getMDCPropertyMap());
sortedMap.putAll(getKeyValuePairs(event));
+ // the encoder writes these fields itself, MDC or key-value entries would duplicate them
+ sortedMap.keySet().removeAll(RESERVED_ATTR_NAMES);
serializePowertools(sortedMap, serializer);
serializeMDCEntries(sortedMap, serializer);Verified on d767ff5 (OpenJDK 25.0.5, Maven 3.9.16): the test above (one case per key, plus one with the four names as fluent key-value pairs) fails without this patch and passes with it; no other test results change.
I can submit a PR for this.
Powertools for AWS Lambda (Java) version
2.11.0 (main at d767ff5; same code on current main 6455f45)
AWS Lambda function runtime
Java 25 (reproduced as a unit test on OpenJDK 25.0.5, Apache Maven 3.9.16 — not deployed to Lambda)
Debugging logs
See the failing output under Steps to Reproduce.
Related issues checked
Checked: #1759 / #1822 added reserved keys to StructuredArguments and deliberately left MDC out because of Powertools' own MDC keys — this report covers only the four core keys that constraint does not apply to; #2377 added fluent key-value support by merging pairs into the MDC map without filtering core keys.
Activity