Skip to content

Feature request: Opt-in diagnostic when a metric dimension key is set again with a different value #2733

Open

Description

@hidingmyname

Use case

In larger handlers several helpers may add dimensions to the shared Metrics instance. If two of them use the same key — e.g. default dimensions contain Environment=Production and a later helper calls addDimension("Environment", "Staging") — the later value is used. That is legitimate behaviour, but when a dashboard suddenly shows Staging, it is hard to find which call replaced the value: nothing is logged.

Metrics metrics = MetricsFactory.getMetricsInstance();
metrics.setNamespace("MyNamespace");
metrics.setDefaultDimensions(DimensionSet.of("Environment", "Production"));

// elsewhere, e.g. in a shared helper
metrics.addDimension("Environment", "Staging");   // no indication that an existing key was replaced

Location (aws-powertools/powertools-lambda-java @ default branch 6455f45):

Current code:

    public DimensionSet addDimension(String key, String value) {
        validateDimension(key, value);

        if (dimensions.size() >= MAX_DIMENSION_SET_SIZE) {
            throw new IllegalStateException(
                    "Cannot exceed " + MAX_DIMENSION_SET_SIZE + " dimensions per dimension set");
        }

        dimensions.put(key, value);
        return this;
    }

Problem: Replacing an existing dimension value is silent, so there is nothing to trace back an unexpected value to the call that set it. Last-write-wins itself is correct and documented; only the missing diagnostic is in question.

Reproduction: with the snippet above on main at d767ff5, the only Powertools log line is the unrelated "No metrics were emitted" warning (no metric was added in that test). Current main (6455f45) differs only by docs dependency bumps and a powertools-batch MultiThreadMDC change, so the code paths above are unchanged.

Solution/User Experience

Already implemented in aws-powertools/powertools-lambda-typescript

PR: aws-powertools/powertools-lambda-typescript#3352 — "feat(metrics): warn when overwriting dimension" (merged 2024-11-25T08:55:50Z, merge commit 12f3e448d81).

The TypeScript Metrics utility now warns when addDimension is called with a name that already exists in the dimensions or default dimensions, and routes addDimensions through the same check. Python has the equivalent warning (cloudwatch.py#L323-L328). Those runtimes warn by default; for Java I'm proposing opt-in to avoid adding noise to existing functions — your call.

How powertools-lambda-typescript implemented it:

--- a/packages/metrics/src/Metrics.ts
+++ b/packages/metrics/src/Metrics.ts
@@ -239,6 +239,14 @@ class Metrics extends Utility implements MetricsInterface {
         `The number of metric dimensions must be lower than ${MAX_DIMENSION_COUNT}`
       );
     }
+    if (
+      Object.hasOwn(this.dimensions, name) ||
+      Object.hasOwn(this.defaultDimensions, name)
+    ) {
+      this.#logger.warn(
+        `Dimension "${name}" has already been added. The previous value will be overwritten.`
+      );
+    }
     this.dimensions[name] = value;
   }

Suggested fix

Add an opt-in POWERTOOLS_METRICS_WARN_ON_DIMENSION_OVERWRITE environment variable: when it is true, EmfMetricsLogger.addDimension logs a warning the first time (per execution environment) a key that is already present as a default or custom dimension is set to a different value; last-write-wins and the default behaviour stay unchanged, same-value sets are not reported, and the setting is listed in the metrics docs:

diff --git a/docs/core/metrics.md b/docs/core/metrics.md
--- a/docs/core/metrics.md
+++ b/docs/core/metrics.md
@@ -119,6 +119,7 @@ Metrics has three global settings that will be used across all metrics emitted.
 | **Service**          | Optionally, sets **service** metric dimension across all metrics e.g. `payment` | `POWERTOOLS_SERVICE_NAME`          | `service`           |
 | **Function name**    | Function name used as dimension for the cold start metric                       | `POWERTOOLS_METRICS_FUNCTION_NAME` | `functionName`      |
 | **Disable Metrics**  | Optionally, disables all metrics flushing                                       | `POWERTOOLS_METRICS_DISABLED`      | N/A                 |
+| **Warn on dimension overwrite** | Optionally, logs a warning (once per key) when a dimension is set again with a different value | `POWERTOOLS_METRICS_WARN_ON_DIMENSION_OVERWRITE` | N/A |
 
 !!! tip "Use your application or main service as the metric namespace to easily group all metrics"
 
diff --git a/powertools-metrics/src/main/java/software/amazon/lambda/powertools/metrics/internal/EmfMetricsLogger.java b/powertools-metrics/src/main/java/software/amazon/lambda/powertools/metrics/internal/EmfMetricsLogger.java
--- a/powertools-metrics/src/main/java/software/amazon/lambda/powertools/metrics/internal/EmfMetricsLogger.java
+++ b/powertools-metrics/src/main/java/software/amazon/lambda/powertools/metrics/internal/EmfMetricsLogger.java
@@ -20,6 +20,8 @@ import java.time.Instant;
 import java.util.HashMap;
 import java.util.LinkedHashMap;
 import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
 import java.util.concurrent.atomic.AtomicBoolean;
 import java.util.function.Consumer;
 
@@ -45,6 +47,10 @@ public class EmfMetricsLogger implements Metrics {
     private static final Logger LOGGER = LoggerFactory.getLogger(EmfMetricsLogger.class);
     private static final String COLD_START_METRIC = "ColdStart";
     private static final String METRICS_DISABLED_ENV_VAR = "POWERTOOLS_METRICS_DISABLED";
+    private static final String WARN_ON_DIMENSION_OVERWRITE_ENV_VAR =
+            "POWERTOOLS_METRICS_WARN_ON_DIMENSION_OVERWRITE";
+    // Dimension keys already reported as overwritten, to warn at most once per key per execution environment
+    private static final Set<String> REPORTED_OVERWRITTEN_DIMENSIONS = ConcurrentHashMap.newKeySet();
 
     private final software.amazon.cloudwatchlogs.emf.logger.MetricsLogger emfLogger;
     private final EnvironmentProvider environmentProvider;
@@ -53,6 +59,7 @@ public class EmfMetricsLogger implements Metrics {
     private Map<String, String> defaultDimensions = new HashMap<>();
     private final Map<String, Object> properties = new HashMap<>();
     private final AtomicBoolean hasMetrics = new AtomicBoolean(false);
+    private final Map<String, String> customDimensions = new HashMap<>();
 
     public EmfMetricsLogger(EnvironmentProvider environmentProvider, MetricsContext metricsContext) {
         this.emfLogger = new software.amazon.cloudwatchlogs.emf.logger.MetricsLogger(environmentProvider,
@@ -78,6 +85,8 @@ public class EmfMetricsLogger implements Metrics {
         dimensionSet.getDimensions().forEach((key, val) -> {
             try {
                 emfDimensionSet.addDimension(key, val);
+                warnIfDimensionOverwritten(key, val);
+                customDimensions.put(key, val);
             } catch (Exception e) {
                 // Ignore dimension errors
             }
@@ -107,6 +116,7 @@ public class EmfMetricsLogger implements Metrics {
                 // Ignore dimension errors
             }
         });
+        customDimensions.clear();
         emfLogger.setDimensions(emfDimensionSet);
         // Store a copy of the default dimensions
         this.defaultDimensions = new LinkedHashMap<>(dimensions);
@@ -145,6 +155,7 @@ public class EmfMetricsLogger implements Metrics {
     public void clearDefaultDimensions() {
         emfLogger.resetDimensions(false);
         defaultDimensions.clear();
+        customDimensions.clear();
     }
 
     @Override
@@ -173,6 +184,7 @@ public class EmfMetricsLogger implements Metrics {
     private void clearCustomDimensions() {
         // Reset all dimensions in the EMF logger
         emfLogger.resetDimensions(false);
+        customDimensions.clear();
 
         // Re-apply default dimensions if they exist
         if (!defaultDimensions.isEmpty()) {
@@ -188,6 +200,18 @@ public class EmfMetricsLogger implements Metrics {
         }
     }
 
+    private void warnIfDimensionOverwritten(String key, String value) {
+        if (!"true".equalsIgnoreCase(System.getenv(WARN_ON_DIMENSION_OVERWRITE_ENV_VAR))) {
+            return;
+        }
+        String previousValue = customDimensions.containsKey(key) ? customDimensions.get(key)
+                : defaultDimensions.get(key);
+        if (previousValue != null && !previousValue.equals(value) && REPORTED_OVERWRITTEN_DIMENSIONS.add(key)) {
+            LOGGER.warn("Dimension \"{}\" has already been added. The previous value (\"{}\") will be overwritten "
+                    + "with \"{}\".", key, previousValue, value);
+        }
+    }
+
     @Override
     public void captureColdStartMetric(Context context,
             software.amazon.lambda.powertools.metrics.model.DimensionSet dimensions) {

Verified on d767ff5 (OpenJDK 25.0.5, Maven 3.9.16): with POWERTOOLS_METRICS_WARN_ON_DIMENSION_OVERWRITE=true set in the test setup, a new DimensionOverwriteWarningTest (default dimension Environment=Production, then addDimension("Environment", "Staging")) fails without this patch and passes with it; no other test results change.

I can submit a PR for this.

Alternative solutions

- Always warn (as in Python/TypeScript): simplest, but changes log output for everyone who overrides on purpose.
- Strict mode that throws on duplicate keys: would break documented, legitimate overrides; not proposed.
- Users wrap Metrics in their own helper that checks before adding: possible, but every team reimplements it.

Acknowledgment

Acknowledgment

Related issues checked

Checked: #1863 (new metrics module), #2153/#2154 (flushSingleMetric/flushMetrics) and #1848 (metrics abstraction) do not discuss notifying on replaced dimensions.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions