| title | Troubleshooting Python automatic instrumentation issues |
|---|---|
| linkTitle | Troubleshooting |
| weight | 40 |
| cSpell:ignore | ASGI gunicorn uvicorn |
The Python package installs require gcc and gcc-c++, which you may need to
install if you’re running a slim version of Linux, such as CentOS.
{{< tabpane text=true >}} {{% tab "CentOS" %}}
yum -y install python3-devel
yum -y install gcc-c++{{% /tab %}} {{% tab "Debian/Ubuntu" %}}
apt install -y python3-dev
apt install -y build-essential{{% /tab %}} {{% tab "Alpine" %}}
apk add python3-dev
apk add build-base{{% /tab %}} {{< /tabpane >}}
{#bootstrap-using-uv}
Running opentelemetry-bootstrap -a install when using the
uv package manager may result in errored or
unexpected dependency setups.
Instead, you can generate OpenTelemetry requirements dynamically and install
them using uv.
First, install the appropriate packages (or add them to your project file and
run uv sync):
uv add opentelemetry-distro opentelemetry-exporter-otlpNow, you can install the auto instrumentation:
uv run opentelemetry-bootstrap -a requirements | uv add --requirement -Finally, use uv run to start your application (see
Configuring the agent):
uv run opentelemetry-instrument python myapp.pyPlease note that you have to reinstall the auto instrumentation every time you
run uv sync or update existing packages. It is therefore recommended to make
the installation part of your build pipeline.
The debug mode can be enabled in the Flask app like this:
if __name__ == "__main__":
app.run(port=8082, debug=True)The debug mode can break instrumentation from happening because it enables a
reloader. To run instrumentation while the debug mode is enabled, set the
use_reloader option to False:
if __name__ == "__main__":
app.run(port=8082, debug=True, use_reloader=False)A pre-fork server, such as Gunicorn with multiple workers, could be run like this:
gunicorn myapp.main:app --workers 4However, specifying more than one --workers may break the generation of
metrics when auto-instrumentation is applied. This is because forking, the
creation of worker/child processes, creates inconsistencies between each child
in the background threads and locks assumed by key OpenTelemetry SDK components.
Specifically, the PeriodicExportingMetricReader spawns its own thread to
periodically flush data to the exporter. See also issues
#2767 and
#3307.
After forking, each child seeks a thread object in memory that is not actually
run, and any original locks may not unlock for each child. See also forks and
deadlocks described in Python issue 6721.
There are some workarounds for pre-fork servers with OpenTelemetry. The following table summarizes the current support of signal export by different auto-instrumented web server gateway stacks that have been pre-forked with multiple workers. See below for more details and options:
| Stack with multiple workers | Traces | Metrics | Logs |
|---|---|---|---|
| Uvicorn | x | x | |
| Gunicorn | x | x | |
| Gunicorn + UvicornWorker | x | x | x |
To auto-instrument a server with multiple workers, it is recommended to deploy
using Gunicorn with uvicorn.workers.UvicornWorker if it is an Asynchronous
Server Gateway Interface (ASGI) app (FastAPI, Starlette, etc). The UvicornWorker
class is specifically designed to handle forks with preservation of background
processes and threads. For example:
opentelemetry-instrument gunicorn \
--workers 4 \
--worker-class uvicorn.workers.UvicornWorker \
--bind 0.0.0.0:8000 \
myapp.main:appInitialize OpenTelemetry inside the worker process with
programmatic auto-instrumentation
after the server fork, instead of with opentelemetry-instrument. For example:
from opentelemetry.instrumentation.auto_instrumentation import initialize
initialize()
from your_app import appIf using FastAPI, note that initialize() must be called before importing
FastAPI because of how instrumentation is patched. For example:
from opentelemetry.instrumentation.auto_instrumentation import initialize
initialize()
from fastapi import FastAPI
app = FastAPI()
@app.get("/")
async def root():
return {"message": "Hello World"}Then, run the server with:
uvicorn main:app --workers 2If using Gunicorn, then you can initialize OpenTelemetry with
programmatic auto-instrumentation
as part of your post-fork hooks, instead of with opentelemetry-instrument. For
example, in a file gunicorn_config.py that is alongside your_app.py:
from opentelemetry.instrumentation.auto_instrumentation import initialize
def post_fork(server, worker):
initialize()Then, run with:
gunicorn --config gunicorn_config.py --workers=2 your_app:appConsider using a recent version of
Prometheus to receive OTLP
metrics directly. Set up a PeriodicExportingMetricReader and one OTLP worker
per process to push to Prometheus server. We recommend not using
PrometheusMetricReader with forking -- see issue
#3747.
Alternatively, use a single worker in pre-fork with zero-code instrumentation:
opentelemetry-instrument gunicorn your_app:app --workers 1To debug Python gRPC connectivity issues, set the following gRPC debug environment variables:
export GRPC_VERBOSITY=debug
export GRPC_TRACE=http,call_error,connectivity_state
opentelemetry-instrument python YOUR_APP.py