← All platforms

Apache Beam error tracking

A pipeline runs across machines. Its Python errors can still have one home. The Sentry Beam integration sends them to Telebugs on your server.

~/hoofnotes  pip install sentry-sdk
~/hoofnotes  nvim pipeline.py
import sentry_sdk
from sentry_sdk.integrations.beam import BeamIntegration
sentry_sdk.init(
# Copy the complete Telebugs DSN, key and path included.
dsn="YOUR_TELEBUGS_DSN",
integrations=[BeamIntegration()],
)
from hoofnotes.pipeline import run
run()
~/hoofnotes  python pipeline.py --requirements_file requirements.txt

Hang on, I already use Sentry

Keep sentry-sdk and change its destination to your Telebugs DSN. Your error reports stay on your server. You don’t need a Sentry account.

What part of Beam does this cover?

The Python integration captures exceptions in ParDo lifecycle methods. It is an experimental Sentry integration, not a promise to capture every failure from every runner.

What about remote workers?

They need the SDK too. Telebugs can receive their error reports, but a connection on the machine launching the pipeline alone doesn’t cover all workers.