Netdata Zulip Bot - Development Instructions#

This repository implements a Zulip bot that receives incoming webhook notifications from Netdata Cloud and posts the resulting notifications to a Zulip topic.

Core Requirements#

Technology Stack#

• Language: Python with uv package manager
• Framework: Python zulip_bots PyPI package for Zulip integration
• Web Server: FastAPI for webhook endpoint
• Deployment: Standalone service

Netdata Integration#

• Webhook Format: Follow the Netdata Cloud webhook notification format from:
  • URL: https://raw.githubusercontent.com/netdata/netdata/refs/heads/master/integrations/cloud-notifications/metadata.yaml
  • Section: id: notify-cloud-webhook
• Notification Types: Handle both alert notifications and reachability notifications

Zulip Configuration#

• Configuration Source: The .zuliprc file should contain:
  • Zulip site URL
  • Bot email and API key
  • Target stream/channel for posting messages
• Topic Organization: Messages should be posted to topics based on severity:
  • critical, warning, clear for alerts
  • reachability for host status changes
• Message Format: Rich markdown with:
  • Alert details and timestamps
  • Markdown-formatted alert URLs for easy access to Netdata Cloud

Security Requirements#

• TLS/HTTPS: The service must listen on HTTPS (not HTTP)
• Let's Encrypt: Use Let's Encrypt to automatically issue SSL certificates for the public hostname
• Mutual TLS: Netdata uses mutual TLS for authentication
  • The server must validate Netdata's client certificate
  • Support configuration of client CA certificate path

Service Architecture#

• Standalone Service: Run as an independent service
• Webhook Endpoint: Expose /webhook/netdata for receiving notifications
• Health Check: Provide /health endpoint for monitoring
• Structured Logging: Use JSON-structured logs for production monitoring

Implementation Notes#

Configuration Management#

• Support both .zuliprc file and environment variables
• Provide sample configuration files with --create-config flag
• Server configuration via environment variables:
  • SERVER_DOMAIN: Public domain for Let's Encrypt
  • SERVER_PORT: HTTPS port (default: 8443)
  • SERVER_ENABLE_MTLS: Enable mutual TLS
  • SERVER_CLIENT_CA_PATH: Path to Netdata client CA

Message Processing#

1. Receive Netdata webhook POST request
2. Validate and parse JSON payload
3. Determine notification type (alert vs reachability)
4. Format message with appropriate emoji and markdown
5. Send to configured Zulip stream/topic

Error Handling#

• Validate all incoming payloads
• Log errors without exposing internal details
• Return appropriate HTTP status codes
• Implement retry logic for Zulip API failures

Testing#

Run tests with:

uv run python -m pytest tests/ -v

Deployment#

The service should be deployable via:

• Systemd service (see examples/netdata-zulip-bot.service)
• Docker container (see Dockerfile and docker-compose.yml)
• Automated setup script (scripts/setup.sh)

Development Commands#

• Install dependencies: uv sync
• Create config samples: uv run netdata-zulip-bot --create-config
• Run tests: uv run python -m pytest tests/
• Start service: uv run netdata-zulip-bot

Important Reminders#

• Always validate Netdata webhook payloads before processing
• Ensure SSL certificates are properly configured before production deployment
• Test mutual TLS authentication with actual Netdata Cloud webhooks
• Monitor service logs for webhook processing errors
• Keep Zulip API credentials secure and never commit them to the repository