Monitoring Internal API Latency Using Uptime Kuma Push Monitors

In modern microservice architectures, monitoring is often hindered by network segmentation. While external services are easily probed by public-facing monitoring tools, internal services—those residing deep within a private VPC or behind a firewall—are often invisible to standard HTTP-based uptime checkers. If you are struggling to track the latency of these internal components, the Uptime Kuma push monitor provides a robust, developer-friendly solution to bridge this visibility gap. This guide explores the architectural shift required to move from pull-based monitoring to push-based heartbeats, ensuring your internal infrastructure remains observable without compromising security.

Understanding the Push Monitoring Pattern

Traditional monitoring relies on a "pull" model, where a central server periodically sends requests to your service to check its status. This requires the monitoring server to have network access to your service. In a secure, production-grade environment, this is often impossible due to strict ingress rules. When services are isolated in private subnets, allowing inbound traffic from a monitoring server requires opening firewall ports, which increases the attack surface.

An Uptime Kuma push monitor reverses this relationship. Instead of the monitor reaching out to the service, the service (or a sidecar process) sends a "heartbeat" signal to Uptime Kuma. This heartbeat is a simple HTTP GET request. If Uptime Kuma does not receive this signal within a specified timeframe, it marks the service as down. This pattern is ideal for internal microservices, scheduled cron jobs, or background workers that only need to report their health periodically. By shifting the responsibility to the client, you maintain a strict security posture where only outbound traffic is permitted from your private application layer.

Prerequisites and Setup

Before implementing this, ensure you have a running instance of Uptime Kuma. If you haven't deployed it yet, refer to our guide on setup-uptime-kuma-docker-compose. We assume you are running your microservices in a Docker environment. The goal is to have your application code—or a small script running alongside it—send a signal to the Uptime Kuma API whenever a critical task completes or at a regular interval. You will need network connectivity between your application containers and the Uptime Kuma instance, typically managed via a shared Docker network or an internal load balancer.

Configuring the Push Monitor in Uptime Kuma

1. Log into your Uptime Kuma dashboard.
2. Click "Add New Monitor".
3. Select "Push" as the monitor type.
4. Give it a descriptive name, such as auth-service-latency-check.
5. Set the "Heartbeat Interval". This is the frequency at which you expect to receive a signal. For latency tracking, you might set this to 60 seconds. If a heartbeat is missed, the monitor will trigger an alert based on your notification settings.
6. Save the monitor. Uptime Kuma will generate a unique "Push URL" containing a token. This URL is your endpoint for reporting status. Keep this URL secure; while it is not a full authentication credential, it is the only thing required to update the status of that specific monitor.

Integrating the Heartbeat into Your Microservice

To measure latency, you need to calculate the time taken for a specific operation and include that duration in the push request. Uptime Kuma supports appending a msg or ping parameter to the push URL to track response times. This allows you to treat the push monitor as a time-series data point rather than just a binary up/down state.

Example: Python Implementation

If your microservice is written in Python, you can use the requests library to send the heartbeat. Here is a robust implementation that measures the execution time of a database query and reports it back to the monitor.

import time
import requests
import logging

# Configure logging for better observability
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

def perform_task_and_report():
    start_time = time.time()
    
    try:
        # Simulate your internal API logic
        # result = db.query("SELECT 1")
        time.sleep(0.5) # Simulated latency
        
        duration_ms = int((time.time() - start_time) * 1000)
        
        # Uptime Kuma Push URL
        push_url = "http://uptime-kuma:3001/api/push/YOUR_TOKEN_HERE"
        
        # Append the ping parameter to track latency
        params = {"ping": duration_ms, "msg": "OK"}
        
        response = requests.get(push_url, params=params, timeout=5)
        response.raise_for_status()
        logger.info(f"Heartbeat sent: {duration_ms}ms")
        
    except Exception as e:
        logger.error(f"Failed to send heartbeat: {e}")

if __name__ == "__main__":
    perform_task_and_report()

By including the ping parameter, Uptime Kuma automatically charts the latency data in the dashboard, allowing you to visualize performance trends over time without exposing your service to the internet.

Handling Trade-offs and Reliability

While the push model is excellent for internal services, it introduces a dependency: your service must be able to reach the Uptime Kuma instance. If the network path between your service and Uptime Kuma is unstable, you will receive false-positive alerts. This is a common failure mode in distributed systems where the monitoring infrastructure itself becomes a single point of failure.

To make this setup more resilient:

• Retry Logic: Always wrap your heartbeat request in a retry loop. If the first attempt fails due to a transient network blip, a second attempt 5 seconds later can prevent an unnecessary alert. Use exponential backoff to avoid overwhelming the network during a recovery phase.
• Timeouts: Set a strict timeout on your heartbeat request. You do not want your main application thread to hang because the monitoring server is slow to respond. A timeout of 2-5 seconds is generally sufficient for internal network calls.
• Asynchronous Execution: If possible, send the heartbeat in a background thread or a separate process. The monitoring signal should never block the primary execution path of your microservice. Using a task queue like Celery or a simple background thread ensures that your API latency is not artificially inflated by the monitoring logic itself.
• Circuit Breaking: If your application is under extreme load, consider skipping the heartbeat reporting to prioritize actual user requests. You can implement a simple circuit breaker that stops sending heartbeats if the application's internal error rate exceeds a threshold.

Monitoring Internal API Latency at Scale

When managing dozens of microservices, manually creating push monitors becomes tedious and error-prone. You can automate this by using the Uptime Kuma API to provision monitors programmatically during your CI/CD pipeline. By creating a script that interacts with the Uptime Kuma API, you can ensure that every new service deployed to your cluster has a corresponding monitoring endpoint ready to receive data.

Furthermore, consider the impact on your network traffic. If you have hundreds of services sending heartbeats every minute, the Uptime Kuma instance will experience significant load. Ensure your Uptime Kuma container has sufficient CPU and memory resources. You might also consider grouping services that share a lifecycle to reduce the number of individual monitors, though this comes at the cost of granularity.

Diagnostics: Troubleshooting Failed Heartbeats

If your monitor shows as "Down" even though your service is running, follow these diagnostic steps to isolate the issue:

1. Check Connectivity: From inside the container running your microservice, try to curl the Uptime Kuma push URL. If it fails, you have a network or firewall issue. Verify that the Docker network configuration allows traffic between the service container and the Uptime Kuma container.
2. Verify the Token: Ensure the token in your application matches the one generated by Uptime Kuma exactly. A single missing character will cause the request to be rejected, often resulting in a 404 or 401 error code.
3. Inspect Logs: Check the logs of your Uptime Kuma container. It will often log unauthorized or malformed requests, which can help you identify if the payload is being sent incorrectly. Use docker logs -f uptime-kuma to see real-time interaction data.
4. Check Clock Skew: If you are using time-sensitive authentication or logic, ensure your container clocks are synchronized via NTP. Significant drift can cause issues with timestamp-based validation if you implement custom security layers.
5. Check Resource Constraints: If your Uptime Kuma instance is under heavy load, it might drop incoming requests. Check the container's CPU and memory usage using docker stats to ensure it is not being throttled.

Advanced Considerations: Security and Data Integrity

When you move monitoring to a push model, you are essentially opening a "write" path into your monitoring system. While Uptime Kuma is generally secure, it is important to treat the push URL as a sensitive credential. If an attacker gains access to your push URL, they could send fake "UP" signals, masking a service outage. To mitigate this, consider placing your Uptime Kuma instance behind a VPN or an internal reverse proxy that requires authentication for all incoming requests.

Additionally, consider the data format. The ping parameter is useful, but it only provides a single data point. If you need more complex metrics, such as memory usage, disk I/O, or custom business metrics, you might need a more robust observability stack like Prometheus and Grafana. However, for simple uptime and latency checks, the push monitor pattern remains the most accessible and easiest to maintain. It strikes a balance between complexity and utility that is perfect for most small to medium-sized internal infrastructure projects.

Conclusion

The push monitor pattern is a powerful tool for developers who need to monitor internal services without compromising security. By shifting the responsibility of "checking" from the monitor to the service itself, you gain deep visibility into the performance of your private infrastructure. Remember to implement robust error handling in your heartbeat logic to ensure that your monitoring system remains a source of truth rather than a source of noise. With proper configuration and automated provisioning, this approach can scale to support complex microservice environments, providing the insights you need to maintain high availability and performance.