How to Get Notified When a Jenkins Agent Goes Offline

Decide what should wake someone up

Not every offline agent is an incident. Before you pick a tool, sort the cases:

• Unexpected disconnects: the agent process died, the host rebooted or the network dropped. These should alert.
• Launch failures: Jenkins tried to start the agent and couldn't. These usually need a person too.
• Planned downtime: someone marked the agent temporarily offline for maintenance. You may want a quieter channel, or a note to the team first.
• Short-lived cloud agents: Kubernetes or EC2 agents that are removed on purpose. Alerting on each one is noise; whether enough agents with a label are online is usually the better signal.

Your options

What Jenkins does on its own

The Jenkins nodes documentation describes how Jenkins monitors each agent's disk space, temporary space, swap, clock and response time, and takes an agent offline when one of them crosses its threshold. That protects your builds, but Jenkins core sends no message when it happens. You see it on Manage Jenkins → Nodes or not at all.

Email per agent: Mail Watcher

The Mail Watcher plugin emails you when an agent goes offline, comes back online, or is marked temporarily offline or back online. You set the recipients on each agent's configuration page, under Node Properties. If plain email for a handful of agents is all you need, it's the simplest choice.

Your existing monitoring stack: Prometheus and Alertmanager

If your team already runs Prometheus, the Jenkins Prometheus plugin publishes Jenkins metrics at /prometheus/, and Alertmanager handles grouping, deduplication, routing and silences. This keeps Jenkins alerts with the rest of your infrastructure alerts, at the cost of writing and maintaining the alert rules yourself.

A polling script

A script can read the agent list from the Jenkins API (/computer/api/json) on a schedule and send a message when an agent shows as offline. It works, but you own every edge case: expired credentials, a login page instead of JSON, Jenkins restarting, agents that were deleted, outages shorter than the polling interval, and not sending the same alert every run. Don't run the check as a Jenkins job on the agent you're watching: when the agent dies, so does the check.

A plugin built for agent alerts: Worker Sentinel

The Worker Sentinel plugin for Jenkins sends agent alerts to Slack, PagerDuty, Twilio SMS or email, with a grace period, deduplication, flapping detection and routing rules built in. The rest of this guide uses it as the worked example.

How Worker Sentinel notices an offline agent

Worker Sentinel runs inside Jenkins and listens for Jenkins' own agent events: an agent going offline, being marked temporarily offline, or failing to launch. It also re-checks every agent every 5 minutes, so it catches anything an event missed. It doesn't probe your agent hosts itself: it reports what Jenkins sees.

After Jenkins notices the disconnect, Worker Sentinel waits for the grace period (60 seconds by default) before it alerts, so a brief network blip doesn't page anyone. The alert therefore arrives a little over a minute after Jenkins notices, not a minute after the agent actually failed.

Set it up, step by step

1. Install the plugin

Download the .hpi file from your OzenOps dashboard and upload it under Manage Jenkins → Plugins → Advanced settings, then restart Jenkins. See the installation section of the docs.

2. Validate the license

Go to Manage Jenkins → System, find the Ozen License section, paste your key into API Key and select Validate License. Without a valid license, Worker Sentinel keeps tracking agents but sends no external notifications, so check the status before you go further. The licensing section covers renewals.

3. Add a destination

A fresh install has the Enable Plugin box ticked in the Worker Sentinel Configuration section, but no destinations, so nothing is sent yet. Select Add Destination, pick Slack, PagerDuty, Twilio SMS or email, and fill in its credentials. Each channel has its own section in the docs: Slack, PagerDuty, Twilio SMS and email. Use the destination's test button (Test Connection, or Test Configuration for email) to check the credentials.

4. Check the timing settings

Under Alert Settings, the default grace period is 60 seconds (anywhere from 0 to 3600) and the deduplication window is 300 seconds. Under Flapping Detection, an agent that changes state 5 times within 300 seconds is treated as flapping. A destination can override the grace period and the deduplication window; left blank, it uses the global value. See global settings and notification destinations.

5. Decide on routing

With no enabled routing policy, every alert goes to every enabled destination. That's the right start. Add policies later to send certain agents, labels or severities to certain destinations, as described under routing policies. Once you add one, alerts it doesn't match no longer go everywhere.

6. Save, then test with a real outage

The test button proves the credentials, not the whole path. Select Save, then take a test agent offline and wait for the alert:

• The quick test: on the agent's page, mark it temporarily offline. Worker Sentinel treats that as down, so it alerts after the grace period. This proves detection, license, routing and delivery, but not how a crash is reported.
• The realistic test: stop the agent process or block its connection on a test agent. This is the path a real failure takes.

Then bring the agent back. Recovery notifications are on by default, and go to the destinations that were alerted about the outage.

If the alert doesn't arrive

• The agent came back inside the grace period. No outage alert is sent, and so no recovery alert either. That's by design.
• The license isn't valid. External notifications stop until it is.
• A routing policy didn't match. Once policies exist, check that one covers the agent's name or label and the alert's severity.
• Deduplication or flapping kicked in. A repeat alert within the deduplication window, or an agent that keeps bouncing, is held back on purpose.
• The message couldn't leave the controller. Check outbound network access to the provider, the credentials' permissions and, for email, the SMTP settings. Worker Sentinel sends from a bounded queue: if it's full, a send is dropped and logged as a warning in the Jenkins log.

What an in-Jenkins alert can't cover

• The controller itself. A plugin runs inside Jenkins, so if the controller is down, nothing inside it can tell you. Watch the controller from outside, with an uptime check on its URL.
• Root cause. The alert says the agent is offline and gives Jenkins' reason. It doesn't diagnose the network or the host behind it.
• Planned work. Marking an agent temporarily offline sends an alert too, so tell your team before maintenance or route those agents to a quieter channel.

Next steps

Read the full Worker Sentinel documentation for every setting, or see what the plugin includes and start a free trial on the Worker Sentinel product page.