Troubleshoot WorkQueue Issues Using Repairqueue Command on ESA

Introduction

This document describes the usage of the hidden CLI command repairqueue on the Cisco Email Security Appliance (ESA) and how to validate queue recovery.

Issue

Reasons to run the repairqueue command:

• Error stating that the workqueue is not mounted. This usually is the result of queue corruption after an improper power cycle or reboot of the appliance.
• A known defect can require this command as a workaround (such as Cisco bug ID CSCuw22284 - Email queue corrupts after a mail-processing service crash or improper shutdown).
• Application faults, such as those referencing gcq.py, or the queue management subsystem.
• Status Detail or workqueue > rate report negative numbers.

• Status or Status Detail reports an Oldest Message value that exceeds the configured bounce profile threshold.

  • Default threshold: 259200 seconds (3 days).
  • To check the threshold, enter bounceconfig > edit, then select the Default profile. Locate the line, Please enter the maximum number of seconds a message can stay in the queue before being hard bounced.

Reasons to not run the repairqueue command:

• Slow workqueue processing is not a valid reason to run a queue repair. Administrators often confuse slow workqueue processing with queue corruption. A slow workqueue usually is due to repeated processing of the same message, or of multiple messages, because a service is over-utilizing system resources. These repeated-processing scenarios are often not repaired by simply running repairqueue. Further troubleshooting of the affected service or services is required.

Environment

The information in this document is based on these software and hardware versions:

• ESA, all hardware and virtual appliances running AsyncOS 11.0.0-264 or newer

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. In live networks, confirm the potential impact of each command before proceeding.

Cisco recommends knowledge of these topics:

• System capacity, system monitoring, system health, and overall processing of messages through the ESA workqueue.
• Overall ESA administration.

Note: See the ESA User Guide or the ESA GUI Online Help for details.

Resolution

The CLI command repairqueue does not repair all workqueue issues or corruption. The utility attempts a best-effort repair of the workqueue.

1. Run status detail and confirm whether the workqueue shows symptoms already described in Issue, such as negative values or a queue mount problem.
2. Run workqueue and choose the Rate option. Confirm whether the reported rates support the trigger condition.
3. Compare the Oldest Message value in status or status detail with the configured bounce profile threshold. If the value exceeds the threshold, continue with the repair procedure.

Warning: Running repairqueue can result in loss of active messages in the workqueue.

4. Run repairqueue. The first process runs prompts for permission once to proceed and execute the repair:

   esa.example.com> repairqueue
   
   Do you want to repair or clean the queue?
   1. Repair.
   2. Clean.
   [1]> 1
   
   The mail flow will be stopped throughout the repair/cleanup process
   WARNING:
   This utility does a best effort to repair the queue.
   Not all queues corruptions can be repaired.
   Are you sure you want to proceed? [N]> y
   Checking generation checksum files
   ...
   <<<SNIP FOR BREVITY>>>
   ...
   done
   Repair succeeded
   Starting Hermes
   Hermes Started
   
   Log into the system and verify the status of the system.

   Note: On a virtual ESA, ignore the output, "Waiting for the queue to mount: Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory" documented under known defect Cisco bug ID CSCuz28415.

5. Once the repair process completes and the workqueue is repaired, the appliance retains an old checkpoint of the previous workqueue. To resume writing a new checkpoint for workqueue processing, run repairqueue again, and enter the command to Clean:

   esa.example.com> repairqueue
   
   Do you want to repair or clean the queue?
   1. Repair.
   2. Clean.
   [1]> 2
   
   The mail flow will be stopped throughout the repair/cleanup process
   WARNING:
   There is a backup found this may be the only backup.
   This will remove the old queue.
   Are you sure you want to proceed? [N]> y
   Double confirmation. Are you sure you want to proceed? [N]> y
   Removing old queue
   Cleanup finished

6. Run the status detail command from the CLI, or choose Monitor > System Status in the GUI. Confirm the system status is Online.
7. Review the mail logs to confirm mail processing. From the CLI, run the tail mail_logs command.
8. Run the workqueue command and choose the Rate option (default 10 seconds). Confirm the In and Out rates are reasonable for the current mail flow. If the workqueue has a large backlog, it can take time to drain.

Cause

Common causes of workqueue corruption include:

1. Unexpected reboots or improper shutdowns (for example, power interruptions). Depending on what backend processes are running at the time, the queue can become corrupt or fail to mount after reboot. In this case, the CLI can report errors such as:

   esa.example.com> status
   
   Enter "status detail" for more information.
   
   Couldn't obtain mail stats - my.esa: The daemon is not responding.

   esa.example.com> status
   
   Enter "status detail" for more information.
   
   Couldn't obtain mail stats - the queue is not mounted

2. Excessive memory (RAM) utilization. This is commonly associated with listener or mail flow policy settings that allow too many inbound connections or injections. Review listenerconfig and confirm the maximum inbound connections value is appropriate for the deployment.

FAQ

What if my ESA is not running 11.0.0-264 or newer?

Users who have appliances running older versions of AsyncOS that do not have the repairqueue hidden CLI command option can open a support case in order to have a Cisco Support engineer assist with the procedure. A support tunnel needs to be opened and available for Cisco Support to access the appliance and run the repairqueue command. Contact Cisco Support to open an active support case.

Does workqueue corruption mean mail loss?

In most cases, corruption does not equal mail loss. The queue is corrupt due to meta-data related to message processing that is no longer on the appliance. This is record-keeping for processing between the queue, reporting, and message tracking. Running repairqueue rebuilds the ESA meta-data and cleans up any misreporting between the services and processing.

Are there any repercussions to workqueue corruption?

The ESA can run for a long time on a corrupted queue, and most messages still process. However, the appliance can appear sluggish, or some messages can never clear, as indicated by the Oldest Message value in the status command (significantly older than the bounceconfig threshold). When AsyncOS restarts with a corrupted queue, the queue can fail to mount. The corruption can occur earlier and remain unnoticed until the appliance restarts, at which point the queue fails to mount.

How long can the repair script take to complete?

Repairing the workqueue can take anywhere from 10 seconds to several hours, depending on the state of the ESA and how many messages are currently processing through an active workqueue. A workqueue repair on lower-end appliance with full queues at the time of corruption could take several hours.

What happens if the repairqueue cannot run or does not complete?

In certain situations, (for example, an over-full queue on an appliance) the repairqueue is not able to complete. if repairqueue does not complete after 4 hours, the queue is most likely unrepairable and the only recourse is to build a new queue. For such advanced issues, Contact Cisco Support to open an active support case and have Cisco Support assist.

Related Content

• Cisco Email Security Appliance End-User Guides
• Cisco Email Security Appliance Command References
• What is the order of processing in the work queue? What is the work queue pipeline?
• Technical Support & Documentation - Cisco Systems

Revision History

Revision	Publish Date	Comments
2.0	02-Oct-2026	Recertification
1.0	10-Aug-2017	Initial Release