Intermittent 404s in IBM App Connect Enterprise: troubleshooting stale HTTP listener registrations

An HTTP 404 Not Found from IBM App Connect Enterprise normally suggests a simple problem: the URI is wrong or the API is not deployed. But there is a more difficult failure mode where the API is deployed, the Integration Node listener is running, and the same endpoint works one minute and returns 404 after a deployment or restart.

In that situation, repeatedly redeploying BAR files can make the problem even harder to understand. One API starts working, another begins returning 404, and the listener itself still looks healthy.

This guide focuses on that pattern and on a particularly important place to look when ACE uses the Integration Node HTTP listener: the MQ-backed SYSTEM.BROKER.WS.* state used by the listener and Integration Servers.

The symptom pattern

The issue is worth investigating as a listener-registration problem when several of these symptoms occur together:

• The REST API or web-service flow is deployed and reported as running.
• The Integration Node HTTP/HTTPS listener is enabled.
• Only some URLs return 404; others on the same listener continue to work.
• A redeploy appears to fix one endpoint but the 404 moves to another endpoint.
• The listener process remains alive, so a simple process check does not explain the failure.
• MQ is configured as the Integration Node's default Queue Manager.

If the failing endpoint changes after deployments, stop treating the problem as a normal routing typo. Validate the listener's registration state.

How the node-wide HTTP listener works

IBM documents two listener models in App Connect Enterprise: an Integration Node listener and listeners embedded in individual Integration Servers. The node-wide listener is shared across Integration Servers and requires access to system queues on the Queue Manager configured for the Integration Node.

At a high level, the Integration Server updates the listener registration information, while the node-wide listener browses that information to determine which endpoint URLs it should expose.

Why duplicate WS.ACK registrations can create an intermittent 404

In the failure pattern that motivated this article, the healthy registration state had one set of messages in SYSTEM.BROKER.WS.ACK. The failing state contained two sets.

The important behavior is:

1. An Integration Server deploys or updates a flow that uses the node-wide listener.
2. The Integration Server updates the registration message for the exposed endpoints.
3. The listener browses the registration message to build the endpoint list it will expose.
4. If an older duplicate registration set remains ahead of the current set, the listener can continue reading stale endpoint information.
5. An endpoint that is deployed correctly can therefore return 404 because the listener's visible registration list does not contain the current endpoint.

This also explains a confusing observation: redeploying one flow can make that flow work while another URL begins failing. The deployment changes the registration messages being read, but it does not necessarily remove the underlying duplicate state.

Before touching MQ: classify the 404

Not every ACE 404 is a WS.ACK problem. Use the symptom to decide which branch to investigate first.

Step 1 - Confirm which listener the flow is using

Start by confirming whether the HTTP or SOAP flow uses the Integration Node listener or the Integration Server embedded listener.

# Node-wide listener
mqsireportproperties <NODE> -b NodeHttpListener -o HTTPConnector -r
mqsireportproperties <NODE> -b NodeHttpListener -o HTTPSConnector -r

# Integration Server listener selection
mqsireportproperties <NODE> -e <SERVER> \
  -o ExecutionGroup -n httpNodesUseEmbeddedListener

mqsireportproperties <NODE> -e <SERVER> \
  -o ExecutionGroup -n soapNodesUseEmbeddedListener

If the flow is using an embedded listener, SYSTEM.BROKER.WS.ACK is not the first place to troubleshoot an HTTP-listener registration problem.

Step 2 - Verify the listener and port

Confirm that the node-wide listener process exists and the configured port is actually bound.

ps -ef | grep biphttplistener
ss -lntp | grep '<listener-port>'

Also check syslog for listener startup failures. An embedded listener and the node-wide listener are separate processes and cannot bind to the same TCP port. A typical ACE error for that condition is BIP3144S.

Step 3 - Confirm the Queue Manager is healthy

The Integration Node listener requires a local Queue Manager. If MQ is unavailable, the listener cannot use the system queues to communicate correctly with the flows, and HTTP requests can fail with 404 or 500 responses.

Before investigating stale registrations, verify that:

• The Queue Manager is running and stable.
• The SYSTEM.BROKER.WS.* queues are accessible.
• The queues are not inhibited or full.
• There are no MQ resource, lock-wait or availability events coinciding with the HTTP failure.

If the Queue Manager itself is unstable, fix that first. A registration investigation performed while MQ is repeatedly failing will produce misleading results.

Step 4 - Inspect the SYSTEM.BROKER.WS queues

Use runmqsc to capture the queue state before making any change.

runmqsc <QM>

DISPLAY QLOCAL(SYSTEM.BROKER.WS.ACK) \
  CURDEPTH MAXDEPTH IPPROCS OPPROCS PUT GET

DISPLAY QLOCAL(SYSTEM.BROKER.WS.INPUT) \
  CURDEPTH IPPROCS OPPROCS

DISPLAY QLOCAL(SYSTEM.BROKER.WS.REPLY) \
  CURDEPTH IPPROCS OPPROCS

The queue depth by itself is only a clue. Browse the messages and compare correlation IDs, message lengths and registration content with a known-good state.

amqsbcg SYSTEM.BROKER.WS.ACK <QM>

Things to look for:

• More registration messages than the normal baseline.
• Two copies of the same correlation-ID categories.
• Old and new endpoint lists coexisting.
• Registration data that does not match the currently deployed APIs.

Step 5 - Compare ACE's view of the endpoints

Do not rely on MQ alone. Capture ACE's current view of the HTTP connector and the generated endpoint configuration before and after reproducing the failure.

mqsireportproperties <NODE> -e <SERVER> \
  -o HTTPConnector -r

If you are using plugin configuration generated by ACE, preserve the corresponding wsplugin6.conf file as part of the evidence set.

A very useful comparison is:

Step 6 - Freeze deployments before recovery

Do not remove system-queue messages while deployments, starts, stops or listener changes are happening. Registration messages can be changing at the same time, which makes it easy to remove the wrong state.

Recovery options

Option 1 - Remove only the stale registration set

If the correct and stale registration messages can be identified confidently, remove only the old set, then immediately confirm that the listener is browsing the current registration information.

This is the least disruptive recovery, but also the option that requires the most certainty about which messages are correct.

Option 2 - Rebuild the registration state

In a controlled maintenance window, clearing the WS.ACK registration state and restarting the affected Integration Server causes its endpoints to be registered again. This was the recovery that restored the healthy registration state in the investigated failure.

The important point is that restart alone is not necessarily a cleanup. The WS system-queue messages are persistent, so a stale duplicate state can survive normal process or Queue Manager restarts.

Option 3 - Redeploy a dummy endpoint

A temporary diagnostic workaround is to redeploy a harmless dummy endpoint. This can move the stale registration effect away from a business endpoint and restore previously failing URLs.

It is useful for proving the registration theory, but it does not remove the bad state. Do not treat it as the permanent fix.

Validation after recovery

Do more than test the one URL that originally failed.

1. Confirm the WS.ACK registration state has returned to the known-good baseline.
2. Invoke every critical API exposed through the listener.
3. Perform a controlled deploy or stop/start of one web-service flow.
4. Check WS.ACK again and confirm that the registration count does not grow unexpectedly.
5. Invoke the APIs again and confirm the 404 has not moved to another endpoint.
6. Monitor through at least one normal deployment cycle.

What could create the duplicate state?

This is where the evidence needs to be separated from speculation. The duplicate registration state was the confirmed reason the listener exposed the wrong endpoint list. The underlying event that originally created the duplicate set was not conclusively identified.

Investigation areas included:

• Deploy or redeploy timing.
• Integration Server or Integration Node restart/crash history.
• Queue Manager instability or resource contention.
• More than one Integration Node using the same default Queue Manager.
• Listener configuration changes.
• Accidental overlap between embedded and node-wide listener ports.

IBM support noted that Integration Servers normally replace the corresponding registration message before writing the updated one, making a natural duplicate state unexpected. Once the duplicate state exists, however, it can persist because both the listener and Integration Servers continue operating on the first matching registration message.

Multiple Integration Nodes sharing one Queue Manager

This deserves its own check because it can cause similar endpoint-registration failures even though it was not identified as the mechanism that created the duplicate set in this incident.

Current IBM documentation allows multiple Integration Nodes to specify the same Queue Manager, but warns that deployment issues can occur when different Integration Nodes share a Queue Manager, use the Integration Node listener, and reuse a unique endpoint suffix.

For simpler operational isolation, many environments choose one default Queue Manager per Integration Node listener. If a Queue Manager is intentionally shared, document the endpoint-suffix design and test deployment isolation carefully.

Embedded listener or node-wide listener?

Neither model is universally better. The key is to know which model your flows use and avoid accidentally configuring both listener processes on the same TCP port.

A practical troubleshooting sequence

Prevention

• Baseline the WS system queues. Know the normal queue depth and correlation-ID pattern for your environment.
• Monitor MQ health. The node-wide listener depends on MQ; Queue Manager instability can surface as HTTP failures.
• Keep listener ports unambiguous. Node-wide and embedded listeners cannot own the same port on the same host.
• Control Queue Manager sharing. If multiple Integration Nodes share a QM, ensure endpoint identifiers do not collide.
• Capture before/after deployment evidence. Queue state, connector state and endpoint lists make intermittent registration problems much easier to diagnose.
• Do not automate destructive cleanup. An unexpected duplicate state is an exception that should trigger investigation, not a scheduled queue-clear job.

IBM documentation to keep nearby

• IBM App Connect Enterprise - HTTP listeners
• IBM App Connect Enterprise - Processing HTTP messages
• IBM App Connect Enterprise - Creating an Integration Node
• Features that require a local IBM MQ server and Queue Manager

Final take

When an ACE API is deployed but the node-wide listener intermittently returns 404, do not stop at “the listener process is running.” The listener's registration state can be wrong even when the process and TCP port are healthy.

The most valuable diagnostic pattern is: confirm MQ health → inspect SYSTEM.BROKER.WS.ACK → compare the registration set with the deployed endpoints → recover the stale state in a controlled window → prove the state remains healthy after another deployment.

And one final distinction matters: finding duplicate WS.ACK registrations explains the 404 behavior, but it does not automatically explain how the duplicate registrations were created. Preserve that difference in the incident report. A good troubleshooting result separates the confirmed failure mechanism from the still-unproven trigger.