IBM Aspera HSTS High Availability with an External Load Balancer

IBM Aspera High-Speed Transfer Server (HSTS) can be deployed as an active/active transfer service behind an external load balancer. Clients connect to one VIP or FQDN, while the load balancer distributes new transfer sessions across multiple HSTS nodes and removes unhealthy nodes from service.

The idea is simple. The design details are not. A FASP transfer uses both TCP for secure session management and UDP for the data flow. For a transfer to work correctly, both parts of the same session must reach the same HSTS node. That requirement drives the load-balancer design.

What are we trying to achieve?

The target is an HSTS service that remains available when a transfer node fails and can also use multiple healthy nodes at the same time.

• One client endpoint. Users and applications connect to a VIP/FQDN instead of an individual HSTS server.
• Active/active HSTS. All healthy transfer nodes can receive new sessions.
• Load sharing. Incoming transfers are distributed across available nodes.
• Automatic failover. A failed node is removed from the pool so new sessions go elsewhere.
• Transfer resume. Interrupted transfers can resume without retransmitting data already received, provided the cluster storage design supports it.

Target architecture

The external load balancer sits in front of the HSTS nodes. All transfer nodes remain active, and the load balancer decides where each new incoming session should go.

Why an external load balancer?

IBM's architecture document lists an external load balancer as one of the preferred HSTS high-availability methods. Compared with simple DNS rotation, it gives the platform direct awareness of node availability and allows failed nodes to be dynamically removed from the pool.

The most important rule: keep TCP and UDP on the same HSTS node

A FASP transfer session has two parts:

• Session management: secure TCP using SSH or HTTPS.
• Data transfer: UDP.

If the load balancer sends the TCP connection to HSTS1 but sends the corresponding UDP flow to HSTS2, the transfer session is broken.

IBM's document recommends client IP stickiness when a standard IP load balancer does not understand the relationship between the TCP and UDP portions of FASP. Traffic from the same client IP is then directed to the same HSTS member for the persistence period.

For HSTS, persistence is not simply a web-session optimization. It is part of keeping the FASP control and data flows together.

Core design requirements

Step 1 - Build identical HSTS nodes

Start with HSTS nodes that are as identical as possible: same HSTS release, same operating-system baseline, same transfer-user definitions, same policies and the same relevant aspera.conf settings.

IBM specifically states that aspera.conf must be identical across cluster members. In the same configuration area, cluster_id.conf contains the common cluster identifier while node_id.conf remains unique per node.

Step 2 - Use storage that all HSTS nodes can access

Shared file storage is a central part of this HA pattern. The transferred content must be visible to every HSTS node that may receive or resume a session.

For filesystem-based transfers, FASP creates a companion resume file with the .aspx suffix in the same folder as the target file. If a transfer is interrupted and the client reconnects to a different HSTS member, that node can use the shared target data and resume information to continue the transfer.

This is why an HSTS cluster is more than two independent servers behind a VIP. The nodes must see the same transfer data.

Step 3 - Decide how Node API state will be handled

HSTS includes a Redis database used mainly by the REST Node API and applications that consume it. The database can contain:

• Node API users, passwords, transfer users and ACLs.
• Access-key information.
• Recent and ongoing transfer information.
• File IDs and file-permission information.

IBM's general rule for an HSTS cluster is to use the documented HA Redis backend. That design requires Redis quorum and therefore normally drives a minimum of three Redis systems.

The source document also describes a simpler case for integrations that use only static Node API data: identical local database content can be populated on each HSTS node instead of running a Redis cluster, provided the consuming application does not depend on dynamic file IDs/permissions and gathers transfer information from members individually.

Step 4 - Create the external load-balancer service

Create a virtual service using a VIP/FQDN that clients will use for HSTS transfers. Add each HSTS transfer server to the backend pool.

The exact terminology depends on the load-balancer product, but the logical configuration looks like this:

Virtual service:
  Name: aspera-hsts
  VIP: 203.0.113.20
  FQDN: hsts.example.com

Backend pool:
  HSTS1: 10.10.10.11
  HSTS2: 10.10.10.12

Persistence:
  Client / source IP stickiness

Health:
  Remove unhealthy members from new-session routing

The source document gives F5 BIG-IP as one example of an external load balancer, but the architectural requirement is product-independent: the device must be able to route the relevant TCP and UDP traffic consistently and maintain session affinity.

Step 5 - Configure persistence correctly

For a standard load balancer, use a persistence method that sends the same client's TCP and UDP traffic to the same backend HSTS node.

A simple source-IP persistence model looks conceptually like this:

Client A → TCP session → HSTS1
Client A → UDP flow    → HSTS1

Client B → TCP session → HSTS2
Client B → UDP flow    → HSTS2

This also explains one trade-off of source-IP stickiness: many sessions originating from the same NAT address may remain pinned to the same HSTS node even when other cluster members have available capacity. If that traffic pattern matters, the load-balancing design should be tested carefully.

Step 6 - Configure health checks that test the service, not only the server

A server replying to ping does not prove that HSTS can accept a transfer. IBM lists several layers of health validation:

The closer the health check is to a real transfer, the better it represents actual service availability. The trade-off is additional complexity and monitoring traffic.

Step 7 - Make the load-balancer tier highly available too

Putting multiple HSTS nodes behind one load-balancer appliance removes the HSTS node as a single point of failure but may simply move that single point of failure to the load balancer.

IBM's document is explicit: for a full HA solution, the external load balancers themselves should also be deployed redundantly.

A complete design therefore looks like:

What happens during an HSTS node failure?

Assume two HSTS nodes are active and the load balancer has already established healthy pool membership.

1. A client starts a FASP session and is sent to HSTS1.
2. HSTS1 fails because of network, hardware, storage, processing or software failure.
3. The health-check mechanism detects the failure.
4. The load balancer stops sending new sessions to HSTS1.
5. New sessions are directed to another healthy HSTS node.
6. An interrupted FASP transfer can reconnect and resume, using the shared target data and FASP resume mechanism.

The key distinction is that a load balancer does not magically move an already-running UDP session to another HSTS process. The transfer is interrupted, then FASP's resume capability allows it to continue after reconnection without retransmitting the entire file.

Failure scenarios to test

Validation plan

I would validate the architecture in this order:

1. Direct-node test. Transfer successfully to every HSTS server individually.
2. Shared-storage test. Confirm all transfer nodes see the same target files and resume data.
3. VIP test. Start transfers through the load-balancer FQDN.
4. Distribution test. Verify new sessions are reaching multiple HSTS members.
5. Affinity test. Confirm the TCP and UDP portions of each transfer stay on one node.
6. Node-failure test. Remove or stop one HSTS member and verify new transfers continue on the remaining node.
7. Resume test. Interrupt a transfer, reconnect through the VIP and verify the transfer resumes rather than restarting from zero.
8. Load-balancer failover test. Fail the active load-balancer instance and verify the VIP remains available.

External load balancer vs simpler HA approaches

Common design mistakes

• Load-balancing TCP and UDP independently. They belong to the same FASP session and must reach the same HSTS member.
• Keeping node-local transfer storage. Another HSTS node cannot transparently continue the same clustered transfer if the target data is visible only on the failed server.
• Letting configurations drift. Cluster members should expose the same service behavior.
• Checking only ping. A server can answer ICMP while HSTS, storage or Node API is unavailable.
• Making one load balancer the new SPOF. HA HSTS behind a single non-redundant load balancer is not end-to-end HA.
• Ignoring Redis/application requirements. Node API consumers that depend on dynamic data may require an HA Redis backend, not separate node-local copies.

Production checklist

Final take

An external load balancer is one of the cleanest ways to build active/active HSTS high availability. It gives users a single endpoint, spreads new transfer sessions across healthy nodes and automatically avoids failed servers.

But the load balancer is only one part of the architecture. A correct HSTS HA design also depends on TCP/UDP session affinity, synchronized HSTS configuration, shared transfer storage, the right Redis/Node API state model, meaningful health checks and a redundant load-balancer tier.

Get those pieces right, and an HSTS node becomes a replaceable member of the transfer service rather than a server that users depend on directly.