Sharing our QA process on certified exchanges
Certified connectors should passed the QA connector pull request checklist
- Each connector should be able to perform as expected
- Compatible with most of the hummingbot strategies and for Liquidity Mining purposes
- Add labels on issues according to priority and how it impacts the connector performance
Connecting an API key
- Connect when a valid API key is added
- Throws an error or warning if the API key is invalid, expired, or has other issues.
- The same API key can be used on multiple bots unless specified that can only be used in one instance at a time.
Pulling the balance
- Displays the current available balance and should match the balance shown in the exchange.
- The allocation should be updated whenever there is an open order.
Market availability during the strategy creation
- Autocomplete lists should be available when setting a market during the strategy creation.
- All markets should work when the created strategy is started.
Compatibility with the available strategies
- The connector should work on any of the strategies available in the client unless the connector is intended for a specific strategy.
- The connector should work both as a maker and taker exchange (spot connectors only)
Price and balance updates
- In the status window, the prices are constantly being updated whenever a change takes place in the order book.
- Available balances are updated whenever an order is created or canceled.
- A consistent number of orders are created except if there are multiple bots using the asset, there are hanging orders or an order is removed due to a specific parameter.
Order submission and cancellation
- Created orders must include an order ID with broker prefix if there is.
- Orders are submitted without any error in the client.
- Submitted order should match the information of the open order in the exchange.
- Orders are canceled without any error.
- Orders are not getting stuck or left out unless it is a manual order.
- The client should not cancel orders that are not created within the instance such as manual orders, orders created by other instances, or third-party bots.
- Gracefully rejects/cancels orders that don’t pass the exchange rule.
Fast refresh rate
- Gracefully cancels orders: No stuck or lost orders
- Balance updates accordingly
- There should be no error in the logs during the fast cancellation.
- Filled orders are tracked and saved if a filled event took place.
- Rate limit warnings are thrown whenever the request is close to maximizing the allowed limit.
- Stops placing an order when the rate limit is reached but maintains the connectivity with the exchange.
Long refresh rate
- Maintains connectivity when there is an open order.
- Gracefully cancel an order when not filled.
- No error should come up during the period that an order is opened (exceptions if there are network issues)
- If a network issue took place, the open orders should be tracked once the connection is established.
- Tracked and save filled events that may happen during a disconnection.
- Simultaneously submitting multiple orders without any error.
- Simultaneously cancels all the orders without any error.
- Available balance and allocations are adjusted accordingly.
- Order amounts and levels are adjusted accordingly based on the available balance.
- Continuously tracked hanging orders created within the instance.
- Tracked and saved a filled hanging order
- Hanging order should be canceled whenever the strategy is stopped or reached the cancelation percentage.
- Hanging orders remain uncanceled whenever non-hanging orders are refreshed.
- No hanging order duplication
- Open orders in each bot should not be affected by cancelation events happening in another bot.
- There should be no conflicting order IDs
- Orders book in the client is in sync with the order book in the exchange.
- Constantly update whenever there is a change in the exchange.
- Prices are updated constantly (status and ticker) whenever there is a change in the exchange.
- Full and partial fills are both tracked and recorded properly.
- Filled order information should match the trade history in the exchange.
- Correctly display filled order information
- No duplicate orders
- Save partial fills
- Display only orders created by the instance
- Correctly displays asset information
- Should use the trade fee if available otherwise an estimate fee is used.
- The transaction fee recorded in the client (CSV or SQLite) should match the fee shown in the trade history in the exchange.
- When enabled, all filled trades are aggregated in Datadog
- If the bot is stopped and a filled event took place within the anonymized_metrics interval, this should still be aggregated.
We have setup a dedicated QA test server for running long term test bots with different strategies and setup. Part of the task is to monitor these test bots and review the log files, check the exchange if there are stuck orders and check the trade data if there are filled trade events.