QA process for certified exchanges

Hi Everyone!

Sharing our QA process on certified exchanges

QA testing upon pull request submission and review

Certified connectors should passed the QA connector pull request checklist

  • Each connector should be able to perform as expected
  • Compatible with most of the hummingbot strategies and for Liquidity Mining purposes
  • Add labels on issues according to priority and how it impacts the connector performance

QA connector pull request review checklist

Connecting an API key

  • Connect when a valid API key is added
  • Throws an error or warning if the API key is invalid, expired, or has other issues.
  • The same API key can be used on multiple bots unless specified that can only be used in one instance at a time.

Pulling the balance

  • Displays the current available balance and should match the balance shown in the exchange.
  • The allocation should be updated whenever there is an open order.

Market availability during the strategy creation

  • Autocomplete lists should be available when setting a market during the strategy creation.
  • All markets should work when the created strategy is started.

Compatibility with the available strategies

  • The connector should work on any of the strategies available in the client unless the connector is intended for a specific strategy.
  • The connector should work both as a maker and taker exchange (spot connectors only)

Price and balance updates

  • In the status window, the prices are constantly being updated whenever a change takes place in the order book.
  • Available balances are updated whenever an order is created or canceled.
  • A consistent number of orders are created except if there are multiple bots using the asset, there are hanging orders or an order is removed due to a specific parameter.

Order submission and cancellation

  • Created orders must include an order ID with broker prefix if there is.
  • Orders are submitted without any error in the client.
  • Submitted order should match the information of the open order in the exchange.
  • Orders are canceled without any error.
  • Orders are not getting stuck or left out unless it is a manual order.
  • The client should not cancel orders that are not created within the instance such as manual orders, orders created by other instances, or third-party bots.
  • Gracefully rejects/cancels orders that don’t pass the exchange rule.

Fast refresh rate

  • Gracefully cancels orders: No stuck or lost orders
  • Balance updates accordingly
  • There should be no error in the logs during the fast cancellation.
  • Filled orders are tracked and saved if a filled event took place.
  • Rate limit warnings are thrown whenever the request is close to maximizing the allowed limit.
  • Stops placing an order when the rate limit is reached but maintains the connectivity with the exchange.

Long refresh rate

  • Maintains connectivity when there is an open order.
  • Gracefully cancel an order when not filled.
  • No error should come up during the period that an order is opened (exceptions if there are network issues)
  • If a network issue took place, the open orders should be tracked once the connection is established.
  • Tracked and save filled events that may happen during a disconnection.

Multiple orders

  • Simultaneously submitting multiple orders without any error.
  • Simultaneously cancels all the orders without any error.
  • Available balance and allocations are adjusted accordingly.
  • Order amounts and levels are adjusted accordingly based on the available balance.

Hanging orders

  • Continuously tracked hanging orders created within the instance.
  • Tracked and saved a filled hanging order
  • Hanging order should be canceled whenever the strategy is stopped or reached the cancelation percentage.
  • Hanging orders remain uncanceled whenever non-hanging orders are refreshed.
  • No hanging order duplication

Multiple bots

  • Open orders in each bot should not be affected by cancelation events happening in another bot.
  • There should be no conflicting order IDs

Data integrity

  • Orders book in the client is in sync with the order book in the exchange.
  • Constantly update whenever there is a change in the exchange.
  • Prices are updated constantly (status and ticker) whenever there is a change in the exchange.

Filled event

  • Full and partial fills are both tracked and recorded properly.
  • Filled order information should match the trade history in the exchange.

History

  • Correctly display filled order information
    • No duplicate orders
    • Save partial fills
    • Display only orders created by the instance
  • Correctly displays asset information

Trade fees

  • Should use the trade fee if available otherwise an estimate fee is used.
  • The transaction fee recorded in the client (CSV or SQLite) should match the fee shown in the trade history in the exchange.

Data aggregation

  • When enabled, all filled trades are aggregated in Datadog
  • If the bot is stopped and a filled event took place within the anonymized_metrics interval, this should still be aggregated.

QA process for connector maintenance

We have setup a dedicated QA test server for running long term test bots with different strategies and setup. Part of the task is to monitor these test bots and review the log files, check the exchange if there are stuck orders and check the trade data if there are filled trade events.

1 Like

Wow, that’s quite an extensive checklist, and I imagine it will continue to grow as more features are added in Hummingbot codebase. :muscle:

Cheers to QAs’ dedication in making Hummingbot running as smoothly as possible in every release! :clinking_glasses:

I would like to clarify about partial fills.

I understand bots are tracking these by observing the ask’s order_amount decreasing whenever a sell order is partially filled. However, can’t usually see any message about partial fill, and history command also do not specify partial fills.

On your part, how do you track or verify these partial fills?

How do you differentiate these in result of history command?

Hi cgambit,

Good day and thanks for the feedback. In the event of partial fills the client should be able to capture the trade and still record it on history.
Lets try the scenario below:

  • The client created bid order of 30USDT for 60s
  • The client had a partial fill of bid order 15/30USDT
  • Within 60s order refresh time:
    • If the remaining of the order is not filled, The history command should stil be able to record the partially filled order
    • if the order has been complete filled, it should add a log on output panel that the order has been completed
    • After 60s, the client should proceed with creating new set of orders.
    • The trade data between the history --verbose, CSV file and exchange’s trade history

How do you track or verify these partial fills?
- First is to run history --verbose command and check if there are partial fills. We should be able to notice this on the amount tab since its not the same order_amount.
- Review the log file based on the order price and timestamp recorded on the history command
- Compare the trade data you have on the client from the CSV and exchange’s trade history

How do you differentiate these in result of history command?
It should be noticeable on the history --verbose command, on the screenshot below we happen to had a partial fills for perpetual market-making:

1 Like

Awesome, @rapcmia ! That’s super useful! Thank you. :partying_face: