Support for external DBs

Author: fengtality 0x352

Original Post Date: 3 Jul 2022

This pull request and forum thread from a long-time community member was the result of a discussion I had with him on Discord regarding support for DBs other than SQLite in Docker images.

While I agree with the review comments that we shouldn’t be adding external dependencies to the Docker image without additional context, I also think that OP’s comment below is valid:

Because when running a cluster of docker containers and all of them write to different SQlite files at the same time, it really cause problems with disk thrashing (backlog disk I/O leading to stalled instance, leading to high memory consumption)

As context, Hummingbot was originally designed as a single bot client, but now that the user base has grown and gotten more sophisticated/professional, many users are running 10+ instances, so I think it makes sense to start exploring how to optimize DB support for larger use cases. Currently, Hummingbot supports writing to external databases, but it’s not an out of the box feature.

Let us know what you think! Is expanding DB support generally a direction that Hummingbot Foundation should explore (relative to other features that we could explore)?

Author: Dardonacci 0x78a

If you are running multiple bots I think using sqlite is not a good solution because this database will be local and you will need to create some ETL to move the data to a centralized database. I think that having all the information in the same place is a must in terms of analyze the performance of the bots.

Adding all DB drivers to the dependencies will also not be an efficient solution because we will be adding things that we won’t use.

What I did: I built my custom image with psycopg2 because I was using a postgresql db.

What about having a docker image for each db? or al least for postgresql or mysql?

Author: fengtality 0x352

Rather than adding custom Docker images for each DB, would it be easier to add a documentation page that shows users how to build custom Docker images that write to other DBs?