Easier strategy creation

Author: fengtality 0x352

Original Post Date: 12 Jan 2022

Related to @zappra‘s scripts thread, there have been discussions with Discord community member @klpanagi about simplifying the creation of Hummingbot strategies.

In addition, CoinAlpha engineering lead @nullably has created a few related exploratory issues:

Note that there may be some architectural / refactoring work needed to support these cases, and this is still very preliminary. Nonetheless, we’d like to open up the discussion to everyone in the community.

Relevant discussions from Discord

@fengtality :

Thanks for kicking off the discussion @klpanagi! I think an important first step is to define the level of abstraction. Since strategies can be quite different and achieve different purposes, especially user generated ones, I think it’d be simpler to start at a lower abstraction level and define a framework that simplifies the creation of a strategy itself. This should be easier to scope because a strategy is simply a looping process that performs the following actions each tick:

  1. Fetch data from an array of markets (exchange-pair combinations)
  2. Cancel orders on those markets
  3. Create orders on those markets

I feel like an interface should let the user to specify the list of the markets, expose helper functions to help the user retrieve data each tick (outstanding orders, top bid/ask, bid and ask orders, etc), and have cancel_order and create_order higher order functions that users can control more precisely. Everything else in between is just pure logic and can be performed in normal Python.

@klpanagi:

I will definitely agree for the initial definition of the abstraction level and build it in layers.

Ok, I see that you are talking about a DSL for **Creating Strategies” , not for customizing currently supported.

In this case, yes we should initially split down and implement (reuse from codebase) basic operations needed to create strategies, in the form of a framework in Python.

Another solution is Refactoring the current implementation to allow using different entities, such as for example MarketDataFeeder and OrderExecutor, for easily creating custom bots in Python, which can use custom / community-driven indicators.

Similar to what I was thinking there. Thanks for pointing out the common market actions on each tick.

So, if I understand correctly, you suggest to initially implement a framework in Python for easily creating strategies. This means we will implement a new way of creating and executing hummingbot bots.

Can you provide an example of how you imagine writing custom strategies? (you act as an end-user here :slightly_smiling_face: )

@fengtality :

Sure, this is totally off the cuff but here’s an example or a simple arbitrage strategy in pseudo-code:

markets = Markets( ["binance", "BTC-USD"], ["kucoin", "BTC-USDT"]
binance_px = markets[0].top_bid
kucoin_px = markets[1].top_ask
 
# create orders
if (binance_px <= kucoin_px)
   markets[0],create_order("limit", "buy", 1000)
   markets[1],create_order("limit", "sell", 1000)

# cancel orders
if (markets[0].outstanding_orders.length > 0)
   markets[0].cancel_all_orders()
if (markets[1].outstanding_orders.length > 0)
   markets[1].cancel_all_orders()

IMO, it’s similar to how Keras offers a simpler entry point into creating a Tensorflow model. It’s limited in scope, but advanced users can always create their ML model in native Tensorflow

@phgnomo:

@klpanagi welcome! Really good to see someone developing this idea, because it’s something we discussed internally a few times but the team never had the proper time to work on it.

I can’t help much with the coding part, but as someone that have been playing around for some time creating trading bots even on TradFi platforms, I am really interested in helping with this side project. We can schedule a call to exchanges some ideas, but as a starting point, my opinion is that we should aim to having something similar to MQL5 (Metatrader) and Pinescript (Tradingview).

I think the simplification of strategy creation shouldn’t remove the potential power of writing it directly in python, but allow must used functions on trading (like “creat buy order, with x volume at y price”) to be added to the strategy by a simple function call.

@klpanagi:

Thanks @fengtality! Hello @phgnomo!

This approach of developing strategies in Python language is actually what I was searching for when I initially started searching for crypto amm bot frameworks. Though, hummingbot client does not currently support this approach due to the actual implementation. We must develop a more abstract implementation of Strategy and refactor (atleast) the HummingApplication class to support the execution, or implement a new Executor.

I will agree that this approach will enable building custom AMM bots, using more abstract textual and graphical languages in the future.

At this point, I would like us to talk about the future intentions, in order to build it correctly from scratch. @phgnomo pointed some example platforms which provide easy scripting of their strategies. I will go one step forward… I have discussed the idea of creating strategies via a graphical language with experience traders and the feedback was very positive. Imagine having a canvas where you can drag&drop indicators, oscillators, algorithms, market_operations etc, connect them in graph, parameterize the blocks and auto-generate and execute the bot via a single click.

In any case, to go there, we must have a high-level framework (Hummingbot Strategy DSL in Python) to easily create and deploy custom strategies. Injecting source code or performing partial or complete code-generation won’t be a problem in the future, If you want to follow that path. Secondly, I would implement an L2 textual DSL for traders with high-level semantics, similar to what you mentioned (“create buy order, with x volume at y price”). Finally, I would build a graphical DSL for traders on top of the textual one.

I will have a look at MQL5 (Metatrader) later one today, just to have an idea of their approach… I know Pinescript , it could be better… :yum:

Sure, lets schedule a call to exchange ideas and craft some initial implementation thoughts.

@phgnomo:

MQL5 is a modified C++ language.

By visual drag&drop, i am guessing you are thinking in something like this:

Yeah, i think it would be a really awesome tool to have, but my guess is that it would need some time to make it work properly.

My suggestion is that we start with the L2 textual DSL, in a way that would make it similar to writing Python (The same way MQL5 is a modified C++ language), but leave the door open on the architecture design for a future implementation of a graphic strategy builder. Here is MQL5 documentation: MQL5 Reference - How to use algorithmic/automated trading language for MetaTrader 5

@klpagani:

Exactly! :slightly_smiling_face: And yeah, It will need time and a lot of effort to build it. That is why I also think of a three phase schedule. 1) High-level Python framework (aka DSL in python), 2) Trader-oriented Textual DSL, 3) Graphical DSL. By utilizing model-driven development techniques, we will be able to transform models from 3 to 2 to 1 and able to perform code-generation or model interpretation at the level of Python. You can also have a look at the NodeRED and NoFlo low-code platforms for the visual paradigm of DSLs. Though these DSLs are specialized for the domain of IoT, but the concept is similar.

Author: Jelle {Cryptobot} 0xe8f

Jelle {Cryptobot}
0xe8f

6 months ago

I really like the idea of a visual editor, this makes it easy for everyone to make strategies themselves. I also think that is a big step for now, and maybe a DSL is a good first step. I have been using Mudrex in the past, this is a website that lets you buy and sell based on a lot of different (complex) rules and let you backtest automatically.

Just wanted to point that out as Hummingbot could create something similar where someone could set rules to set certain spreads based on indicators, cancel orders etc. based on rules in a drag and drop interface. See image of the interface of Mudrex.

Author: klpanagi 0xfba

It is great to see people digging into this conversation already.

I did woke up today with this “title” in my mind: The open-source Trader-centric DSL for high-frequency AMM bots

To get to the target, I thing we shall initially define the user group(s). I will try to explain why this is so important. In DSL engineering it is mandatory to select the target group, because based on that we build the semantics of the language (e.g the grammar in case of textual languages). For example, a hardcore developer may want access to general-purpose functionalities, while a trader user wants (mostly) to use high-level, human-readable, trader-centric concepts. Selection of the user-group is crucial for the definition of the concepts (Market / Parameter / Strategy / etc) of the language. This will also minimize the error space while creating strategies, because the domain is initially modeled (concepts and their relations) along with logical constraints (e.g. maxspread cannot be less than min_spread ).

I am preparing a simple example of a textual DSL, where you can create models of strategies, without the logic part (no implementation). This example will also demonstrate a workaround on the issue raised by CoinAlpha engineering lead @nullably (Introduce new Lite strategies. · Issue #4715 · hummingbot/hummingbot · GitHub), using code-generation techniques. The idea is that, given a strategy model file, the “[strategy]config map.py”, “conf*[strategy]* TEMPLATE.yml”, “[strategy].py” (without implementation) and the “start.py” files can be auto-generated. To predispose you, the grammar at the moment looks like this:

name: SimpleStrategyModel version: “0.0.1” description: “Example of a simple strategy model” markets: [ Market(Binance(‘MY_API_KEY’, ‘MY_API_SECRET’), Pair(‘ADA-USDT’)), Market(AscentEX(‘MY_API_KEY’, ‘MY_API_SECRET’), Pair(‘ADA-USDT’)) ] parameters: [ order_refresh_time : int = 10, max_spread : float = 2.0, (dynamic) spread : float = 0.5 ]

I think I will have a part of the grammar and the code-generator ready in a few days (for proof-of-concept)…

I am also digging in the source code of hummingbot client the last 2 days, in order to understand the implementation architecture and I see the need of refactoring to provide high-level operations, as @fengtality mentioned.

Maybe the “Workspace”, “Project” and “Build” concepts should also be considered for strategy development…

Author: klpanagi 0xfba

I have a first working version of what I promised!

I’ve implemented a DSL for the definition and auto-generation of new strategies. Basically, in its current state, it is something like a “Hbot Strategy Project Generator”.

A code generator is implemented for the generation of all required files, as mentioned in this issue, based on input strategy models. Furthermore, a CLI is provided which allows for building new strategy using a local repo of hummingbot client.

The project is hosted on github. You can find more information below:

Currently, I am working on also initialize markets in the auto-generated [strategy].py file.

1 Like