stevehopwoodforex.com
https://www.stevehopwoodforex.com/phpBB3/
Print view

Data from Desky - Machine Learning
https://www.stevehopwoodforex.com/phpBB3/viewtopic.php?t=6058
Page 1 of 1
Author:  Danyblu [ Wed Jul 07, 2021 11:31 am ]
Post subject:  Data from Desky - Machine Learning

Hello everyone, I don't know if this fits here, but I think TDesk could be perfect for handling this so I decided to post this here:

Recently I've done a course on Machine Learning (ML) and want to start a ML-project on Forex. Esentially I want to create a ML-model which can calculate the credibility of certain indicators or EA's.

The idea I had was to use Tdesk to generate the supersignal of many combined EA's or indicators, measure how much money the supersignal-caused trade generated, and with this data conclude if a single indicator or EA signal is important to look at when entering a trade. (Maybe there isn't even a supersignal needed, just trade based on one indicator or EA, and look at what other indis or EA's signaled at the time of the trade.)

The most important (and dificult) thing in ML is to get reliable data. I thought maybe someone already uses TDesk to save single signals and/or trades through Desky in a csv-file. There will be lots of dificulties in this project for me, as I am pretty new to all of this, but the first and most important thing would be to have data.

I'll have to think about which data I need exactly , but for now I just wanted to ask if someone already uses Tdesk to save data to a csv-file and has some tips or information for me?

Extracting data using MT5 backtesting could also be a solution, but I don't know yet if that would work.

Thank you! :)
Author:  SteveHopwood [ Sun Jul 11, 2021 10:44 am ]
Post subject:  Data from Desky - Machine Learning

I will give this a bump in my wrup later on today.

Lots of people may be willing to help but you would need to explain how to send data from TDesk to a .csv file. Not many will know this - for sure I have no clue.

:xm: :rocket:
Author:  tomele [ Sun Jul 11, 2021 10:52 am ]
Post subject:  Data from Desky - Machine Learning

Nor do I. Well, at least without quite some additional coding.
Author:  Danyblu [ Sun Jul 11, 2021 6:43 pm ]
Post subject:  Data from Desky - Machine Learning

I've done some research and I think it could work like this:

This code could be called everytime a supersignal forms.
I don't know exactly how TDesk works, but I'm assuming that TDesk is written in one file which somehow gets all its variables from different drones and these variables could say something like "buy" or "veto".

The important part is the FileWrite function, which just writes another line in the csv-file through a file_handle.

Code: Select all

input string             InpFileName="file.csv";  // file name
input string             InpDirectoryName="Datafolder"; // directory name

int file_handle=FileOpen(InpDirectoryName+"//"+InpFileName,FILE_READ|FILE_WRITE|FILE_CSV);
if(file_handle!=INVALID_HANDLE)
     {
      PrintFormat("%s file is available for writing",InpFileName);
      PrintFormat("File path: %s\\Files\\",TerminalInfoString(TERMINAL_DATA_PATH));
      FileWrite(file_handle,"Time","Indicator1","Indicator2","EA1");
      for(int i=0;i<number_of_signals;i++)
         FileWrite(file_handle,time[i],indicator1_signal[i],indicator2_signal[i],EA1_signal[i]);
      FileClose(file_handle);
      PrintFormat("Data is written, %s file is closed",InpFileName);
     }
   else
      PrintFormat("Failed to open %s file, Error code = %d",InpFileName,GetLastError());

Author:  Danyblu [ Sun Jul 11, 2021 6:47 pm ]
Post subject:  Data from Desky - Machine Learning

Here is a practical example I found on how to use this. This is a MQL5 script you can run, which saves timepoints in a csv-file when macd formed a new buy or sell signal. I ran it on MT5 and it worked.

Code: Select all

//+------------------------------------------------------------------+
//|                                               Demo_FileWrite.mq5 |
//|                        Copyright 2013, MetaQuotes Software Corp. |
//|                                              https://www.mql5.com |
//+------------------------------------------------------------------+
#property copyright "Copyright 2013, MetaQuotes Software Corp."
#property link      "https://www.mql5.com"
#property version   "1.00"
//--- show the window of input parameters when launching the script
#property script_show_inputs
//--- parameters for receiving data from the terminal
input string             InpSymbolName="EURUSD";           // currency pair
input ENUM_TIMEFRAMES    InpSymbolPeriod=PERIOD_H1;        // time frame
input int                InpFastEMAPeriod=12;              // fast EMA period
input int                InpSlowEMAPeriod=26;              // slow EMA period
input int                InpSignalPeriod=9;                // difference averaging period
input ENUM_APPLIED_PRICE InpAppliedPrice=PRICE_CLOSE;      // price type
input datetime           InpDateStart=D'2012.01.01 00:00'; // data copying start date
//--- parameters for writing data to file
input string             InpFileName="MACD.csv";  // file name
input string             InpDirectoryName="Data"; // directory name
//+------------------------------------------------------------------+
//| Script program start function                                    |
//+------------------------------------------------------------------+
void OnStart()
  {
   datetime date_finish; // data copying end date
   bool     sign_buff[]; // signal array (true - buy, false - sell)
   datetime time_buff[]; // array of signals' arrival time
   int      sign_size=0; // signal array size
   double   macd_buff[]; // array of indicator values
   datetime date_buff[]; // array of indicator dates
   int      macd_size=0; // size of indicator arrays
//--- end time is the current time
   date_finish=TimeCurrent();
//--- receive MACD indicator handle
   ResetLastError();
   int macd_handle=iMACD(InpSymbolName,InpSymbolPeriod,InpFastEMAPeriod,InpSlowEMAPeriod,InpSignalPeriod,InpAppliedPrice);
   if(macd_handle==INVALID_HANDLE)
     {
      //--- failed to receive indicator handle
      PrintFormat("Error when receiving indicator handle. Error code = %d",GetLastError());
      return;
     }
//--- being in the loop until the indicator calculates all its values
   while(BarsCalculated(macd_handle)==-1)
      Sleep(10); // pause to allow the indicator to calculate all its values
//--- copy the indicator values for a certain period of time
   ResetLastError();
   if(CopyBuffer(macd_handle,0,InpDateStart,date_finish,macd_buff)==-1)
     {
      PrintFormat("Failed to copy indicator values. Error code = %d",GetLastError());
      return;
     }
//--- copy the appropriate time for the indicator values
   ResetLastError();
   if(CopyTime(InpSymbolName,InpSymbolPeriod,InpDateStart,date_finish,date_buff)==-1)
     {
      PrintFormat("Failed to copy time values. Error code = %d",GetLastError());
      return;
     }
//--- free the memory occupied by the indicator
   IndicatorRelease(macd_handle);
//--- receive the buffer size
   macd_size=ArraySize(macd_buff);
//--- analyze the data and save the indicator signals to the arrays
   ArrayResize(sign_buff,macd_size-1);
   ArrayResize(time_buff,macd_size-1);
   for(int i=1;i<macd_size;i++)
     {
      //--- buy signal
      if(macd_buff[i-1]<0 && macd_buff[i]>=0)
        {
         sign_buff[sign_size]=true;
         time_buff[sign_size]=date_buff[i];
         sign_size++;
        }
      //--- sell signal
      if(macd_buff[i-1]>0 && macd_buff[i]<=0)
        {
         sign_buff[sign_size]=false;
         time_buff[sign_size]=date_buff[i];
         sign_size++;
        }
     }
//--- open the file for writing the indicator values (if the file is absent, it will be created automatically)
   ResetLastError();
   int file_handle=FileOpen(InpDirectoryName+"//"+InpFileName,FILE_READ|FILE_WRITE|FILE_CSV);
   if(file_handle!=INVALID_HANDLE)
     {
      PrintFormat("%s file is available for writing",InpFileName);
      PrintFormat("File path: %s\\Files\\",TerminalInfoString(TERMINAL_DATA_PATH));
      //--- first, write the number of signals
      FileWrite(file_handle,sign_size);
      //--- write the time and values of signals to the file
      for(int i=0;i<sign_size;i++)
         FileWrite(file_handle,time_buff[i],sign_buff[i]);
      //--- close the file
      FileClose(file_handle);
      PrintFormat("Data is written, %s file is closed",InpFileName);
     }
   else
      PrintFormat("Failed to open %s file, Error code = %d",InpFileName,GetLastError());
  }
EDIT: Thats how the first lines of the resulting csv file looks (FALSE means sell signal, TRUE means buy signal):

1882
02.01.2012 04:00 FALSE
03.01.2012 03:00 TRUE
04.01.2012 14:00 FALSE
09.01.2012 22:00 TRUE
11.01.2012 02:00 FALSE
12.01.2012 12:00 TRUE
13.01.2012 16:00 FALSE
17.01.2012 04:00 TRUE
23.01.2012 02:00 FALSE
23.01.2012 12:00 TRUE
25.01.2012 13:00 FALSE
25.01.2012 20:00 TRUE
27.01.2012 09:00 FALSE
27.01.2012 11:00 TRUE
30.01.2012 13:00 FALSE
31.01.2012 03:00 TRUE
Author:  tomele [ Wed Jul 14, 2021 10:50 am ]
Post subject:  Data from Desky - Machine Learning

Thank you for explaining the FileWrite function to me. However, I have written some ten thousand lines of MQL code and there has been one or another occasion of writing data to files. I would eventually manage to add some code to TDesk that writes CSV files containing the data you outlined.

But before I spend any time on this, I want to see some meat. Please outline your plan. What will you be doing with that CSV file? With regard to your thread title, which "machine" will "learn" what from it? I am not interested in buzz words, only in technical details.

Thanks in advance.
Author:  Danyblu [ Wed Jul 14, 2021 8:24 pm ]
Post subject:  Data from Desky - Machine Learning

tomele » Wed Jul 14, 2021 11:50 am wrote:Thank you for explaining the FileWrite function to me. However, I have written some ten thousand lines of MQL code and there has been one or another occasion of writing data to files. I would eventually manage to add some code to TDesk that writes CSV files containing the data you outlined.

But before I spend any time on this, I want to see some meat. Please outline your plan. What will you be doing with that CSV file? With regard to your thread title, which "machine" will "learn" what from it? I am not interested in buzz words, only in technical details.

Thanks in advance.
I don’t really know how to answer the question. In case you want concrete steps I can give you an overview: At first I would use the pandas library to import the data to use it with python. With python I can split the data into data that is used to train a model (which means the model will be trying to predict something by connections it saw in the train data), and into data which will be used to test how accurate the model can predict something (by using the r2 value, also called the “coefficient of determination”). That’s called a train-test-split.

An easy example would be having the following unrealistic data:
Signal, profit
80, 10
85, 20
90, 30
100, 50

There are many different models, in this case the simplest model, linear regression, would be obvious to use, which tries to draw a straight line through the data points to put simply.
If the train data would consist of the first three rows, the resulting model would predict the profit by following formula: y = 2*x – 150
Then the r2 value would be calculated by predicting that the profit at a signal of 100 would be 50, and conclude that our model is perfect since our test-data confirms that (-> r2 value = 1).
Obviously that’s a bad example just to explain how linear regression works (in two-dimensional space).

I guess my starting point would be to try out different models and comparing the resulting r2-values. Another interesting thing would be to test if the models get more accurate when leaving out certain data like an indicator. We could find out that a certain indicator brings no important additional information when used with others, or that we can predict best when using a certain subset of indicators used by SPB. In theory a more advanced model could even tell us that the EA “ABC” is always wrong with USDCAD at 5pm when another signal says buy EURCHF at the same time.

I don’t know if that answers your question, you can also write me a pm if you have additional questions or want to insult me for having to read all that :lol:

I want to add that the data would have to contain a “success” column to make any predictions, like profit resulting from a traded supersignal. That would be more difficult to add.
Author:  tomele [ Sat Jul 17, 2021 6:31 pm ]
Post subject:  Data from Desky - Machine Learning

Danyblu wrote:An easy example would be having the following unrealistic data:
Signal, profit
80, 10
85, 20
90, 30
100, 50
I expected something like this. The above couldn't be more far from being an "easy example". Here are (some of) the problems you will encounter:

1) TDesk doesn't know anything about profits of single trades. TDesk doesn't trade, that is Desky's domain. TDesk could only tell you at which exact point in time which exact signal occured. It even can't tell you anything about the indicator settings that did lead to a specific signal.

2) The profit of a trade is determined by many factors. Trade management (exit) has a much bigger effect than trade entry. We once even tested an EA with random entries to proof that. Not to mention complex startegies like grid and basket trading. What do you think why Desky has all those inputs and how will you take them into account? Your above table will soon grow to a 3-digit number of columns with all of them but one representing factors that determine the outcome of the profit column.

The nitty-gritty is that you will have to somehow permute all the factor combinations that make sense. This will be an overwhelming number of scenarios. You won't be able to master them by recording anything in real time. I am sorry to say this, but IMHO your idea leads nowhere.

However, there probably is a way to achieve what you are looking for. If I understand you right, you want to somehow score indicators by their aptness for trade entries. Funny enough, this is at the heart of a project I have started recently. The tool I am using is the MT5 simulator/backtester. I have started to create a multi-pair EA building kit that allows me to backtest and optimize any given indicator in combination with different trading and money management strategies.

Instead of inventing the wheel a second time, you might direct your attention to the MT5 strategy tester. Meanwhile it is a very powerful tool and has solved most of the problems that kept SHF members from backtesting all the years. Tick data, slippage, commission models, multi-pair trading and distributed computing to name a few.

Probably something profitable will result from this new approach.
Author:  Danyblu [ Mon Jul 19, 2021 8:01 am ]
Post subject:  Data from Desky - Machine Learning

Thank you for your input!

I expected the profit column to be a problem, I thought maybe Desky could remember opened trades and add profit to the relevant row as soon as it's closed, but I didn't know how hard that would be.
I had some thoughts to make exit as easy as possible, like just a fixed TP and SL for every trade.
What do you think why Desky has all those inputs and how will you take them into account? Your above table will soon grow to a 3-digit number of columns with all of them but one representing factors that determine the outcome of the profit column.
I don't know exactly what you mean by this, do these Desky-inputs stay constant after "activating" Desky? If so, they wouldn't be relevant for the evaluation and wouldn't need to be extra columns.
Also, there are some models you can use for a large number of columns without losing much speed, so just for the model this wouldn't be a problem either. For these models there can't be an overwhelming number of scenarios as they can work very efficiently with any number of factors.


I will think about what I can do with the MT5 simulator. I know it internally saves information of each simulated trade (profit), so you are definitely right that this will be a more practical approach. I will look into it.
I have started to create a multi-pair EA building kit that allows me to backtest and optimize any given indicator in combination with different trading and money management strategies.
Will you post about this project of yours? I would love to see this building kit to get an idea on how I could work with the MT5 strategy tester for my own project!

Thank you for your time! :)
All times are UTC Page 1 of 1