1

Below is my code, which starts with 1,000,000 permutations, then eliminates permutations that don't make sense (for example, a row where indicator_01 is greater than 6 and is less than 4). In this example, I am then working with 166,375 permutations.

The code is iterating through each of the 166,375 rows, and then doing a groupby to sum up the pnl (profit $), then produce an output which shows the best overall profit.

This process takes ~30 minutes. I understand from reading many questions here, that utilizing numpy could potentially speed this up considerably. I've been working on this for a few days and getting nowhere, so I'm posting my original code (iterating through each with lambda rows: and looking for the community's help.

###########################################################
### Backtest Optimizer - Brute Force - Three indicators ###
###########################################################

import pandas as pd
import itertools
import time
import numpy as np

pd.set_option('display.max_rows', 15)
tz = timezone('America/Denver')

x = 'QQQ'
trades_threshold = 1000 

print('Backtest Optimizer started at',datetime.now(tz).strftime('%H:%M:%S'))
checkpoint_01 = time.perf_counter()

filename = str('/content/'+x+'_summarized_trades.csv')
summarized_trades = pd.read_csv(filename)
print('Number of trades',len(summarized_trades))

##############################
list_of_indicators = ['rsi','std_pct','atr_5m/last_trade']
##############################

indicator_01 = list_of_indicators[0]
print('Indicator_01 is ',indicator_01)
indicator_02 = list_of_indicators[1]
print('Indicator_02 is ',indicator_02)
indicator_03 = list_of_indicators[2]
print('Indicator_03 is ',indicator_03)


summarized_trades['indicator_01_quantile'] = pd.qcut(summarized_trades[indicator_01].values, 10).codes
summarized_trades['indicator_02_quantile'] = pd.qcut(summarized_trades[indicator_02].values, 10).codes
summarized_trades['indicator_03_quantile'] = pd.qcut(summarized_trades[indicator_03].values, 10).codes
summarized_trades['indicator_01_quantile_min'] = pd.qcut(summarized_trades[indicator_01], 10).apply(lambda x: x.left)
summarized_trades['indicator_01_quantile_max'] = pd.qcut(summarized_trades[indicator_01], 10).apply(lambda x: x.right)
summarized_trades['indicator_02_quantile_min'] = pd.qcut(summarized_trades[indicator_02], 10).apply(lambda x: x.left)
summarized_trades['indicator_02_quantile_max'] = pd.qcut(summarized_trades[indicator_02], 10).apply(lambda x: x.right)
summarized_trades['indicator_03_quantile_min'] = pd.qcut(summarized_trades[indicator_03], 10).apply(lambda x: x.left)
summarized_trades['indicator_03_quantile_max'] = pd.qcut(summarized_trades[indicator_03], 10).apply(lambda x: x.right)

summarized_trades['number_of_trades'] = 1
summarized_trades['number_of_winners'] = summarized_trades.net_pnl.apply(lambda x: 1 if x >0 else 0)


### Group into similar quantile buckets
grouped_trades = summarized_trades.groupby(by=['indicator_01_quantile','indicator_02_quantile','indicator_03_quantile']).agg({'net_pnl':'sum','number_of_trades':'sum','number_of_winners':'sum','indicator_01_quantile_min':'min','indicator_01_quantile_max':'max','indicator_02_quantile_min':'min','indicator_02_quantile_max':'max','indicator_03_quantile_min':'min','indicator_03_quantile_max':'max'})
grouped_trades = grouped_trades.reset_index()  # Need to do this, otherwise unhashable
print('Length of grouped trades =',len(grouped_trades))
display(grouped_trades)



output = []
default_possibilities = [0,1,2,3,4,5,6,7,8,9] # Works for using 10 quantiles
possibilities = list(itertools.product(default_possibilities,default_possibilities,default_possibilities,default_possibilities,default_possibilities,default_possibilities))  # each of the three indicators can have 10x10 permutations
print('length of possibilities',len(possibilities))
possibilities = [item for item in possibilities if item[1]>=item[0] and item[3]>=item[2] and item[5]>=item[4]] #only keep combinations that make sense (e.g., tech indicator >2 and <4, but not >6 and <3)
print('length of revised possibilities',len(possibilities))

### iterate through the combinations that make sense
for i in possibilities:
  a = i[0]
  b = i[1]
  c = i[2]
  d = i[3]
  e = i[4]
  f = i[5]
  ### for each row, create new column to filter on, and if row is YES and has meets minimum trades_threshold, output to dataframe the results
  ### UTILIZE NUMPY INSTEAD OF LAMBDA ROW????
  grouped_trades['filter_01'] = grouped_trades.apply(lambda row: 'YES' if row['indicator_01_quantile'] >= a and row['indicator_01_quantile'] <= b and row['indicator_02_quantile'] >= c and row['indicator_02_quantile'] <= d and row['indicator_03_quantile'] >= e and row['indicator_03_quantile'] <= f else 'NO', axis=1)
  grouped_trades_filtered = grouped_trades.loc[(grouped_trades['filter_01']=='YES')]
  number_of_trades = grouped_trades_filtered.number_of_trades.sum()
  if number_of_trades < trades_threshold:
    continue
  else:
    number_of_winners = grouped_trades_filtered.number_of_winners.sum()
    net_pnl = grouped_trades_filtered.net_pnl.sum()
    output.append({'tech_indicator_01_min':a,'tech_indicator_01_max':b, 'tech_indicator_02_min':c,'tech_indicator_02_max':d, 'tech_indicator_03_min':e,'tech_indicator_03_max':f,'number_of_trades':number_of_trades,'number_of_winners':number_of_winners,'net_pnl':net_pnl})

output_df = pd.DataFrame(output)
print('Length of output',len(output_df))
output_df['win_rate'] = output_df.number_of_winners/output_df.number_of_trades
final_df = output_df.copy() ### if comparing with threshold above
final_df.sort_values(by='net_pnl', ascending=False, inplace=True)
display(final_df)
checkpoint_02 = time.perf_counter()
print('Total time',checkpoint_02-checkpoint_01)

# Take row 1 of the final_df, and take each value and use quantile
tech_indicator_01_min_quantile = final_df.iloc[0]['tech_indicator_01_min']
print(tech_indicator_01_min_quantile)
tech_indicator_01_max_quantile = final_df.iloc[0]['tech_indicator_01_max']
print(tech_indicator_01_max_quantile)
indicator_01_min_df = grouped_trades.loc[(grouped_trades.indicator_01_quantile == tech_indicator_01_min_quantile)]
tech_indicator_01_min_value = indicator_01_min_df.iloc[0]['indicator_01_quantile_min']
print(tech_indicator_01_min_value)
indicator_01_max_df = grouped_trades.loc[(grouped_trades.indicator_01_quantile == tech_indicator_01_max_quantile)]
tech_indicator_01_max_value = indicator_01_max_df.iloc[0]['indicator_01_quantile_max']
print(tech_indicator_01_max_value)

tech_indicator_02_min_quantile = final_df.iloc[0]['tech_indicator_02_min']
print(tech_indicator_02_min_quantile)
tech_indicator_02_max_quantile = final_df.iloc[0]['tech_indicator_02_max']
print(tech_indicator_02_max_quantile)
indicator_02_min_df = grouped_trades.loc[(grouped_trades.indicator_02_quantile == tech_indicator_02_min_quantile)]
tech_indicator_02_min_value = indicator_02_min_df.iloc[0]['indicator_02_quantile_min']
print(tech_indicator_02_min_value)
indicator_02_max_df = grouped_trades.loc[(grouped_trades.indicator_02_quantile == tech_indicator_02_max_quantile)]
tech_indicator_02_max_value = indicator_02_max_df.iloc[0]['indicator_02_quantile_max']
print(tech_indicator_02_max_value)

tech_indicator_03_min_quantile = final_df.iloc[0]['tech_indicator_03_min']
print(tech_indicator_03_min_quantile)
tech_indicator_03_max_quantile = final_df.iloc[0]['tech_indicator_03_max']
print(tech_indicator_03_max_quantile)
indicator_03_min_df = grouped_trades.loc[(grouped_trades.indicator_03_quantile == tech_indicator_03_min_quantile)]
tech_indicator_03_min_value = indicator_03_min_df.iloc[0]['indicator_03_quantile_min']
print(tech_indicator_03_min_value)
indicator_03_max_df = grouped_trades.loc[(grouped_trades.indicator_03_quantile == tech_indicator_03_max_quantile)]
tech_indicator_03_max_value = indicator_03_max_df.iloc[0]['indicator_03_quantile_max']
print(tech_indicator_03_max_value)


print(indicator_01,'>=',tech_indicator_01_min_value,'and',indicator_01,'<=',tech_indicator_01_max_value,'and',indicator_02,'>=',tech_indicator_02_min_value,'and',indicator_02,'<=',tech_indicator_02_max_value,'and',indicator_03,'>=',tech_indicator_03_min_value,'and',indicator_03,'<=',tech_indicator_03_max_value)

Edit: Here's an output using random data enter image description here

4
  • 1
    Can you provide a sample of grouped_trades, please?
    – Corralien
    Jun 4 at 16:51
  • @Corralien grouped_trades = pd.Dataframe({'indicator_01_quantile': {0: 0, 1: 0}, 'indicator_02_quantile': {0: 0, 1: 0}, 'indicator_03_quantile': {0: 0, 1: 1}, 'net_pnl': {0: 9.613, 1: -33.544}, 'number_of_trades': {0: 23, 1: 16}, 'number_of_winners': {0: 11, 1: 9}, 'indicator_01_quantile_min': {0: 1.497, 1: 1.497}, 'indicator_01_quantile_max': {0: 33.278, 1: 33.278}, 'indicator_02_quantile_min': {0: -0.000763, 1: -0.000763}, 'indicator_02_quantile_max': {0: 0.00107, 1: 0.00107}, 'indicator_03_quantile_min': {0: -0.00654, 1: 0.0035}, 'indicator_03_quantile_max': {0: 0.00035, 1: 0.00356}}
    – Jim
    Jun 4 at 18:22
  • Well, I figured out that I was making a simple mistake while trying to figure out how to use np.where. Below, I've got the corrected code for np.where, which takes about 350 seconds to run through, versus a bit over 2,000 seconds using lambda row: The issue was that I wasn't putting each condition in its own parenthesis between the & in my np.where statement with multiple conditions. This was giving an error of 'ambiguous result.
    – Jim
    Jun 4 at 22:51
  • Here's the correct code using np.where instead of lambda rows: grouped_trades['filter_01'] = np.where((grouped_trades['indicator_01_quantile'] >=a) & (grouped_trades['indicator_01_quantile'] <=b) & (grouped_trades['indicator_02_quantile'] >=c) & (grouped_trades['indicator_02_quantile'] <=d) & (grouped_trades['indicator_03_quantile'] >=e) & (grouped_trades['indicator_03_quantile'] <=f),'YES','NO')
    – Jim
    Jun 4 at 22:54

1 Answer 1

1

You can use numpy broadcasting (to compute the outer product of 2 arrays) to vectorize your operations. This is a common pattern to calculate the same operations for all rows at once (if you have enough memory...).

What you have to understand:

  • xxx[:, None]: add a new axis array([0, 1, 2])[:, None] => array([[0], [1], [2]])
  • np.sum(..., axis=n): compute the sum along an axis, not on whole array (dimension reduction).

The code below can be reduced at the expense of readability. So, replace your loop by this code:

# Convert to numpy
arr = np.array(possibilities)
lb, ub = arr[:, 0::2], arr[:, 1::2]

cols = ['indicator_01_quantile', 'indicator_02_quantile', 'indicator_03_quantile']
qtl = grouped_trades[cols].values

# Compute with number broadcasting
m1 = np.all(qtl >= lb[:, None], axis=2)  # lower bounds
m2 = np.all(qtl <= ub[:, None], axis=2)  # upper bounds
m3 = np.all(m1 & m2, axis=1)

number_of_trades = np.sum(m3[:, None] * grouped_trades['number_of_trades'].values, axis=1)
m4 = number_of_trades >= trades_threshold
number_of_winners = np.sum(m4[:, None] * grouped_trades['number_of_winners'].values, axis=1)
net_pnl = np.sum(m4[:, None] * grouped_trades['net_pnl'].values, axis=1)

# Create output dataframe
cols = ['tech_indicator_01_min', 'tech_indicator_01_max', 'tech_indicator_02_min',
        'tech_indicator_02_max', 'tech_indicator_03_min', 'tech_indicator_03_max']
df1 = pd.DataFrame(arr[m4], columns=cols)
df2 = pd.DataFrame({'number_of_trades': number_of_trades[m4],
                    'number_of_winners': number_of_winners[m4],
                    'net_pnl': net_pnl[m4]})
output_df = pd.concat([df1, df2], axis=1)
4
  • Thanks for the advice here, and this does increase speed dramatically, although I am not getting the expected result. the output_df I am getting when I insert your code instead of my for loop is returning only one row of data (where each indicator has a min of 0 and max of 9). When I am running my code on the same data, I am getting back an output_df with a length of 834 rows - all possible combinations where number_of_trades >= trades_threshold
    – Jim
    Jun 9 at 14:38
  • Can you share your data and the output of your run?
    – Corralien
    Jun 9 at 14:52
  • I'm not sure how to upload the data, but if we replace the data source with some random data, it should provide the same result -- only getting one row in output_df #summarized_trades = pd.read_csv(filename) summarized_trades = pd.DataFrame(np.random.randint(-100,100,size=(1000,4)), columns=['rsi','std_pct','atr_5m/last_trade','net_pnl'])
    – Jim
    Jun 9 at 16:31
  • I've added an image of the output to the original question when using the random data setup from the prior comment
    – Jim
    Jun 9 at 16:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.