Let me take a real example to address this question
I needed to calculate a weighted moving average on my ohlc data, I have about 134000 candles with a symbol for each to do so
- Option 1 Do it in Python/Node etc etc
- Option 2 Do it in SQL itself!
Which one is better?
- If I had to do this in Python, essentially, I would have to fetch all stored records at the worst, case, perform the computation and save everything back which in my opinion is a huge wastage of IO
- Weighted moving average changes everytime you get a new candle meaning I would be doing massive amounts of IO at regular intervals which is not a
good opinion in my sign
- In SQL, all I have to do is probably write a trigger that computes and stores everything so only need to fetch the final WMA values for each pair every now and then and that is so much more efficient
Requirements
- If I had to calculate WMA for every candle and store it, I would do it on Python
- But since I only need the last value, SQL is much faster than Python
To give you some encouragement, this is the Python version to do a weighted moving average
WMA done through code
import psycopg2
import psycopg2.extras
from talib import func
import timeit
import numpy as np
with psycopg2.connect('dbname=xyz user=xyz') as conn:
with conn.cursor() as cur:
t0 = timeit.default_timer()
cur.execute('select distinct symbol from ohlc_900 order by symbol')
for symbol in cur.fetchall():
cur.execute('select c from ohlc_900 where symbol = %s order by ts', symbol)
ohlc = np.array(cur.fetchall(), dtype = ([('c', 'f8')]))
wma = func.WMA(ohlc['c'], 10)
# print(*symbol, wma[-1])
print(timeit.default_timer() - t0)
conn.close()
WMA Through SQL
"""
if the period is 10
then we need 9 previous candles or 15 x 9 = 135 mins on the interval department
we also need to start counting at row number - (count in that group - 10)
For example if AAPL had 134 coins and current row number was 125
weight at that row will be weight = 125 - (134 - 10) = 1
10 period WMA calculations
Row no Weight c
125 1
126 2
127 3
128 4
129 5
130 6
131 7
132 8
133 9
134 10
"""
query2 = """
WITH
condition(sym, maxts, cnt) as (
select symbol, max(ts), count(symbol) from ohlc_900 group by symbol
),
cte as (
select symbol, ts,
case when cnt >= 10 and ts >= maxts - interval '135 mins'
then (row_number() over (partition by symbol order by ts) - (cnt - 10)) * c
else null
end as weighted_close
from ohlc_900
INNER JOIN condition
ON symbol = sym
WINDOW
w as (partition by symbol order by ts rows between 9 preceding and current row)
)
select symbol, sum(weighted_close)/55 as wma
from cte
WHERE weighted_close is NOT NULL
GROUP by symbol ORDER BY symbol
"""
with psycopg2.connect('dbname=xyz user=xyz') as conn:
with conn.cursor() as cur:
t0 = timeit.default_timer()
cur.execute(query2)
# for i in cur.fetchall():
# print(*i)
print(timeit.default_timer() - t0)
conn.close()
Believe it or not, the query runs faster than the Pure Python version of doing a WEIGHTED MOVING AVERAGE!!! I went step by step into writing that query so hang in there and you ll do just fine
Speed
0.42141127300055814 seconds Python
0.23801879299935536 seconds SQL
I have 134000 fake OHLC records in my database divided amongst a 1000 stocks so that is an example of where SQL can outperform your app server