0

We have SAP HANA 1.0 SP11. We have one requirement where we need to calculate current stock at store, material level on daily basis. No of rows expected are around 250 million.

Currently we use procedure for same. Flow of procedure is as follows -

begin

t_rst = select * from <LOGIC of deriving current stock on tables MARD,MARC,MBEW>;

select count(*) into v_cnt from :t_rst;

v_loop = v_cnt/2500000;


FOR X in 0 .. v_loop  DO

INSERT INTO CRRENT_STOCK_TABLE
SELECT * FROM :t_rst LIMIT 2500000 OFFSET :count;
COMMIT;
count := count + 2500000;    
END FOR;

end;

Row count of result set t_rst is around 250 million. Total execution time of procedure time is around 2.5 hours. Few times procedure goes into long running state resulting into error. We run this procedure in non peak hours of business so load on system is almost nothing.

Is there a way, we can load data in target table in parallel threads and reduce loading time. Also, is there way to bulk insert efficiently in HANA.

Query for t_rst fetches first 1000 rows in 5 minutes.

6
  • What is the reason for this batching construct? Does your system lack resources to support a simple INSERT ...(SELECT ...))? Also, do you control the delta merge of the target table during your data load in any way?
    – Lars Br.
    Jan 21, 2019 at 4:10
  • Have you considered the EXPORT/IMPORT SQL statement? Might not be ideal but il allows you to manage threads, locks etc? The downside will be that you will physically create a file on the server but running an export with no data will allow you to get rid of it (if I'm not mistaken). Jan 21, 2019 at 8:47
  • @LarsBr. If I insert in single statement, won't it Delta merge after inserting 220 million records resulting in large amount of redo log and huge memory consumption
    – Anirudh D
    Jan 21, 2019 at 9:15
  • Do you need all columns? Instead of using "*" for all columns, maybe you can only copy required data into target table.
    – Eralper
    Jan 21, 2019 at 16:05
  • @Eralper t_rst contains required columns only
    – Anirudh D
    Jan 21, 2019 at 16:08

1 Answer 1

0

As Lars, mentioned the total resource usage will not change effectively

But if you have limited time (non-peak hours) and if the system configuration will overcome to the requirements of parallel execution, maybe you can try using

BEGIN PARALLEL EXECUTION 
   <stmt> 
END;

Please refer to reference documentation

After you calculate v_loop value, you know how many times you have to run following INSERT command

INSERT INTO CRRENT_STOCK_TABLE
SELECT * FROM :t_rst LIMIT 2500000 OFFSET :count;

I'm not sure how to convert above code into a dynamic calculation for PARALLEL EXECUTION

But you can assume let's say 10 parallel processes, and run that many INSERT command by modifying the OFFSET clause according to calculated values

The ones that you exceed will run for zero rows which will not harm the overall process

As a response to @LarsBr. , as he mentioned there are limitations that will prevent parallel execution

Restrictions and Limitations

The following restrictions apply:

Modification of tables with a foreign key or triggers are not allowed

Updating the same table in different statements is not allowed

Only concurrent reads on one table are allowed. Implicit SELECT and SELCT INTO scalar variable statements are supported.

Calling procedures containing dynamic SQL (for example, EXEC, EXECUTE IMMEDIATE) is not supported in parallel blocks

Mixing read-only procedure calls and read-write procedure calls in a parallel block is not allowed.

These limitations saying, insert to same table will not be possible from different executions and dynamic SQL cannot be used too

4
  • 1
    This will not work on as the insert targets the same table. This is explicitly not allowed for PARALLEL block DML. (help.sap.com/viewer/de2486ee947e43e684d39702027f8a94/2.0.03/…)
    – Lars Br.
    Jan 22, 2019 at 6:09
  • @LarsBr.&@Eralper Will it work if I partition the target table and load data into diffrent partitions
    – Anirudh D
    Jan 22, 2019 at 6:14
  • Thank you for response, I added limitations that preventing use of parallel execution in this case. Can we insert into different partitions of the same table with parallel execution?
    – Eralper
    Jan 22, 2019 at 6:27
  • 4
    Yes, parallelized INSERT/UPDATE into different partitions is possible and you don't even need to use the explicit parallel clause for that. Just running the single INSERT INTO (SELECT...) will already parallelize the writing part.
    – Lars Br.
    Jan 22, 2019 at 6:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.