2

I have a server that has great specs, dual 6 core 3.3GHz 196GB ram with RAID 10 across 4 10K SAS drives. I wrote a script that should download each of the North America files and process them one by one rather than the entire section all at once.

processList.sh:

wget http://download.geofabrik.de/north-america/us/alabama-latest.osm.pbf  -O ./geoFiles/north-america/us/alabama-latest.osm.pbf
osm2pgsql -d gis --create --slim  -G --hstore --tag-transform-script ~/src/openstreetmap-carto/openstreetmap-carto.lua -C 2000 --number-processes 15 -S ~/src/openstreetmap-carto/openstreetmap-carto.style ./geoFiles/north-america/us/alabama-latest.osm.$


while read in; 
   do wget http://download.geofabrik.de/$in -O ./geoFiles/$in; 
      osm2pgsql -d gis --append --slim  -G --hstore --tag-transform-script ~/src/openstreetmap-carto/openstreetmap-carto.lua -C 2000 --number-processes 15 -S ~/src/openstreetmap-carto/openstreetmap-carto.style ./geoFiles/$in;
done < maplist.txt

At first it starts out processing at nearly 400K points/second, then slows to 10k or less

osm2pgsql version 0.96.0 (64 bit id space)

Using lua based tag processing pipeline with script /root/src/openstreetmap-carto/openstreetmap-carto.lua
Using projection SRS 3857 (Spherical Mercator)
Setting up table: planet_osm_point
Setting up table: planet_osm_line
Setting up table: planet_osm_polygon
Setting up table: planet_osm_roads
Allocating memory for dense node cache
Allocating dense node cache in one big chunk
Allocating memory for sparse node cache
Sharing dense sparse
Node-cache: cache=2000MB, maxblocks=32000*65536, allocation method=11
Mid: pgsql, cache=2000
Setting up table: planet_osm_nodes
Setting up table: planet_osm_ways
Setting up table: planet_osm_rels

Reading in file: ./geoFiles/north-america/us/alabama-latest.osm.pbf
Using PBF parser.
Processing: Node(5580k 10.7k/s) Way(0k 0.00k/s) Relation(0 0.00/s))

I applied the performance stuff from https://wiki.openstreetmap.org/wiki/Osm2pgsql/benchmarks for Postgresql:

shared_buffers = 14GB
work_mem = 1GB                          
maintenance_work_mem = 8GB

effective_io_concurrency = 500          
max_worker_processes = 8                
max_parallel_workers_per_gather = 2     
max_parallel_workers = 8                

checkpoint_timeout = 1h         
max_wal_size = 5GB
min_wal_size = 1GB
checkpoint_completion_target = 0.9      

random_page_cost = 1.1                  
min_parallel_table_scan_size = 8MB
min_parallel_index_scan_size = 512kB
effective_cache_size = 22GB

Though it starts out well, it quickly deteriorates within about 20 seconds. Any idea why? I looked at top, but it didn't show anything really:

top - 22:48:46 up  3:11,  2 users,  load average: 3.49, 4.03, 3.38
Tasks: 298 total,   1 running, 297 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.0 us,  0.0 sy,  0.0 ni, 87.5 id, 12.5 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 19808144+total, 19237500+free,   780408 used,  4926040 buff/cache
KiB Swap: 29321212 total, 29321212 free,        0 used. 19437014+avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                                                                                                
16156 root      20   0 50.819g  75920   8440 S   0.7  0.0   0:02.81 osm2pgsql                                                                                                                                                                              
16295 root      20   0   42076   4156   3264 R   0.3  0.0   0:00.27 top                                                                                                                                                                                    
    1 root      20   0   37972   6024   4004 S   0.0  0.0   0:07.10 systemd                                                                                                                                                                                
    2 root      20   0       0      0      0 S   0.0  0.0   0:00.02 kthreadd                                                                                                                                                                               
    3 root      20   0       0      0      0 S   0.0  0.0   0:00.05 ksoftirqd/0                                                                                                                                                                            
    5 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/0:0H                                                                                                                                                                           
    6 root      20   0       0      0      0 S   0.0  0.0   0:00.58 kworker/u64:0                                                                                                                                                                          
    8 root      20   0       0      0      0 S   0.0  0.0   0:01.79 rcu_sched                                                                                                                                                                              
    9 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcu_bh                                                                                                                                                                                 
   10 root      rt   0       0      0      0 S   0.0  0.0   0:00.05 migration/0                                                                                                                                                                            
   11 root      rt   0       0      0      0 S   0.0  0.0   0:00.03 watchdog/0    

It had a large load average without listing anything as using it. Here are the results from iotop

Total DISK READ :       0.00 B/s | Total DISK WRITE :     591.32 K/s
Actual DISK READ:       0.00 B/s | Actual DISK WRITE:     204.69 K/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND                                                                                                                                                                                      
28638 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.60 % [kworker/u65:1]
20643 be/4 postgres    0.00 B/s  204.69 K/s  0.00 %  0.10 % postgres: wal writer process
20641 be/4 postgres    0.00 B/s  288.08 K/s  0.00 %  0.00 % postgres: checkpointer process
26923 be/4 postgres    0.00 B/s   98.55 K/s  0.00 %  0.00 % postgres: root gis [local] idle in transaction
    1 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % init
    2 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kthreadd]
    3 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/0]
    5 be/0 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kworker/0:0H]
    6 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kworker/u64:0]
    8 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [rcu_sched]
    9 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [rcu_bh]
   10 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/0]
   11 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/0]
   12 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/1]
4
  • 12.5 wa could indicate a slow storage. Check iotop for read / write performance.
    – scai
    Aug 1, 2018 at 6:42
  • Thanks for the idea, I haven't used iotop much. I added the results to my post, what do you think?
    – Alan
    Aug 1, 2018 at 7:08
  • The write speed seems to be rather slow, especially since the CPU is almost idle. However I'm not sure how an import looks like. If you are still stuck then try asking at help.openstreetmap.org
    – scai
    Aug 3, 2018 at 7:27
  • 1
    I will provide a more detailed answer later but I found that one of the HDD in the raid had failed. Using the hdparm -tT /dev/sda command, I was able to see the buffered read speeds were down to 5MB/sec instead of 200MB where it should have been. I removed the faulty drive which increased it's speed, but also replaced it with an M2 SSD with a speed of 1000MB/sec. I will let you know the results when I get it all finished.
    – Alan
    Aug 7, 2018 at 14:42

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Browse other questions tagged or ask your own question.