I'm trying to build a ETL toolkit with pandas, hdf5.
My plan was
- extracting a table from mysql to a DataFrame;
- put this DataFrame into a HDFStore;
But when i was doing the step 2, i found putting a dataframe into a *.h5 file costs too much time.
- the size of table in source mysql server: 498MB
- 52 columns
- 924,624 records
- the size of *.h5 file after putting the dataframe inside : 513MB
- the 'put' operation costs 849.345677137 seconds
My questions are:
Is this time costs normal?
Is there any way to make it faster?
Update 1
thanks Jeff
my codes are pretty simple:
extract_store = HDFStore('extract_store.h5')
extract_store['df_staff'] = df_staff- and when i trying 'ptdump -av file.h5', i got an error, but i still could load the dataframe object from this h5 file:
tables.exceptions.HDF5ExtError: HDF5 error back trace
File "../../../src/H5F.c", line 1512, in H5Fopen
unable to open file File "../../../src/H5F.c", line 1307, in H5F_open
unable to read superblock File "../../../src/H5Fsuper.c", line 305, in H5F_super_read
unable to find file signature File "../../../src/H5Fsuper.c", line 153, in H5F_locate_signature
unable to find a valid file signatureEnd of HDF5 error back trace
Unable to open/create file 'extract_store.h5'
- some other infos:
- pandas version: '0.10.0'
- os: ubuntu server 10.04 x86_64
- cpu: 8 * Intel(R) Xeon(R) CPU X5670 @ 2.93GHz
- MemTotal: 51634016 kB
I will update the pandas to 0.10.1-dev and try again.
Update 2
- I had updated pandas to '0.10.1.dev-6e2b6ea'
- but the time costs wasn't decreased, it costs 884.15 s seconds this time
- the output of 'ptdump -av file.h5 ' is :
/ (RootGroup) '' /._v_attrs (AttributeSet), 4 attributes: [CLASS := 'GROUP', PYTABLES_FORMAT_VERSION := '2.0', TITLE := '', VERSION := '1.0'] /df_bugs (Group) '' /df_bugs._v_attrs (AttributeSet), 12 attributes: [CLASS := 'GROUP', TITLE := '', VERSION := '1.0', axis0_variety := 'regular', axis1_variety := 'regular', block0_items_variety := 'regular', block1_items_variety := 'regular', block2_items_variety := 'regular', nblocks := 3, ndim := 2, pandas_type := 'frame', pandas_version := '0.10.1'] /df_bugs/axis0 (Array(52,)) '' atom := StringAtom(itemsize=19, shape=(), dflt='') maindim := 0 flavor := 'numpy' byteorder := 'irrelevant' chunkshape := None /df_bugs/axis0._v_attrs (AttributeSet), 7 attributes: [CLASS := 'ARRAY', FLAVOR := 'numpy', TITLE := '', VERSION := '2.3', kind := 'string', name := None, transposed := True] /df_bugs/axis1 (Array(924624,)) '' atom := Int64Atom(shape=(), dflt=0) maindim := 0 flavor := 'numpy' byteorder := 'little' chunkshape := None /df_bugs/axis1._v_attrs (AttributeSet), 7 attributes: [CLASS := 'ARRAY', FLAVOR := 'numpy', TITLE := '', VERSION := '2.3', kind := 'integer', name := None, transposed := True] /df_bugs/block0_items (Array(5,)) '' atom := StringAtom(itemsize=12, shape=(), dflt='') maindim := 0 flavor := 'numpy' byteorder := 'irrelevant' chunkshape := None /df_bugs/block0_items._v_attrs (AttributeSet), 7 attributes: [CLASS := 'ARRAY', FLAVOR := 'numpy', TITLE := '', VERSION := '2.3', kind := 'string', name := None, transposed := True] /df_bugs/block0_values (Array(924624, 5)) '' atom := Float64Atom(shape=(), dflt=0.0) maindim := 0 flavor := 'numpy' byteorder := 'little' chunkshape := None /df_bugs/block0_values._v_attrs (AttributeSet), 5 attributes: [CLASS := 'ARRAY', FLAVOR := 'numpy', TITLE := '', VERSION := '2.3', transposed := True] /df_bugs/block1_items (Array(19,)) '' atom := StringAtom(itemsize=19, shape=(), dflt='') maindim := 0 flavor := 'numpy' byteorder := 'irrelevant' chunkshape := None /df_bugs/block1_items._v_attrs (AttributeSet), 7 attributes: [CLASS := 'ARRAY', FLAVOR := 'numpy', TITLE := '', VERSION := '2.3', kind := 'string', name := None, transposed := True] /df_bugs/block1_values (Array(924624, 19)) '' atom := Int64Atom(shape=(), dflt=0) maindim := 0 flavor := 'numpy' byteorder := 'little' chunkshape := None /df_bugs/block1_values._v_attrs (AttributeSet), 5 attributes: [CLASS := 'ARRAY', FLAVOR := 'numpy', TITLE := '', VERSION := '2.3', transposed := True] /df_bugs/block2_items (Array(28,)) '' atom := StringAtom(itemsize=18, shape=(), dflt='') maindim := 0 flavor := 'numpy' byteorder := 'irrelevant' chunkshape := None /df_bugs/block2_items._v_attrs (AttributeSet), 7 attributes: [CLASS := 'ARRAY', FLAVOR := 'numpy', TITLE := '', VERSION := '2.3', kind := 'string', name := None, transposed := True] /df_bugs/block2_values (VLArray(1,)) '' atom = ObjectAtom() byteorder = 'irrelevant' nrows = 1 flavor = 'numpy' /df_bugs/block2_values._v_attrs (AttributeSet), 5 attributes: [CLASS := 'VLARRAY', PSEUDOATOM := 'object', TITLE := '', VERSION := '1.3', transposed := True]
- and I had tried your code below (putting the dataframe into hdfstore with the param 'table' is True) , but got an error instead, it seemed like python's datatime type was not supported :
Exception: cannot find the correct atom type -> [dtype->object] object of type 'datetime.datetime' has no len()
Update 3
thanks jeff. Sorry for the delay.
- tables.version : '2.4.0'.
- yes, the 884 seconds is only the put operation costs without the pull operation from mysql
- a row of dataframe (df.ix[0]):
bug_id 1 assigned_to 185 bug_file_loc None bug_severity critical bug_status closed creation_ts 1998-05-06 21:27:00 delta_ts 2012-05-09 14:41:41 short_desc Two cursors. host_op_sys Unknown guest_op_sys Unknown priority P3 rep_platform IA32 reporter 56 product_id 7 category_id 983 component_id 12925 resolution fixed target_milestone ws1 qa_contact 412 status_whiteboard votes 0 keywords SR lastdiffed 2012-05-09 14:41:41 everconfirmed 1 reporter_accessible 1 cclist_accessible 1 estimated_time 0.00 remaining_time 0.00 deadline None alias None found_in_product_id 0 found_in_version_id 0 found_in_phase_id 0 cf_type Defect cf_reported_by Development cf_attempted NaN cf_failed NaN cf_public_summary cf_doc_impact 0 cf_security 0 cf_build NaN cf_branch cf_change NaN cf_test_id NaN cf_regression Unknown cf_reviewer 0 cf_on_hold 0 cf_public_severity --- cf_i18n_impact 0 cf_eta None cf_bug_source --- cf_viss None Name: 0, Length: 52
- the picture of dataframe( just type 'df' in ipython notebook):
Int64Index: 924624 entries, 0 to 924623 Data columns: bug_id 924624 non-null values assigned_to 924624 non-null values bug_file_loc 427318 non-null values bug_severity 924624 non-null values bug_status 924624 non-null values creation_ts 924624 non-null values delta_ts 924624 non-null values short_desc 924624 non-null values host_op_sys 924624 non-null values guest_op_sys 924624 non-null values priority 924624 non-null values rep_platform 924624 non-null values reporter 924624 non-null values product_id 924624 non-null values category_id 924624 non-null values component_id 924624 non-null values resolution 924624 non-null values target_milestone 924624 non-null values qa_contact 924624 non-null values status_whiteboard 924624 non-null values votes 924624 non-null values keywords 924624 non-null values lastdiffed 924509 non-null values everconfirmed 924624 non-null values reporter_accessible 924624 non-null values cclist_accessible 924624 non-null values estimated_time 924624 non-null values remaining_time 924624 non-null values deadline 0 non-null values alias 0 non-null values found_in_product_id 924624 non-null values found_in_version_id 924624 non-null values found_in_phase_id 924624 non-null values cf_type 924624 non-null values cf_reported_by 924624 non-null values cf_attempted 89622 non-null values cf_failed 89587 non-null values cf_public_summary 510799 non-null values cf_doc_impact 924624 non-null values cf_security 924624 non-null values cf_build 327460 non-null values cf_branch 614929 non-null values cf_change 300612 non-null values cf_test_id 12610 non-null values cf_regression 924624 non-null values cf_reviewer 924624 non-null values cf_on_hold 924624 non-null values cf_public_severity 924624 non-null values cf_i18n_impact 924624 non-null values cf_eta 3910 non-null values cf_bug_source 924624 non-null values cf_viss 725 non-null values dtypes: float64(5), int64(19), object(28)
- after 'convert_objects()':
dtypes: datetime64[ns](2), float64(5), int64(19), object(26)
- and putting the converted dataframe into hdfstore costs: 749.50 s :)
- it seems that reducing the number of 'object' dtypes is the key to decrease time costs
- and putting the converted dataframe into hdfstore with the param 'table' is true still returns that error
/usr/local/lib/python2.6/dist-packages/pandas-0.10.1.dev_6e2b6ea-py2.6-linux-x86_64.egg/pandas/io/pytables.pyc in create_axes(self, axes, obj, validate, nan_rep, data_columns, min_itemsize, **kwargs) 2203 raise 2204 except (Exception), detail: -> 2205 raise Exception("cannot find the correct atom type -> [dtype->%s] %s" % (b.dtype.name, str(detail))) 2206 j += 1 2207 Exception: cannot find the correct atom type -> [dtype->object] object of type 'datetime.datetime' has no len()
- I'm trying to put the dataframe without datetime columns
Update 4
- There are 4 columns in mysql whose type is datetime:
- creation_ts
- delta_ts
- lastdiffed
- deadline
After calling the convert_objects():
- creation_ts:
Timestamp: 1998-05-06 21:27:00
- delta_ts:
Timestamp: 2012-05-09 14:41:41
- lastdiffed
datetime.datetime(2012, 5, 9, 14, 41, 41)
- deadline is always None, no matter before or after calling 'convert_objects'
None
- putting the dataframe without column 'lastdiff' costs 691.75 s
- when putting the dataframe without column 'lastdiff' and setting param 'table' equal to True, I got an new error, :
/usr/local/lib/python2.6/dist-packages/pandas-0.10.1.dev_6e2b6ea-py2.6-linux-x86_64.egg/pandas/io/pytables.pyc in create_axes(self, axes, obj, validate, nan_rep, data_columns, min_itemsize, **kwargs) 2203 raise 2204 except (Exception), detail: -> 2205 raise Exception("cannot find the correct atom type -> [dtype->%s] %s" % (b.dtype.name, str(detail))) 2206 j += 1 2207 Exception: cannot find the correct atom type -> [dtype->object] object of type 'Decimal' has no len()
- the type of columns 'estimated_time', 'remaining_time', 'cf_viss' is 'decimal' in mysql
Update 5
- I had transformed these 'decimal' type columns to 'float' type, by the code below:
no_diffed_converted_df_bugs.estimated_time = no_diffed_converted_df_bugs.estimated_time.map(float)
- and now, the time costs is 372.84 s
- but the 'table' version putting still raised an error:
/usr/local/lib/python2.6/dist-packages/pandas-0.10.1.dev_6e2b6ea-py2.6-linux-x86_64.egg/pandas/io/pytables.pyc in create_axes(self, axes, obj, validate, nan_rep, data_columns, min_itemsize, **kwargs) 2203 raise 2204 except (Exception), detail: -> 2205 raise Exception("cannot find the correct atom type -> [dtype->%s] %s" % (b.dtype.name, str(detail))) 2206 j += 1 2207 Exception: cannot find the correct atom type -> [dtype->object] object of type 'datetime.date' has no len()