0

EDIT It appears that it isn't necessarily a problem with the data in row 64. Rather the number 64 itself is magical and causes the problems. As I have continued to troubleshoot the problem, I wrote a script that randomly grabs 63 contiguous rows from the DataFrame and plots them. It runs quickly every time. But if I change it to 64 rows it never works and runs forever. End edit

I'm trying to visualize data in clusters using multidimensional scaling. I've created a DataFrame and that has dimensions of 1000 rows and 1964 columns. When I try to perform multidimensional scaling on the data the process runs forever. Curiously, I can't seem to end the process by doing a ctrl+c.

Through a process of trial and error I have discovered something magical about the 64th row of the dataset. If I run the process on 63 rows the whole thing is done in a couple seconds. If I bump that up to 64 rows, though, it will never end.

I'm really at a loss as to how I should even approach troubleshooting this. I went through the 1964 columns looking for differences between row 63 and row 64 hoping to find a strange value or something but nothing jumped out at me. Any other way I can get an idea of why 64 rows is so magical?

import pandas as pd
from pandas import DataFrame as df
from sklearn.metrics.pairwise import euclidean_distances
from sklearn.metrics.pairwise import manhattan_distances
from sklearn import manifold
from matplotlib import pyplot as plt
import prettyplotlib as ppl

malware_df = df.from_csv('malware_features.csv')

plottable = malware_df[[c for c in malware_df.columns if c != 'hash']]
plottable = plottable.head(63) # change this to 64 and everything stops working

euc = euclidean_distances(plottable)
mds = manifold.MDS(n_jobs=-1, random_state=1337, dissimilarity='precomputed')
pos_e = mds.fit(euc).embedding_

plottable['xpos'] = pos_e[:,0]
plottable['ypos'] = pos_e[:,1]
with ppl.pretty:
    fig, ax = ppl.subplots(figsize=(6,8))
ppl.scatter(ax, plottable.xpos, plottable.ypos)
plt.show()

Here is a link where you can download the file I'm using if that helps.https://drive.google.com/file/d/0BxZZOOgLl7vSTUlxc1BmMUFmTVU/edit?usp=sharing

2
  • Is 64 still magic if you drop one of the columns? My hypothesis is that you could be running into some memory boundary and thrashing or similar Sep 5, 2014 at 8:40
  • Sorry about that @AndyHayden I corrected the code snippet to explain how pos_e was being initialized. It is the positions of the points based on euclidean distance. Sep 8, 2014 at 16:01

1 Answer 1

1
+50

This has to be something with versions. In my computer (2003, 1 AMD core, 2 Gb RAM), this code runs in ~3 seconds:

#import pandas as pd
from pandas import DataFrame as df
from sklearn.metrics.pairwise import euclidean_distances
#from sklearn.metrics.pairwise import manhattan_distances
from sklearn import manifold
from matplotlib import pyplot as plt 
import prettyplotlib as ppl 

malware_df = df.from_csv('malware_features.csv')

plottable = malware_df[[c for c in malware_df.columns if c != 'hash']]
plottable = plottable.head(128) # change this to 64 and everything stops working

euc = euclidean_distances(plottable)
mds = manifold.MDS(n_jobs=-1, random_state=1337, dissimilarity='precomputed')
pos_e = mds.fit(euc).embedding_

plottable['xpos'] = pos_e[:,0]
plottable['ypos'] = pos_e[:,1]

fig, ax = ppl.subplots(figsize=(6,8))

ppl.scatter(ax, plottable.xpos, plottable.ypos)
plt.show()

To produce this graphic:

enter image description here

Notice I tried with 128 after trying 64 and not failing, to see what happened, changed the with ppl.pretty that raises an Error, and everything run fine. This is my pip freeze:

brewer2mpl==1.4
matplotlib==1.4.0
numpy==1.9.0
pandas==0.14.1
prettyplotlib==0.1.7
reportlab==3.1.8
scikit-learn==0.15.2
scipy==0.14.0

And python 2.7.3.

1
  • That was totally it. I didn't think of looking to see if my packages were out of date. I just ran this with 2000 rows and it ran for between 7 to 10 seconds. Sep 10, 2014 at 14:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.