0

I am trying to cluster retail data in order to extract groupings of customers based on 6 input features. The data has a shape of (1712594, 6) in the following format:

Features

I've spilt the 'Department' categorical variable into binary n-dimensional array using Pandas get_dummies(). I'm aware this is not optimal but I just wanted to test it out before trying out Gower Distances.

The Elbow method gives the following output: Elbow Method

USING: I'm using Python and Scikitlearn's KMeans because the dataset is so large and the more complex models are too computationally demanding for Google Colab.

OBSERVATINS: I'm aware that columns 1-5 are extremely correlated but the data is limited Sales data and little to no data is captured about Customers. KMeans is very sensitive to inputs and this may affect the WCSS in the Elbow Method and cause the straight line but this is just an inclination and I don't have any quantitative backing to support the argument. I'm a Junior Data Scientist so knowledge about technical foundations of Clustering models and algorithms is still developing so forgive me if I'm missing something.

WHAT I'VE DONE: There were massive outliers that were skewing the data (this is a Building Goods company and therefore most of their sale prices and quantities fall within a certain range. But ~5% of the data contained massive quantity entries (eg. a company buying 300000 bricks at R3/brick) or massive price entries (eg. company buying an expensive piece of equipment).

I've removed them and maintained ~94% of the data. I've also removed the returns made by customers (ie. negative quantities and prices) under the inclination that I may create a binary variable 'Returned' to capture this feature. Here are some metrics:

These are some metrics before removing the outliers: Input statistics before Outlier removal

and these are the metrics after Outlier removal: Input statistics AFTER outlier removal

KMeans uses Euclidean distances. I've used both Scikitlearn's StandardScaler and RobustScaler when scaling without any significant changes in both. Here are some distribution plots and scatter plots for the 3 numeric variables: DistPlot of Quantity DistPlot of Price DistPlot of Gross Profit

Quantity to Price Quantity to GP Gp to Price all 3

Anybody have any practical/intuitive reasoning as to why this may be happening? Open to any alternative methods to use as well and any help would be much appreciated! Thanks

8
  • 1
    Does each row correspond to a sample? Did you standardize your columns? Which distance did you use? DId you try to visualize the data to understand what's going on?
    – PejoPhylo
    May 15, 2020 at 13:25
  • can you include more information... like whether the data was scaled etc.. It might be that the data is like this, but can't say more until you described whats done
    – StupidWolf
    May 15, 2020 at 23:15
  • @PejoPhylo I've updated the post to add some of the workings I've done May 16, 2020 at 10:51
  • U might want to sample e dataset n plot to see if there is any structure.. w a 2d density. From what it looks like now kmeans is just going to keep cutting the data
    – StupidWolf
    May 16, 2020 at 11:12
  • And it might well be like that. The elbow plot.. really depends on Ur data. Many times u just don't find it
    – StupidWolf
    May 16, 2020 at 11:12

1 Answer 1

0

I am not an expert, in my experience with scikit learn cluster analysis I find that when the features are really similar in magnitude K-means clustering usually does not fulfill the job. I will first try to use a StandardScaler to see if normalizing the data makes the clustering more efficient. the elbow plot shows that with more n_neighbors you get higher accuracy, and by the looks of the plot and the plots you provide, I would think the data is too similar, making it hard to separate into groups (clusters). Adding an additional feature made up of your data can do the trick.

  1. I would try normalizing the data first, standard scaler.
  2. If the groups are still not very clear with a simple plot of the data I would create another column made up of the combination of the others columns.
  3. I would not suggest using DBSCAN, since the eps parameter (distance) would have to be tunned very finely and as you mention is more computationally expensive.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.