3

I am using the LMNN module from scikit-learn metric_learning (http://contrib.scikit-learn.org/metric-learn/index.html), and I am attempting to recover the linear transformation matrix (L.T) from the learned Mahalanobis (M) matrix.

The reason I am trying to recover this linear transformation is that I am fitting my dataset using cloud compute, but am testing it on a local machine. This means I can not save or recover the LMNN model after fitting on cloud compute, but I can save the learned M matrix and use a decomposition to find the learned linear transformation. I can then apply that learned linear transformation to my test sets on a local machine.

The problem is that I can't seem to reconcile the results from the LMNN module's built in transformation with the learned linear transformation from the decomposed M matrix. Here's an example:

import numpy as np
from metric_learn import LMNN
from sklearn.datasets import load_iris
iris_data = load_iris()
X = iris_data['data']
Y = iris_data['target']

lmnn = LMNN(k=5, learn_rate=1e-6)
X_transformed = lmnn.fit_transform(X, Y)
M_matrix = lmnn.get_mahalanobis_matrix()
array([[ 2.47937397,  0.36313715, -0.41243858, -0.78715282],
       [ 0.36313715,  1.69818843, -0.90042673, -0.0740197 ],
       [-0.41243858, -0.90042673,  2.37024271,  2.18292864],
       [-0.78715282, -0.0740197 ,  2.18292864,  2.9531315 ]])

# cholesky decomp of M_matrix
eigvalues, eigcolvectors = np.linalg.eig(M_matrix)
eigvalues_diag = np.diag(eigvalues)
eigvalues_diag_sqrt = np.sqrt(eigvalues_diag)
L = eigcolvectors.dot(eigvalues_diag_sqrt.dot(np.linalg.inv(eigcolvectors)))
L_transpose = np.transpose(L)
L_transpose.dot(L) # check to confirm that matches M_matrix
array([[ 2.47937397,  0.36313715, -0.41243858, -0.78715282],
       [ 0.36313715,  1.69818843, -0.90042673, -0.0740197 ],
       [-0.41243858, -0.90042673,  2.37024271,  2.18292864],
       [-0.78715282, -0.0740197 ,  2.18292864,  2.9531315 ]])

# test fit_transform() vs. transform() using LMNN functions
lmnn.transform(X[0:4, :])
array([[8.2487    , 4.41337015, 0.14988465, 0.52629361],
       [7.87314906, 3.77220291, 0.36015873, 0.525688  ],
       [7.59410008, 4.03369392, 0.17339877, 0.51350962],
       [7.41676205, 3.82012155, 0.47312948, 0.68515535]])

X_transformed[0:4, :]
array([[8.2487    , 4.41337015, 0.14988465, 0.52629361],
       [7.87314906, 3.77220291, 0.36015873, 0.525688  ],
       [7.59410008, 4.03369392, 0.17339877, 0.51350962],
       [7.41676205, 3.82012155, 0.47312948, 0.68515535]])

# test manual transform of X[0:4, :]
X[0:4, :].dot(L_transpose)
array([[8.22608756, 4.45271327, 0.24690081, 0.51206068],
       [7.85071271, 3.81054846, 0.45442718, 0.51144826],
       [7.57310259, 4.06981377, 0.26240745, 0.50067674],
       [7.39356544, 3.85511015, 0.55776916, 0.67615584]])

As seen above, the first four rows of the original dataset X[0:4, :] when transformed by the LMNN module (using either fit_transform(X, Y) or transform(X[0:4, :]) give different results from the manual transformation.

class MetricTransformer(six.with_metaclass(ABCMeta)):

  @abstractmethod
  def transform(self, X):
    """Applies the metric transformation.
    Parameters
    ----------
    X : (n x d) matrix
      Data to transform.
    Returns
    -------
    transformed : (n x d) matrix
      Input data transformed to the metric space by :math:`XL^{\\top}`

class MahalanobisMixin(six.with_metaclass(ABCMeta, BaseMetricLearner,
                                          MetricTransformer)):
  r"""Mahalanobis metric learning algorithms.
  Algorithm that learns a Mahalanobis (pseudo) distance :math:`d_M(x, x')`,
  defined between two column vectors :math:`x` and :math:`x'` by: :math:`d_M(x,
  x') = \sqrt{(x-x')^T M (x-x')}`, where :math:`M` is a learned symmetric
  positive semi-definite (PSD) matrix. The metric between points can then be
  expressed as the euclidean distance between points embedded in a new space
  through a linear transformation. Indeed, the above matrix can be decomposed
  into the product of two transpose matrices (through SVD or Cholesky
  decomposition): :math:`d_M(x, x')^2 = (x-x')^T M (x-x') = (x-x')^T L^T L
  (x-x') = (L x - L x')^T (L x- L x')`
  • What am I missing here?

Thanks!

2
  • Have you tried using numpy.linalg.svd instead of np.linalg.eig?
    – Aramakus
    Commented May 14, 2020 at 23:10
  • I did, same result. Commented May 15, 2020 at 22:47

2 Answers 2

1

metric-learn contributor here, @BeginnersMindTruly you're right, for LMNN we indeed learn the L matrix directly during training, from which we compute M at the end, so computing back the transformation L from M may lead to numerical differences.

As for your particular use case of accessing directly the learned matrix L, you should be able to do that using the components_ attribute of your metric learner, at the end of training.

1

Per contributors from the team who built the module (http://contrib.scikit-learn.org/metric-learn/index.html), this discrepancy is due to floating point precision errors. The LMNN module first computes the linear transformation L.T then computes the M matrix by computing L.T.dot(L). So any attempt to recover the original transformation loses precision both during the computation of M and then the refactoring M.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.