0

I am trying for quite long time to do the following in Python, for a new app that I am currently working on:

Inputs:

  1. list_of_points - n sample points x. Each has d dimensions (each is represented as a list), where d is assumed to be large (the larger the better).

  2. list_of_values - n values "hidden function" f. That is, list_of_values[i] = f(list_of_points[i]) + noise

  3. domain - the d-minsional range on which f is defined, and thus list_of_points are also in this domain. The domain can possibly be the whole R^d.

Output: list of points that are global maxima of f on domain.

In order to filter the noise, I decided to use regression to estimate f, and only then to look for its maxima points.

Summarizing the above to a pseudo-code, we get:

def filter_noise_and_return_global_maxima(list_of_points, list_of_values, domain, degree_of_regression_polynomial):
    f = polynomial_regression(list_of_points, list_of_values, degree = degree_of_regression_polynomial)
    return f.global_maxima(domain)

Now I was trying to figure out how to do that, and I could not find any way to do it, especially if I want the code to run in a reasonable time (considering that the dimension d is rather big).

For the polynomial regression I found the following in the web:

from sklearn.preprocessing import PolynomialFeatures
from sklearn import linear_model

X = [[0.44, 0.68], [0.99, 0.23]]
vector = [109.85, 155.72]
predict= [0.49, 0.18]

poly = PolynomialFeatures(degree=2)
X_ = poly.fit_transform(X)
predict_ = poly.fit_transform(predict)

clf = linear_model.LinearRegression()
clf.fit(X_, vector)
print(clf.predict(predict_))

But unfortunately this raises errors. Also, I am not sure how to produce the polynomial for sympy that is used below.

Finally, for finding the global maxima I found this:

from sympy.calculus.util import *
x = symbols('x')
f = (x**3 / 3) - (2 * x**2) - 3 * x + 1

print(minimum(f, x, ivl))
print(maximum(f, x, ivl))
print(stationary_points(f, x, ivl))

But I am not sure if choosing a symbolic computatin is a good choise as I deal with a large dimension, and also I didn't realize how to use it for the multivariate case. For example, the folloing does not work f = x[0]**2+ 2*x[1].

5
  • Do you have y or dependent variable on your datasets? Or you only have x or independent variables (train and test)? If you don't have y then you cannot fit.
    – ferdy
    Commented Nov 26, 2021 at 5:25
  • Yes, sure. I edited my question to emphasize that
    – Dudi Frid
    Commented Nov 26, 2021 at 7:07
  • Can you give a single sample in the list_of_points?
    – ferdy
    Commented Nov 27, 2021 at 2:36
  • list_of_points=[[1,2,3.2],[2,4.2, 5.1], [-1,0.1, 0.9]] list_of_values=[4,5,6], domain=[[-10,10], [-10,10], [-10, 10]]#-10<=x,y,z<=10, degree_of_regression_polynomial=1
    – Dudi Frid
    Commented Nov 27, 2021 at 7:23
  • p1=[1,2,3.2], p2=[2,4.2, 5.1], p3=[-1,0.1, 0.9] and their corresponding values are 4, 5 and 6 respectively. Was this correct? We will create a model that will fit these values. From the given p1 with dimension 3 plus its value we are now dealing with a 4D data is this right? What exactly is Output: list of points that are global maxima of f on domain. Do you mean that after building our model, we will search the point with a maximum value?
    – ferdy
    Commented Nov 29, 2021 at 0:01

1 Answer 1

0

Here is one approach of how to use linear regression with polynomial features which uses your pseudo code.

Code

import numpy as np
from sklearn.preprocessing import PolynomialFeatures
from sklearn import linear_model
from sklearn.metrics import mean_squared_error


def minimize_mse():
    num_processor = 1
    X_train = [[0.44, 0.68], [0.99, 0.23]]
    X_test = [[0.24, 0.48], [0.29, 0.73]]

    print(f'X_train rows: {np.array(X_train).shape[0]}, X_train columns: {np.array(X_train).shape[1]}')

    y_train = [109.85, 155.72]
    y_test = [0.24, 0.58]

    best_mse = None
    best_degree = 1

    for d in range(1, 4):  # 1 to 3 degrees
        poly = PolynomialFeatures(degree=d)

        # Increase the number of features or column based on degree to improve the fit.
        # Rows or number of datasets are not increased.
        X_train_p = poly.fit_transform(X_train)
        X_test_p = poly.fit_transform(X_test)

        clf = linear_model.LinearRegression(n_jobs=num_processor)  # our model
        clf.fit(X_train_p, y_train)

        y_pred = clf.predict(X_test_p)
        mse = mean_squared_error(y_test, y_pred)
        print(f'degree: {d}, ytest: {y_test}, ypred: {y_pred}, mse: {mse}')

        if best_mse is None or mse < best_mse:
            best_mse = mse
            best_degree = d

    print(f'best mse: {best_mse:0.2f}, best degree: {best_degree}')


# start
minimize_mse()

Output

X_train rows: 2, X_train columns: 2
degree: 1, ytest: [0.24, 0.58], ypred: [108.03336634 100.31267327], mse: 10783.007971615527
degree: 2, ytest: [0.24, 0.58], ypred: [109.1846578  102.28740262], mse: 11106.667105279808
degree: 3, ytest: [0.24, 0.58], ypred: [109.68924338 103.82600437], mse: 11319.43714729889
best mse: 10783.01, best degree: 1
3
  • Thank you for your answer! but I wanted is to make a polynomial regression to get some function f, and then to find f's maxima on some domain, for example on the d-dimensional unit sphere, could you show me how to do this? (this is, actually, where I had the most difficulties)
    – Dudi Frid
    Commented Nov 26, 2021 at 14:50
  • But you already wrote input: n sample points of a function f that means the f() is an input. Sorry, I think you need to describe the problem properly.
    – ferdy
    Commented Nov 26, 2021 at 15:16
  • As you suggest, I edited my question, and simplified it to the main difficulty I face.
    – Dudi Frid
    Commented Nov 26, 2021 at 20:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.