33

I am a newbie in Machine learning and Natural language processing.

I am always confused between what are those three terms?

From my understanding:

class: The various categories our model output. Given a name of person identify whether he/she is male or female?

Lets say I am using Naive Bayes classifier.

What would be my features and parameters?

Also, what are some of the aliases of the above words which are used interchangeably.

Thank you

6 Answers 6

32

Let's use the example of classifying the gender of a person. Your understanding about class is correct! Given an input observation, our Naive Bayes Classifier should output a category. The class is that category.

Features: Features in a Naive Bayes Classifier, or any general ML Classification Algorithm, are the data points we choose to define our input. For the example of a person, we can't possibly input all data points about a person; instead, we pick a few features to define a person (say "Height", "Weight", and "Foot Size"). Specifically, in a Naive Bayes Classifier, the key assumption we make is that these features are independent (they don't affect each other): a person's height doesn't affect weight doesn't affect foot size. This assumption may or not be true, but for a Naive Bayes, we assume that it is true. In the particular case of your example where the input is just the name, features might be frequency of letters, number of vowels, length of name, or suffix/prefixes.

Parameters: Parameters in Naive Bayes are the estimates of the true distribution of whatever we're trying to classify. For example, we could say that roughly 50% of people are male, and the distribution of male height is a Gaussian distribution with mean 5' 7" and standard deviation 3". The parameters would be the 50% estimate, the 5' 7" mean estimate, and the 3" standard deviation estimate.

Aliases: Features are also referred to as attributes. I'm not aware of any common replacements for 'parameters'.

I hope that was helpful!

1
  • Thank you for amazing answer. Had slight idea of the answer, but still confirmed. Thank you.
    – Jay Patel
    Mar 5, 2016 at 21:50
20

@txizzle explained the case of Naive Bayes well. In a more general sense:

Class: The output category of your data. You can call these categories as well. The labels on your data will point to one of the classes (if it's a classification problem, of course.)

Features: The characteristics that define your problem. These are also called attributes.

Parameters: The variables your algorithm is trying to tune to build an accurate model.

As an example, let us say you are trying to decide to whether admit a student to gard school or not based on various factors like his/her undergrad GPA, test scores, scores on recommendations, projects etc. In this case, the factors mentioned above are your features/attributes, whether the student is given an admit or not become your 2 classes, and the numbers which decide how these features combine together to get your output become your parameters. What the parameters actually represent depends on your algorithm. For a Neural Net, it's the weights on the synaptic links. Similarly, for a regression problem, the parameters are the coefficients of your features when they are combined.

2
  • so feature engineering is first step for all the ML methods ?(supervised and unsupervised) May 1, 2019 at 18:52
  • So the term "class" cannot be applied to any categorical variable in the Dataframe but only to the target variable?
    – user140259
    May 5, 2021 at 22:48
11

take a simple linear classification problem-

y={0 if 5x-3>=0 else 1}

here y is class, x is feature, 5,3 are parameters.

2
  • 2
    Great and underrated answer! Answers should also be rated by their shortness. This one conveys the most useful information in the least amount of chars.
    – Moritz W
    Dec 5, 2020 at 8:56
  • I agree with @MoritzW. This is a very underrated answer and very insightful, visual and concise. Jan 26, 2021 at 15:48
0

I just wanted to add a definition that distinguishes between attributes and features, as these are often used interchangeably, and it may not be correct to do so. I'm quoting 'Hands-On Machine Learning with SciKit-Learn and TensorFlow'.

In Machine Learning an attribute is a data type (e.g., “Mileage”), while a feature has several meanings depending on the context, but generally means an attribute plus its value (e.g., “Mileage = 15,000”). Many people use the words attribute and feature interchangeably, though.

0

I like the definition in “Hands-on Machine Learning with Scikit and Tensorflow” (by Aurelian Geron) where ATTRIBUTE = DATA TYPE (e.g., Mileage) FEATURE = DATA TYPE + VALUE (e.g., Mileage = 50000)

Regarding FEATURE versus PARAMETER, based on the definition in Geron’s book I used to interpret FEATURE as the variable and the PARAMETER as the weight or coefficient, such as in the model below Y = a + b*X

X is the FEATURE a, b are the PARAMETERS

However, in some publications I have seen the following interpretation: X is the PARAMETER a, b are the WEIGHTS

So, lately, I’ve begun to use the following definitions:

FEATURE = variables of the RAW DATA (e.g., all columns in the spreadsheet)

PARAMETER = variables used in the MODEL (ie after selecting the features that will be in the model)

WEIGHT = coefficients of the parameters of the MODEL

Thoughts ?

0

Let's see if this works :)

Imagine you have an excel spreadsheet which has data about a specific product and the presence of 7 atomic elements in them.

[product] [calcium] [magnesium] [zinc] [iron] [potassium] [nitrogen] [carbon]

Features - are each column except the product because all the other columns are independent, coexisting, has measurable impact on the target i.e. the product. You can even choose to combine some of them to be called Essential Elements i.e. dimension reduction to make it more appropriate for analysis. The term "Dimension Reduction" is strictly for explanation here, not be confused by the PCA technique in unsupervised learning. Features are relevant for supervised learning technique.

Now, imagine a cool machine that has the capability of looking at the data above and inferring what the product is.

parameters are like levers and stopcocks to the specific to that machine which you can juggle with, and make sure that if the machine says "It's soap scum" it really/truly is. If you you think about yourself doing the dart board practice, what are the things you'd do to yourself to get closer to the bullseye (balance bias/variance)?

Hyperparameters are like parameters, BUT external to this machine we're talking about. What if the machine parts/mechanical elements are made of a specific compound e.g. carbon fibre or magnesium poly-alloy? How would this change what the machine can/can't do better?

I suppose it's an oversimplification of what things are, but hopefully acceptable?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.