0

I'm trying to create an Autoencoder neural network for finding outliers using Keras TensorFlow, my data is a list of texts with one word per line, it is the following: https://pastebin.com/hEvm6qWg it has 139 lines.

When I fit my model with my data, I get the error:

ValueError: Error when checking input: expected input_1 to have shape (139,) but got array with shape (140,)

But I can't tell why it recognizes it as 140 shape array, my entire code is as follows:

from keras import Input, Model
from keras.layers import Dense
from keras.preprocessing.text import Tokenizer

with open('drawables.txt', 'r') as arquivo:
    dados = arquivo.read().splitlines()

tokenizer = Tokenizer(filters='')
tokenizer.fit_on_texts(dados)

x_dados = tokenizer.texts_to_matrix(dados, mode="freq")

tamanho = len(tokenizer.word_index)

x = Input(shape=(tamanho,))

# Encoder
hidden_1 = Dense(tamanho, activation='relu')(x)
h = Dense(tamanho, activation='relu')(hidden_1)

# Decoder
hidden_2 = Dense(tamanho, activation='relu')(h)
r = Dense(tamanho, activation='sigmoid')(hidden_2)

autoencoder = Model(input=x, output=r)

autoencoder.compile(optimizer='adam', loss='mse')

autoencoder.fit(x_dados, epochs=5, shuffle=False)

I am utterly lost, I can't even tell if my approach to an autoencoder network is the correct one, what am I doing wrong?

1 Answer 1

1

word_index in Tokenizer start from 1 not from zero

Example:

tokenizer = Tokenizer(filters='')
tokenizer.fit_on_texts(["this a cat", "this is a dog"])
print (tokenizer.word_index)

Output:

{'this': 1, 'a': 2, 'cat': 3, 'is': 4, 'dog': 5}

Index is starting from 1 not from zero. So when we create term frequency matrix using these indices

x_dados = tokenizer.texts_to_matrix(["this a cat", "this is a dog"], mode="freq")

The shape of x_dados will be 2x6 because numpy arrays are indexed from 0.

so no:of columns in x_dados = 1+len(tokenizer.word_index)

So to fix your code change

tamanho = len(tokenizer.word_index)

to

tamanho = len(tokenizer.word_index) + 1

Working sample:

dados = ["this is a  cat", "that is a dog and a cat"]*100
tokenizer = Tokenizer(filters='')
tokenizer.fit_on_texts(dados)

x_dados = tokenizer.texts_to_matrix(dados, mode="freq")
tamanho = len(tokenizer.word_index)+1
x = Input(shape=(tamanho,))

# Encoder
hidden_1 = Dense(tamanho, activation='relu')(x)
h = Dense(tamanho, activation='relu')(hidden_1)

# Decoder
hidden_2 = Dense(tamanho, activation='relu')(h)
r = Dense(tamanho, activation='sigmoid')(hidden_2)

autoencoder = Model(input=x, output=r)
print (autoencoder.summary())

autoencoder.compile(optimizer='adam', loss='mse')
autoencoder.fit(x_dados, x_dados, epochs=5, shuffle=False)
4
  • Thanks! I've made the code changes you mentioned, now I'm getting a IndexError: list index out of range error, but I'm lost on what the cause can be.
    – gtbono
    Commented Mar 29, 2019 at 20:20
  • The line of the fit method: autoencoder.fit(x_dados, epochs=5, shuffle=False)
    – gtbono
    Commented Mar 29, 2019 at 20:23
  • 1
    Should not it be autoencoder.fit(x_dados,x_dados, epochs=5, shuffle=False) you are missing your y's (which is same as x in autoencoder)
    – mujjiga
    Commented Mar 29, 2019 at 20:43
  • How could I miss that?? it somehow worked! now I just need to plot it and see if the data made sense, but it ran! Thanks! 😁
    – gtbono
    Commented Mar 29, 2019 at 21:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.