3

I can't find the way that using MySQL database in TensorFlow.

I made the table and sensor data for reference.

Here is the question.

  1. What should I use to read MySQL database in TensorFlow?

  2. I find the way that making CSV file in MySQL, and read it in TensorFlow. But it is not real-time data. I want to use the data in real-time.

Please help me. Thank you.

2 Answers 2

3

Late to the party. But i did it like that:

  1. Read data from mysql in chunks (limit+offset) with pandas df.read_sql() in a separate thread.
  2. yield the data with a generator
  3. create a tf.Dataset with .from_tensor_slice() from the pandas dataframe.
  4. Use model.fit() on the dataset.
  5. Get the next chunk of data from the generator and train on that.

So, while the model is training on the data chunk, the next chunk is being loaded in the background. Ideally, the GPU will never sit idle. This can be adjusted by choosing a good chunk size and defining how many epochs the model should be trained on each chunk.

Relevant links: Make python generator run in background https://stackoverflow.com/a/29528804/5292996 https://www.tensorflow.org/tutorials/load_data/pandas_dataframe

1
  • That sounds great! If you ever find the time to paste a code example. That would be awesome. Jul 22, 2020 at 20:41
0

You'd probably need to just use a standard python interface for mysql and feed it into tensorflow. Here are some examples:

1
  • But this is inefficient is not it? What if you have a big data?
    – ARAT
    Nov 24, 2018 at 4:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.