2

I am trying to use the 128 byte embeddings produced by the pre-trained base of the VGGish model for transfer learning on audio data. Using python vggish_inference_demo.py --wav_file ... to encode my training data to a tfrecord worked fine, but now I want to use this as an input to another model (e.g. a neural network I create with keras or something else). Using some similar questions and the documentation, I go this far with the first embedding record of one file:

tfrecords_filename = 'example1.tfrecord'
record_iterator = tf.python_io.tf_record_iterator(path=tfrecords_filename)
string_record = next(record_iterator)
example = tf.train.SequenceExample()
example.ParseFromString(string_record)
print(example.feature_lists.feature_list['audio_embedding'].feature[0].bytes_list.value)

This produces

[b'\x99\x07\xaa>\xd2_R_\x9f\xbbqN\x99\xa18V\xad\x7f\x93\xf0)\xdd4\x80~\xb0\xa4d\x8e\x85\xb6\x88\xa3?U\xa6Q[\x9b\x038\xff\x00EE>OJ\xa5\xb8\x828)\x97^\x8a\xaa\x12h\xff\xff\xc39\xce\x9b\x13\x80\x00j\xcaZ\xac\xff\xff\x0f\xac\x1c\x90&\xd2.b\xe2{\xc1\x15\xe9\xba\xed\xd4\xa9\xff\xdc\xb5\x99]!\x04\xca\xff\xa6;b\xe0\x19\xbfW\xebP!\xff\xc5\xff\x82\xff\x1a\xbe\xec-h\xff\x8d\xff\r\x96\x00\x00\xff']

I am not even sure what this b'...' is (there's more than 64 and fewer than 128 xs - so not sure how this lines up with anything).

Maybe I am missing some basic Python knowledge here, but how do I turn this into a numeric array of numbers that I can use as an input to some other model?

1 Answer 1

5

It turns out that these are bytes that can be converted to hex, which can be converted to an array of integers between 0 to 255.

import tensorflow as tf
import numpy as np

tfrecords_filename = 'example1.tfrecord'
record_iterator = tf.python_io.tf_record_iterator(path=tfrecords_filename)
string_record = next(record_iterator)
example = tf.train.SequenceExample()
example.ParseFromString(string_record)
hexembed = example.feature_lists.feature_list['audio_embedding'].feature[0].bytes_list.value[0].hex()
arrayembed = [int(hexembed[i:i+2],16) for i in range(0,len(hexembed),2)]
print(arrayembed)

This produces output in the format that I desired:

[153, 7, 170, 62, 210, 95, 82, 95, 159, 187, 113, 78, 153, 161, 56, 86, 173, 127, 147, 240, 41, 221, 52, 128, 126, 176, 164, 100, 142, 133, 182, 136, 163, 63, 85, 166, 81, 91, 155, 3, 56, 255, 0, 69, 69, 62, 79, 74, 165, 184, 130, 56, 41, 151, 94, 138, 170, 18, 104, 255, 255, 195, 57, 206, 155, 19, 128, 0, 106, 202, 90, 172, 255, 255, 15, 172, 28, 144, 38, 210, 46, 98, 226, 123, 193, 21, 233, 186, 237, 212, 169, 255, 220, 181, 153, 93, 33, 4, 202, 255, 166, 59, 98, 224, 25, 191, 87, 235, 80, 33, 255, 197, 255, 130, 255, 26, 190, 236, 45, 104, 255, 141, 255, 13, 150, 0, 0, 255]

2
  • hi, I am trying to use VGGish / Audioset to build a model that can detect certain sounds on my own audio files. Would you please point to clear resources on where to start ? Thanks ! Oct 12, 2022 at 22:13
  • 1
    I'd look at the code in their GitHub repo, if you really want to use VGGish. Otherwise, have a look at more modern alternatives. Recent Kaggle competitions with audio data would be a good starting point (people will usually have extensive discussions and publish notebooks with code, some of which can be extremely nicely written tutorials, while others will be code dumps without explanations).
    – Björn
    Oct 15, 2022 at 6:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.