I have a project that uses Google Dataflow. I have been successfully using the following command (and commands like it) for months to deploy templates.
.venv/bin/python -m dataflow.registry_files.delimited_file --runner=DataflowRunner --region=us-central1 --project=myproject --staging_location=gs://mybucket-staging/staging/gr265 --template_location=gs://mybucket-code/templates/gr265 --temp_location=gs://mybucket-staging/temp/gr265 --no_use_public_ips --save_main_session --setup_file=dataflow/setup.py --projectId=myproject --datasetId=padl_staging --tableId=gr265 --configFile=gs://mybucket-code/registry/registry_files.yaml --fileType=gr265
This command continues to work on windows 10 and Debian machines in my team.
Since I upgraded to catalina (10.15.1, with python 3.7.5 and apache-beam==2.16.0 ) I get the following error:
[libprotobuf ERROR google/protobuf/descriptor_database.cc:58] File already exists in database:
[libprotobuf FATAL google/protobuf/descriptor.cc:1370] CHECK failed: GeneratedDatabase()->Add(encoded_file_descriptor, size):
libc++abi.dylib: terminating with uncaught exception of type google::protobuf::FatalException: CHECK failed: GeneratedDatabase()->Add(encoded_file_descriptor, size):
Abort trap: 6
I have done all of the following, with many reboots:
Run
xcode-select --install
Run
brew update-reset
,brew update
,brew upgrade
, andbrew reinstall python
all to no effect (except afterbrew update-reset
,brew doctor
works again)Run
brew uninstall protobuf
andbrew install protobuf
Run
pip3 uninstall protobuf
outside of the virtual environmentsDeleted and re-created my virtual environments from their requirements files.
Several bits of voodoo involving
/usr/local/include
that I located elsewhere on stack overflow that didn't help.
I wondered if this was just my machine, and unfortunately was able to reproduce it on the other macOS Catalina laptop in my team, but not the laptop still running macOS Mojave.
DirectRunner
or theDataflowRunner
find_package
, when mine libray via theadd_subdirectory
, which triggered the protobuf library rebuild. You must ensure you are linking versus the same library version. Additionally, you should link directly to the cmake target:protobuf::libprotobuf
instead of a path to the library file. That will add all required dependent paths and definitions likePROTOBUF_USE_DLLS
(related to another issue withgoogle::protobuf::internal::ArenaStringPtr::GetNoArena already defined
).