3

I'm using protocol Buffer(proto3) from my latest project. But I encounter below problem. An encoded message generated from python and I'm trying to decode it from Scala (I'm using SPARK). You can see the actual message from "value" and its length 227 as described in console.

The error field just typical string type field. I also have tried to decode string field as "UTF-8" but it is not worked. It looks encode/decode problem. Are there some other guys who encounter similar problems?

Console output

...

value: 
U
$a5a9c2bb-efd2-4ea3-ae0e-7479925a7807"kafka_producer.py*2016-12-18 10:37:25.075614*
data{"url": "some url 0", "title": "some title", "_id": "5815a37c43cfd44120e50538", "content": "some content", "tags": ["football"]}
value length: 227
Exception in thread "streaming-job-executor-0" java.lang.NoSuchMethodError: com.google.protobuf.CodedInputStream.readStringRequireUtf8()Ljava/lang/String;
at org.libero.messages.Messages$Event.<init>(Messages.java:160)
at org.libero.messages.Messages$Event.<init>(Messages.java:117)
at org.libero.messages.Messages$Event$1.parsePartialFrom(Messages.java:1564)
at org.libero.messages.Messages$Event$1.parsePartialFrom(Messages.java:1559)
at com.google.protobuf.CodedInputStream.readMessage(CodedInputStream.java:309)
at org.libero.messages.Messages$EventDataProduced.<init>(Messages.java:1742)
at org.libero.messages.Messages$EventDataProduced.<init>(Messages.java:1697)
at org.libero.messages.Messages$EventDataProduced$1.parsePartialFrom(Me

...

Code fragments

 ....

 nonEmptyMsgs.foreachRDD { msgsRDD =>
      println("Trying to print each RDD: " + msgsRDD)
      println("Count: " + msgsRDD.count())
      val elems = msgsRDD.collect()
      println("Elems: " + elems)
      for (v <- elems) {
        println("key: " + v._1)
        println("value: " + v._2)
        println("value length: " + v._2.length())
        val bytes = new String(v._2).getBytes()
        val event = EventDataProduced.parseFrom(ByteString.copyFrom(bytes))
        println("event: " + event)
      }
    }

...
3
  • 1
    Is this the complete code? please check. Dec 18, 2016 at 3:46
  • Are you sure you're using the correct version of the .jar file? Dec 18, 2016 at 7:53
  • My version of protobuf-java jar is 3.1.0. Dec 18, 2016 at 8:03

2 Answers 2

6

How can I use proto3 with Hadoop/Spark?

as above link, i fixed it.

Just need to relocate the protobuffers and the VerifyError goes away:

      <relocations>
        <relocation>
          <pattern>com.google.protobuf</pattern>
          <shadedPattern>shaded.com.google.protobuf</shadedPattern>
        </relocation>
      </relocations>
1
  • 1
    how to do this in Gradle. I'm getting the same exception but so far no luck with gradle. Mar 27, 2017 at 9:30
1

For those folks how use gradle and faced with the same problem like @Nowa and myself, here's the solution: Add below lines into your build.gradle

apply plugin: 'com.github.johnrengelman.shadow'

shadowJar {
    baseName = "${jarName}"
    zip64 true
    // This is to avoid the version conflicts in between 
    relocate ('com.google.protobuf', 'otherplace.com.google.protobuf')
}

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.