15

I am using Apache Avro.

My schema has map type:

{"name": "MyData", 
  "type" :  {"type": "map", 
              "values":{
                   "type": "record",
                   "name": "Person",
                   "fields":[
                      {"name": "name", "type": "string"},
                      {"name": "age", "type": "int"},

                ]
                }
               }
}

After compile the schema, the genated Java class use CharSequence as the key for the Map MyData.

It is very inconvenient to use CharSequence in Map as key, is there a way to generate String type key for Map in Apache Avro?

P.S.

Problem is that, for example dataMap.containsKey("SOME_KEY") will returns false even though there is such key there, just because it is CharSequence. Besides, put an map entry with a existing key doesn't relpace the old one. That's why I say it is inconvenient to use CharSequence as key.

3
  • Given your comment on the one answer: did you find out exactly what's breaking for you? As in, does the Map you get actually use non-String keys?
    – millimoose
    Nov 1, 2013 at 14:40
  • Millimoose, the generated Map use CharSequence for key.
    – Mellon
    Nov 1, 2013 at 14:41
  • The generated map uses Utf8 by default, and you can choose to make it String. CharSequence is just an interface.
    – Alex A.
    Nov 8, 2013 at 21:38

6 Answers 6

12

This JIRA discussion is relevant. The main point of CharSequence still being used is backwards-compatability.

And like Charles Forsythe pointed out, there has been added a workaround for when String is necessary, by setting the string property in the schema.

 { "type": "string", "avro.java.string": "String" }

The default type here is their own Utf8 class. In addition to manual specification and the pom.xml setting, there is even an avro-tools compile option for it, the -string option:

java -jar avro-tools.1.7.5.jar compile -string schema /path/to/schema .
9

Apparently, there is a workaround for this problem in Avro 1.6. You specify the string type in your project's POM file:

  <stringType>String</stringType>

This is mentioned in this issue is AVRO-803 ... though the plugin's web documentation doesn't reflect this.

3
  • According to that issue, to add insult to injury the CharSequence subclass used is actually an Avro-specific class (Utf8) which they could've easily made hashable/equatable to String to reduce some of the pain.
    – millimoose
    Nov 1, 2013 at 20:52
  • That's a good point. This might be a worthwhile change for Avro if it hasn't already been added. On the other hand, CharSequence doesn't guarantee the equality behavior, so perhaps it is better to use toString() in those cases anyway.
    – Alex A.
    Nov 8, 2013 at 21:44
  • 1
    @millimoose No, that is not possible. You can't make it work with String, since the first thing string does is check if the other object passed in equals is String. No non-string implementation of CharSequence can have hashCode/equals interop properly with String. Jun 21, 2018 at 16:53
6

Apparently, by default, Avro uses CharSequence. I found a way to configure it to convert to String

From Avro 1.6.0 onward, there is an option to have Avro always perform the conversion to String. There are a couple of ways to achieve this. The first is to set the avro.java.string property in the schema to String:

         { "type": "string", "avro.java.string": "String" }

I have not tested this.

2
  • This is the correct way to do this, and this property is present to deal with this exact problem.
    – Alex A.
    Nov 8, 2013 at 21:37
  • 2
    is this a per field configuration? how do you do this for the map key? also, the link is dead.
    – andresp
    May 13, 2015 at 11:33
4

Regardless of whether it's possible to force Avro to use a String, using CharSequence directly is a bad implementation because CharSequence isn't Comparable<CharSequence> and doesn't even specify equality of two identical sequences. I suggest filing this as a bug against Avro.

2
  • In fact even in a fairly trivial case (String and StringBuilder) the hash codes don't match: ideone.com/cX76YN. (They do for StringBuffer and StringBuilder but that's probably a consequence of the latter being mostly a copy of the former with synchronisation removed.)
    – millimoose
    Nov 1, 2013 at 14:47
  • @millimoose StringBuffer and StringBuilder both extend the package-protected AbstractStringBuilder so no surprise that they act alike. Agree that this is an implementation problem. Nov 1, 2013 at 14:50
4

I think explicitly convert String to Utf8 will work. "some_key" -> new Utf8("some_key") and use this as your key for the map.

-1

a quick solution(the value type could be other Objects, now I am):

Map<String, String> convertToStringMap(Map<CharSequence, CharSequence> map){
    if (null == map){
        return null;
    }
    HashMap<String, String> result = new  HashMap<String, String>();
    for(CharSequence key: map.keySet()){
        CharSequence k_value = map.get(key);
        String s_key = key.toString();
        String s_value = k_value.toString();
        result.put(s_key, s_value);
    }
    return result;
}

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.