0

I'm working on a program that uses the twitter4j API to retrieve tweets and use them to analyze clusters, but I am having the strangest problem on about 1 out of ever 2000 tweets that I collect.

I am storing each original tweet with the username at the front in a set of Strings and I also have a set of Strings in which I'm storing "cleaned" tweets, meaning that I have set every letter to lowercase, removed any character which is not alphanumeric, and removed all usernames.

After doing some cluster analysis with the clean tweet they are in a JSON formatted string and then I swap the clean tweets with the original tweets, the following line of code shows how they are replaced once the correct index is found. (jsonResults is a single String that has all the clean tweets in a JSON format, tweetIndex is the index where the cleaned tweet started, espapedOrigTweet is a String with the username and original tweet that has been formatted to work in JSON):

jsonResults = jsonResults.substring(0, tweetIndex) + escapedOrigTweet + jsonResults.substring(tweetIndex + cleanedTweet.length());

About 99.95% of the time this works like a charm and the end result is one long string with all original tweets and usernames in JSON format, but 0.05% of the time it fails horribly.

When it works correctly, the final JSON string which contains all the original tweets and starts like this:

{"results":{"numOfClusters": ...

But, when it fails, the final JSON string starts somewhere in the middle of the tweet with the problem like this:

#MTVStars The Vamps https://t.co/3PGQWnxUMv","0258: ...

Here's some of the output with debugging that I've tried to do. I have a line of code that, after receiving all the tweets, prints the original tweet and user and also what it is like after it is cleaned. Here's the code:

System.out.println(twt + " --- " + cleanTweet(twt));

Here's what output of this line looks like for a normal tweet:

@mangonicci13= Tory donor card https://t.co/rCwZCZBV7k --- tory donor card

And here's the output of that line for a tweet that is a problem:

#MTVStars The Vamps https://t.co/3PGQWnxUMv --- 1 1 vote where are the vamily at mtvstars the vamps

If anyone can help me to be able to identify whatever there is about these broken tweets so that they can be changed in order to not cause Strings to behave erratically, I would very much appreciate it. Thanks for reading!

2
  • 1
    There are far too many things which could be wrong in an application as complex as you describe. I doubt that we will be able to find the problem just from your description. A guess into the blue, though: Are you perhaps using any form of multithreading?
    – Philipp
    Dec 6, 2015 at 17:52
  • The library I use to collect the tweets is multithreaded, 1 thread for every term you search, but I have only been using it to search 1 term. And, I think that I've properly narrowed the problem to coming from the tweet itself, perhaps special character or something else unique in these rare tweets are causing the Java strings to behave strangely.
    – Aaron T
    Dec 6, 2015 at 17:57

1 Answer 1

-1

I was able to find the problem that was causing these Strings to behave so erratically. To do so, I had each bad string print the decimal value of each character in unicode.

Doing this, I quickly found that each problem string had the issue every time there was a character with value 13, a "carriage return" character. I wrote a method to remove any characters with int value 13 from all the tweet strings, and have not run into a problem since.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.