I'm working on a program that uses the twitter4j API to retrieve tweets and use them to analyze clusters, but I am having the strangest problem on about 1 out of ever 2000 tweets that I collect.
I am storing each original tweet with the username at the front in a set of Strings and I also have a set of Strings in which I'm storing "cleaned" tweets, meaning that I have set every letter to lowercase, removed any character which is not alphanumeric, and removed all usernames.
After doing some cluster analysis with the clean tweet they are in a JSON formatted string and then I swap the clean tweets with the original tweets, the following line of code shows how they are replaced once the correct index is found. (jsonResults is a single String that has all the clean tweets in a JSON format, tweetIndex is the index where the cleaned tweet started, espapedOrigTweet is a String with the username and original tweet that has been formatted to work in JSON):
jsonResults = jsonResults.substring(0, tweetIndex) + escapedOrigTweet + jsonResults.substring(tweetIndex + cleanedTweet.length());
About 99.95% of the time this works like a charm and the end result is one long string with all original tweets and usernames in JSON format, but 0.05% of the time it fails horribly.
When it works correctly, the final JSON string which contains all the original tweets and starts like this:
{"results":{"numOfClusters": ...
But, when it fails, the final JSON string starts somewhere in the middle of the tweet with the problem like this:
#MTVStars The Vamps https://t.co/3PGQWnxUMv","0258: ...
Here's some of the output with debugging that I've tried to do. I have a line of code that, after receiving all the tweets, prints the original tweet and user and also what it is like after it is cleaned. Here's the code:
System.out.println(twt + " --- " + cleanTweet(twt));
Here's what output of this line looks like for a normal tweet:
@mangonicci13= Tory donor card https://t.co/rCwZCZBV7k --- tory donor card
And here's the output of that line for a tweet that is a problem:
#MTVStars The Vamps https://t.co/3PGQWnxUMv --- 1 1 vote where are the vamily at mtvstars the vamps
If anyone can help me to be able to identify whatever there is about these broken tweets so that they can be changed in order to not cause Strings to behave erratically, I would very much appreciate it. Thanks for reading!