0

I have a large data set that is parsed by a Ruby script. The script creates a CSV, then uploads it to a Redshift database. The majority of the lines in the logs are successfully uploaded, but many fail to upload due to "extra columns found". I have isolated a case where this happens.

The log data looks like this:

2014-09-22 13:02:16-0400,238 {"Items":[{"PubEndDate":"2002/04/09","ItmId":"1280429264","SourceType":"Government & Official Publications","ReasonCode":"","MyResearchUser":"","ProjectCode":"","PublicationCode":"","PubStartDate":"2002/04/09","ItmFrmt":"KWIC","Subrole":"KWIC","PaymentType":"PrePaid","UsageInfo":"P-1008361-158946-STAFF-null-2195091","Role":"KWIC","RetailPrice":1.19,"EffectivePrice":0,"ParentItemId":"396489"},{"PubEndDate":"2012/04/05","ItmId":"1139461559","SourceType":"Government & Official Publications","ReasonCode":"","MyResearchUser":"","ProjectCode":"","PublicationCode":"","PubStartDate":"2012/04/05","ItmFrmt":"KWIC","Subrole":"KWIC","PaymentType":"PrePaid","UsageInfo":"P-1008365-158946-STAFF-null-2195099","Role":"KWIC","RetailPrice":0.75,"EffectivePrice":0,"ParentItemId":"396490"}]}

I then create a CSV via a Ruby script that looks like this (excuse the large code block, it's a long script):

require 'json'

# add methods to unnest ruby hashes for converting nested json into an array with reasonable values
class Hash
  def unnest
    new_hash = {}
    each do |key,val|
      if val.is_a?(Hash)
        new_hash.merge!(val.prefix_keys("#{key}-"))
      else
        new_hash[key] = val
      end
    end
    new_hash
  end

  def prefix_keys(prefix)
    Hash[map{|key,val| [prefix + key, val]}].unnest
  end
end

def parse(usage)

    usage = usage.gsub(/|/,'').gsub(/\n/, '')
    #Array of all possible keys, make sure all fields in db are filled regardless of how many params are passed into the usage log 
    keys = ["UserAgent","IP","AppId","SessId","JSessionId","LangCd","UsageType","BreadCrumb","AuthType","UsageGroupId","SearchType","ResponseTime","EventType","LandedFirstPage","ReferringUrl","PubEndDate","ItmId","PubStartDate","ItmFrmt","OpenUrlRefId","OpenAccess","LinkSource","SourceType","Subrole","PremId","PaymentType","ObjectType","OrigSite","UsageInfo","Role","DeliveryMethod","ParentItemId","SearchAllProductsFlag","MarketSegment","SearchCount","SearchEngine","QryString","SubjectKey","SearchId","SearchHits","UserInfo-IP","UserInfo-AppId","UserInfo-SessId","UserInfo-UsageGroupId","SearchProductInfo","TurnAwayFlag","LinkOutTarget","LinkOutType","TranslationTime","TextSize","TextType","SourceLang","DestinationLang","ReasonCode","RetailPrice","EffectivePrice","MyResearchUser","ProjectCode","DocID","ListingType","MasterID","TerminatedSessionID","PublicationId","PublicationTitle","ItemTitle","AccessAgreementStatus"]

    items_keys = ["ReferringUrl","PubEndDate","ItmId","SourceType","PubStartDate","PublicationCode","ItmFrmt","PaymentType","ObjectType","OrigSite","UsageInfo","OpenUrlRefId","TurnAwayFlag","OpenAccess","ParentItemId","SearchId","SearchProductInfo","EventName","HistoryId","AlertId","ReasonCode","Origin","MyResearchUser","ProjectCode","Subrole","NumberOfCopies","Role","RetailPrice","EffectivePrice","Multiplier","PublicationId","PublicationTitle","ItemTitle",]
    # extract date and time from json, then parse json to ruby hash

    date = usage.scan(/\d{4}-\d\d-\d\d/).first
    time = usage.scan(/\d\d:\d\d:\d\d/).first
    json = usage.scan(/\{.*\}/).first
    parsed = JSON.parse(json).unnest

    # return array of values, substituting 'Not Listed' for all missing attributes
    result = []
    items_result = []
    result = (0...keys.length).map{ |i| parsed[keys[i]] || 'NA'}
    result.unshift date
    result.unshift time
    result.push "save_space"#usage
    items = JSON.parse(json)

    temp_result = result

    CSV.open("testing.csv", "a+", {:col_sep => "|"}) do |csv|
      begin
       items["Items"].each do |item|
           item_result = (0...items_keys.length).map{ |i| item[items_keys[i]] || "NA" }
           temp_result = (temp_result << item_result).flatten!
     csv << temp_result
     temp_result = result.flatten
       item_result = []
        end
      rescue
           item_result = (0...items_keys.length).map{ |i| "NA" }
     temp_result = (temp_result << item_result).flatten!
     csv << temp_result
     temp_result = result.flatten
       item_result = []
      end
    end
    nil
end

File.readlines("file.log").each do |line|
    parse(line)
end
`ruby upload_csv_to_redshift.rb usage_logs_testing`

This script creates a CSV that looks like this:

13:02:16|2014-09-22|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|save_space|NA|2002/04/09|1280429264|Government & Official Publications|2002/04/09|""|KWIC|PrePaid|NA|NA|P-1008361-158946-STAFF-null-2195091|NA|NA|NA|396489|NA|NA|NA|NA|NA|""|NA|""|""|KWIC|NA|KWIC|1.19|0|NA|NA|NA|NA
13:02:16|2014-09-22|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|NA|save_space|NA|2002/04/09|1280429264|Government & Official Publications|2002/04/09|""|KWIC|PrePaid|NA|NA|P-1008361-158946-STAFF-null-2195091|NA|NA|NA|396489|NA|NA|NA|NA|NA|""|NA|""|""|KWIC|NA|KWIC|1.19|0|NA|NA|NA|NA|NA|2012/04/05|1139461559|Government & Official Publications|2012/04/05|""|KWIC|PrePaid|NA|NA|P-1008365-158946-STAFF-null-2195099|NA|NA|NA|396490|NA|NA|NA|NA|NA|""|NA|""|""|KWIC|NA|KWIC|0.75|0|NA|NA|NA|NA

Which is uploaded to the Redshift database structured like this:

CREATE TABLE usage_logs_test
(
log_id bigint IDENTITY (0,1),
log_time varchar(200),
log_date varchar(200),
UserAgent varchar(max),
IP varchar(max),
AppId varchar(max),
SessId varchar(max),
JSessionId varchar(max),
LangCd varchar(max),
UsageType varchar(max),
BreadCrumb varchar(max),
AuthType varchar(max),
UsageGroupId varchar(max),
SearchType varchar(max),
ResponseTime varchar(max),
EventType varchar(max),
LandedFirstPage varchar(max),
ReferringUrl varchar(max),
PubEndDate varchar(max),
ItmId varchar(max),
PubStartDate varchar(max),
ItmFrmt varchar(max),
OpenUrlRefId varchar(max),
OpenAccess varchar(max),
LinkSource varchar(max),
SourceType varchar(max),
Subrole varchar(max),
PremId varchar(max),
PaymentType varchar(max),
ObjectType varchar(max),
OrigSite varchar(max),
UsageInfo varchar(max),
Role varchar(max),
DeliveryMethod varchar(max),
ParentItemId varchar(max),
SearchAllProductsFlag varchar(max),
MarketSegment varchar(max),
SearchCount varchar(max),
SearchEngine varchar(max),
QryString varchar(max),
SubjectKey varchar(max),
SearchId varchar(max),
SearchHits varchar(max),
UserInfo_IP varchar(max),
UserInfo_AppId varchar(max),
UserInfo_SessId varchar(max),
UserInfo_UsageGroupId varchar(max),
SearchProductInfo varchar(max),
TurnAwayFlag varchar(max),
LinkOutTarget varchar(max),
LinkOutType varchar(max),
TranslationTime varchar(max),
TextSize varchar(max),
TextType varchar(max),
SourceLang varchar(max),
DestinationLang varchar(max),
ReasonCode varchar(max),
RetailPrice varchar(max),
EffectivePrice varchar(max),
MyResearchUser varchar(max),
ProjectCode varchar(max),
DocID varchar(max),
ListingType varchar(max),
MasterID varchar(max),
TerminatedSessionID varchar(max),
PublicationId varchar(max),
PublicationTitle varchar(max),
ItemTitle varchar(max),
AccessAgreementStatus varchar(max),
full_log varchar(max)


ReferringUrl varchar(max),
PubEndDate varchar(max),
ItmId varchar(max),
SourceType varchar(max),
PubStartDate varchar(max),
PublicationCode varchar(max),
ItmFrmt varchar(max),
PaymentType varchar(max),
ObjectType varchar(max),
OrigSite varchar(max),
UsageInfo varchar(max),
OpenUrlRefId varchar(max),
TurnAwayFlag varchar(max),
OpenAccess varchar(max),
ParentItemId varchar(max),
SearchId varchar(max),
SearchProductInfo varchar(max),
EventName varchar(max),
HistoryId varchar(max),
AlertId varchar(max),
ReasonCode varchar(max),
Origin varchar(max),
MyResearchUser varchar(max),
ProjectCode varchar(max),
Subrole varchar(max),
NumberOfCopies varchar(max),
Role varchar(max),
RetailPrice varchar(max),
EffectivePrice varchar(max),
Multiplier varchar(max),
PublicationId varchar(max),
PublicationTitle varchar(max),
ItemTitle varchar(max),
OrigId varchar(200)
);

The problem seems to be that a lot of the data is being duplicated, as though the array I called temp_result is not clearing back to the value of result at the end of the items["Items"].each block.

I realize that this is a very large problem but I've gone through great effort to simplify and generalize it as much as possible while maintaining a working code example.

1 Answer 1

1

You need to remember that arrays are passed by reference in Ruby. Try this:

a = ["YO"]
b = a
b << "HEY"
puts a.inspect
# => ["YO", "HEY"]

Now consider this line in your script:

temp_result = result

and later on

(temp_result << item_result).flatten!

temp_result is not clearing back to the original value of result because they both refer to the same array in memory. The original value of result is gone, overwritten by use of the append in place method <<.

The quickest fix is:

temp_result = result.clone

Also, unless you specifically know the precise reason why you're doing it, you don't want to use bang methods like flatten! in assignment, you want flatten.

3
  • 1
    A better way to show that a variable containing an array and assigning that to another variable both point to the same array in memory, is to use object_id. a = []; b = a. At that point a.object_id # => 70098065043200 and b.object_id # => 70098065043200. (Or some other ID but they'll both be the same.) Oct 3, 2014 at 17:22
  • 2
    I don't think that's better at all as it relies on implicit knowledge. It's perfectly possible to get quite far in to a ruby career without knowing what #object_id represents. A demonstration trumps description if your goal is to maximise the number of readers who may find it useful.
    – SLD
    Oct 3, 2014 at 17:35
  • 1
    This community is a big part of the reason that I love programming. Thanks so much you guys.
    – johncorser
    Oct 3, 2014 at 20:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.