Skip to main content
Filter by
Sorted by
Tagged with
0 votes
0 answers
17 views

Apache pig throwing Nullpointer exception when filtering using a parameter ERROR 2000: Error processing rule PartitionFilterOptimizer

I am trying to store data in Apache hive using filter operation where MYVARIABLE is a parameter sent. Filtering using the variable sends nullpointer exception .This happens when I try to store data in ...
Luff li's user avatar
  • 133
0 votes
0 answers
12 views

Invalid Project field in Pig FOREACH

I have a txt input consists of bigrams, year, count, volume_count, and page_count. The task is to find out the most common bigrams for each year, and if there is a tie, I need to store all bigrams ...
Alfred's user avatar
  • 1
0 votes
1 answer
30 views

ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1200: <line 16, column 46> mismatched input ',' expecting LEFT_PAREN

grunt> joined_data = JOIN filtered_features BY (store, date), sales BY (store, date); 2024-04-02 13:19:05,110 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1200: <line 16, column 46> ...
Md Arif Khan's user avatar
0 votes
0 answers
7 views

Apache PIG ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1200: <line 20, column 44> mismatched input '.' expecting SEMI_COLON

I am trying to execute pig latin script to find TF-IDF value for book dataset My input file bookdataset.txt contains the following lines Document 1: "The quick brown fox jumps over the lazy dog.&...
Ashok Kumar's user avatar
0 votes
0 answers
17 views

Apache-pig not running in local mode

I have set up pig but i am getting this error **pig -x local Error: Could not find or load main class org.apache.pig.Main Caused by: java.lang.ClassNotFoundException: org.apache.pig.Main** pig ...
Mooaz Sayyed's user avatar
0 votes
0 answers
25 views

Error using Pig to move data from HDFS to HBase

I've been trying to move data stored on hdfs to HBase using Pig. PS: I'm running a pseudo-cluster hadoop. Here are the steps that I'm executing # Part 1 - Load Data into HDFS # Create directory hdfs ...
Gabriel Bueno Guimaraes's user avatar
0 votes
1 answer
20 views

How to count value column in Apache Pig while filter out distinct key

I'm trying to count the total number of ordered prouduct(the quantity) by product_id. The data looks like this. (product_id, quantity) which also is (11, 5) (11, 2) (11, 1) (12, 9) (12, 1) (13, 5) (13,...
Brian Ngamsom's user avatar
0 votes
0 answers
20 views

erorr dump apache pig

I load 1 data from hdfs:demo2.txt enter image description here command:data = LOAD 'hdfs://localhost:9000/demo1/demo2.txt' USING PigStorage(',') AS (name:chararray, age:int, city:chararray); then I ...
Thái Hoàng's user avatar
0 votes
1 answer
26 views

How can i find each professional count and age count of a particular country( for eg US) in pig

this is the example. from this i want to find each professional count and each age group count. filter_data = FILTER customer by loc=='us'; grunt> grp_data1 = GROUP filter_data by age; ...
ABHI's user avatar
  • 1
1 vote
2 answers
122 views

Getting Error while running pig script on google Dataproc Cluster, all the parameter are defined correctly

This is my pig script - fs -cp -f gs://$codepath/db_password.sh file://$dataprochome/db_password.sh; fs -cp -f gs://$codepath/jdbc_daily_load_tables.py file://$dataprochome/jdbc_daily_load_tables.py; ...
Big data Pyspark's user avatar
1 vote
0 answers
85 views

What happens if some hadoop map-reduce Job or some hdfs command is running and in the between kerberos ticket expires?

Lets say we have kerberos secured Hadoop Cluster. ( For example Kdc is configured to provide kerberos tickets with validity of 2 hour and renewal period of 3 hours after this so total 5 hours. ) ...
uds0128's user avatar
  • 47
3 votes
1 answer
356 views

Error in submitting a pig job to Google Dataproc with properties file

I'm new to Dataproc and trying to submit a pig job to google dataproc via gcloud gcloud config set project PROJECT gcloud dataproc jobs submit pig --cluster=cluster-workaround --region=us-...
chethi's user avatar
  • 709
0 votes
1 answer
34 views

What is the best way to add BigDecimals in Hadoop?

I need to add BigDecimals in Hadoop. I'm currently using Apache Pig BigDecimalWritable but Pig seems to be completely outdated. <dependency> <groupId>org.apache.pig</groupId> <...
BuckBazooka's user avatar
0 votes
2 answers
32 views

Splitting string into a new line in apache-pig

I want to split the string in a dataset that is joined by a backslash (/) into a new line. The example datatset is: (David Jones / John Smith) I want the result to be: (David Jones) (John Smith) The ...
wasif ali's user avatar
0 votes
1 answer
84 views

Having trouble Loading a multilevel JSON file into Apache Pig and saving it as csv

I'm super new to apache pig and was trying to load a multilevel json into pig and save it as csv. Json file I have - home/vikaspattathe/dataset/sample.json {"_id":{"$oid":"...
Vikas Palakkat's user avatar
0 votes
2 answers
50 views

Apache Pig Sum calculation

I am trying to calculate the total shares but am receiving an error with the code, what am I missing? grunt> history 1 stockprice = load 'PG/TutorialA/input/stockprice.csv' using PigStorage(',') ...
Wing's user avatar
  • 1
0 votes
0 answers
91 views

Store Pig result into Hive table using Hcat - ClassNotFoundException: org.datanucleus.NucleusContext

I want to load the simple json data into hive table without using Json Serde. So I have tried to Load the json file in Pig and store the result into hive table. A = LOAD '/data/pig_files/test.json' ...
Srini's user avatar
  • 23
0 votes
0 answers
41 views

Pig - System cannot find the file specified

I have error in pig latin while i run the basic command "Dump Students" for the file >Students = LOAD 'C:\\Users\\avtar\\OneDrive\\Desktop\\student.txt' USING PigStorage(',') as (id:int, ...
Avtar Singh's user avatar
0 votes
1 answer
90 views

Is there a pig / hadoop property that can be set for PIG_HEAPSIZE?

spark, hadoop, tez, etc. all have a list of properties that can be manually configured. example: yarn.nodemanager.resource.memory-mb or spark.executor.memory or pig.exec.reducers.bytes.per.reducer, ...
cydacoder's user avatar
0 votes
1 answer
315 views

ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2998: Unhandled internal error. org/python/google/common/collect/Lists

I'm just getting started with Pig and I'm facing lots of issues with running my first program. Any help is much appreciated. I've tried resolving using these: ERROR org.apache.pig.tools.grunt.Grunt - ...
Siva Jagadesh's user avatar
0 votes
1 answer
59 views

Merging multiple files in Pig

I have several files (around 10 files) which I would like to merge together in Pig: Student01.txt Student02.txt ... Student10.txt I am aware that I could merge two datasets together by: data = ...
Kusisi Karem's user avatar
0 votes
1 answer
59 views

Write a pig sequence of statements to calculate the number of ratings records and the average rating score by gender

this is what i have done so far. A = LOAD '.mike/users.txt' USING PigStorage() AS (userID: int, age: int, gender: chararray, occupation: chararray, zip: int); B = LOAD '.mike/data.txt' USING ...
hyena's user avatar
  • 1
0 votes
1 answer
52 views

PIG mapreduce output and HIVE

I have a file called test.txt with the records as below (disregard the dots): "1" "a" "x" "2" "b" "y" "3" "c" "z&...
Juan Carlos Castro Piedra's user avatar
0 votes
1 answer
165 views

Sum of Salary in Apache Pig

Create employee and dept tables for the available files emp1.csv and dept.csv. Colnames: Emp: Empno, name, sal, did, branch, dno Dept: deptno, name, loc Retrieve total salaries to be paid for the ...
Arsalan Ansari's user avatar
0 votes
1 answer
205 views

How to execute pig script and save the result in another file?

I have a "solution.pig" file which contain all load, join and dump queries. I need to run them by typing "solution.pig" in grunt> and save all the result in other file. How can ...
user19219870's user avatar
0 votes
1 answer
110 views

Apache Pig : Counting frequencies of "character"

Assume there is a text file named abalone_data with 3 attributes : name, gender and length Kn,M,0.89 Un,F,0.77 An,M,0.89 Az,I,0.55 Au,M,0.72 with M is male, F is female and I is infant. The question ...
Khang Nguyen's user avatar
0 votes
1 answer
88 views

Error 1070 Pig Could not resolve PigSorage using imports:

I'm trying to read a file with pig and I have the error indicated in the title. data = LOAD '/user/cloudera/pigexample/commands' USING PigSorage('\n') as (command:chararray); DUMPdata; The file ...
Agat0's user avatar
  • 11
2 votes
4 answers
87 views

Use REPLACE in multicondition RegExp in Pig Latin

I have this set of data: dump data; This is a sample output: (this dataset is almost a million rows long). ("0",60,0,1,"Fri") ("1",47,0,1,"Mon") ("1",...
Alexandro Giles's user avatar
0 votes
0 answers
53 views

How to remove the "," comma which i am getting before a column value after joining data from 2 files

I am trying to run a pig script which picks data from 2 separate strings (string1, string2). And then using tokenize to separate them and later Input: String1 ="cow moon"; string2 ="...
Manish's user avatar
  • 1
0 votes
2 answers
214 views

Load Text Files and store it in Dataframe using Pyspark

I am migrating pig script to pyspark and I am new to Pyspark so I am stuck at data loading. My pig script looks like: Bag1 = LOAD '/refined/em/em_results/202112/' USING PigStorage('\u1') AS (...
Neel Sharma's user avatar
0 votes
1 answer
98 views

how to extract single element from a bag in pig?

My pig statement generates the following output: ({(10)},5201) ({(20),(20),(20)},3334) ({(30),(30),(30),(30)},4632) ({(40),(40)},3101) ({(50),(50)},3801) ({(60),(60),(60)},3959) But I want to store ...
Enigma's user avatar
  • 11
0 votes
1 answer
171 views

Why am I not getting the NULL values when using FILTER to remove CSV Headers in PIG?

I have this data below in a .csv file: Needed_values,TEMP,Desc ,022.3,NewYork 3,022.30,India ,027.0,Australia ,027.00,Russia 1,027.1,Austria ,027.10,Norway ,036.2,Hungary ,036.20,Lithunia 2,785.52,...
Adarsh NamdevCHS's user avatar
0 votes
0 answers
115 views

Any way to validate pig script before running in hdfs cluster?

I'm new to pig script. The problem I'm facing is inability to validate how syntactically correct my script is. I have to upload it to hdfs cluster and run there just to realize I missed ';' in the end ...
YMC's user avatar
  • 5,322
0 votes
1 answer
58 views

Convert chararray to float

I am new to pig programming. I have one txt file and comma (,) as a delimiter. In amount columns i.e; amt_IN and amy_OUT are of type chararray with data $830.03 and $1392.54 respectively. I need these ...
Sameer Khalid's user avatar
0 votes
1 answer
407 views

Pig Installation error: ERROR pig.Main: ERROR 2998: Unhandled internal error

ERROR pig.Main: ERROR 2998: Unhandled internal error. com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V WARN pig.Main: There is no log file to write to. ERROR ...
Nagulan S's user avatar
  • 752
-1 votes
2 answers
37 views

Is it possible to use/query data using Pig/Tableau or some-other tool from HDFS which was inserted/loaded using a HIVE Managed table?

Is it possible to use or query data using Pig or Drill or Tableau or some other tool from HDFS which was inserted/loaded using a HIVE Managed table; or is it only applicable with data in HDFS which ...
Purushothaman Srikanth's user avatar
1 vote
2 answers
311 views

HIVE/PIG JOIN Based on SUBSTRING match

I have a requirement where I need to JOIN a tweets table with person names, like filtering the tweets if it contains any person name. I have following data: Tweets Table: (70 million records stored as ...
CM1's user avatar
  • 13
0 votes
1 answer
385 views

How to solve problem of DUMP not working in Apache Pig

I have Apache Pig version 0.17.0 (r1797386) . I executed the following code. I have hadoop 2.9.2 on Ubuntu 18.04. While run pig in mapreduce mode it gives following messages: 21/11/12 09:47:37 INFO ...
Sudip Chatterjee's user avatar
0 votes
1 answer
985 views

LOAD error mismatched input '-' expecting RIGHT_PAREN, column with hyphen

My code: book = LOAD '/user/samanthafox/books.csv' USING PigStorage (',') AS (ISBN:chararray, Book-Title:chararray,Book-Author:chararray,Year-Of-Publication:chararray,Publisher:chararray); dump book ;...
Samantha Fox's user avatar
1 vote
1 answer
235 views

Best practices when running Hadoop MapReduce jobs/Hive scripts/Pig scripts etc

I am interested in understanding how ETL jobs like Hadoop MapReduce jobs/Spark Jobs/Hive scripts/Pig scripts are usually deployed in an on premises production/development environment. Are they always ...
user538578964's user avatar
0 votes
1 answer
64 views

Hive Pig anti join

I am trying to merge two datasets: Events and mortality by patientid. Events have all the patientid including the patientid in mortality, but didn't have the timestamp and label (always equal 1) in ...
user2236600's user avatar
0 votes
1 answer
67 views

APACHE PIG - error Projected field [Units_Sold] does not exist in schema: group:chararray,D2:bag{:tuple(Item_Type:chararray,Units_Sold:int)}

Good afternoon - I have a Sales Dataset and am trying to see which Item has the most units sold. Here is my code: Country:chararray, Item_Type:chararray, Sales_Channel:...
Liam Megonigal's user avatar
0 votes
2 answers
114 views

data is converting to binary format while loading data into monet db using Apache pig

I am using MonetDb-Pig layer to load the csv data into Monet db. Internally it is using Binarybulkload commands to load the data but after loading data into table, the csv file values are not not ...
RamohanReddy's user avatar
0 votes
0 answers
107 views

replace empty fields with string in Pig

I have the following table loaded onto Pig using the CSVloader: list = LOAD '/user/department_list.csv' USING CSVLoader(',') AS (employee_name:chararray, city:chararray, department:chararray, ...
Sim_R's user avatar
  • 3
0 votes
0 answers
65 views

Load json files with different schema for a same element in Pig

I'm trying to load json files, and when I load a specific field, pig seems to recognize it for a string or a bag. The Json element "program" in the file can be: "program":"...
ANTHOLB's user avatar
0 votes
0 answers
282 views

Pig: Invalid field projection - does not exist in schema - using BagConcat

I'm trying to filter by a field in the schema that I've created, but I can't figure out how to write the field name appropriately. Here's the schema description: B: {datafu.pig.bags.bagconcat_data2_39:...
Coral Weigel's user avatar
0 votes
2 answers
217 views

I need to filter 2 conditions from the same column in PIG HADOOP

I am new to pig. What am I trying to achieve is getting 2 values from 1 table and find if a certain id has both of these items. For example Product Table Product Name Tofu Ikura Sushi Order table ...
Joey Kang's user avatar
-1 votes
1 answer
71 views

Pig Latin unable to dump

I am unable to use the dump funtion for the alias 'TMP'. I had no issue when loading and dumping the files. I have no issue using the describe 'function' for both tables as well as the alias 'TMP'. I'...
Dumbo The Elephant's user avatar
0 votes
1 answer
235 views

Apache Pig Group by and Filter if multiple values exist?

I am trying to group multiple rows with the same IDs, and then check for each tuple in the group if it contains both values, for example: (10461 , 55 ) (10435 , 17 ) (10435 , 11 ) (10435 , 72 ) (10437 ...
xinekiv680's user avatar
1 vote
1 answer
55 views

Using bicond to get ID?

I have this snippet of table called joinTbl: PRODUCT_ID PRODUCT_NAME ORDER_ID PRODUCT_ID CUSTOMER_ID SALESPERSON_ID UNIT_PRICE 11 CAKE 10946 11 ...
datanewbie96's user avatar

1
2 3 4 5
104