5,192
questions
0
votes
0
answers
17
views
Apache pig throwing Nullpointer exception when filtering using a parameter ERROR 2000: Error processing rule PartitionFilterOptimizer
I am trying to store data in Apache hive using filter operation where MYVARIABLE is a parameter sent.
Filtering using the variable sends nullpointer exception .This happens when I try to store data in ...
0
votes
0
answers
12
views
Invalid Project field in Pig FOREACH
I have a txt input consists of bigrams, year, count, volume_count, and page_count.
The task is to find out the most common bigrams for each year, and if there is a tie, I need to store all bigrams ...
0
votes
1
answer
30
views
ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1200: <line 16, column 46> mismatched input ',' expecting LEFT_PAREN
grunt> joined_data = JOIN filtered_features BY (store, date), sales BY (store, date);
2024-04-02 13:19:05,110 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1200: <line 16, column 46> ...
0
votes
0
answers
7
views
Apache PIG ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1200: <line 20, column 44> mismatched input '.' expecting SEMI_COLON
I am trying to execute pig latin script to find TF-IDF value for book dataset
My input file bookdataset.txt contains the following lines
Document 1: "The quick brown fox jumps over the lazy dog.&...
0
votes
0
answers
17
views
Apache-pig not running in local mode
I have set up pig but i am getting this error
**pig -x local Error: Could not find or load main class org.apache.pig.Main Caused by: java.lang.ClassNotFoundException: org.apache.pig.Main**
pig ...
0
votes
0
answers
25
views
Error using Pig to move data from HDFS to HBase
I've been trying to move data stored on hdfs to HBase using Pig.
PS: I'm running a pseudo-cluster hadoop.
Here are the steps that I'm executing
# Part 1 - Load Data into HDFS
# Create directory
hdfs ...
0
votes
1
answer
20
views
How to count value column in Apache Pig while filter out distinct key
I'm trying to count the total number of ordered prouduct(the quantity) by product_id.
The data looks like this.
(product_id, quantity)
which also is
(11, 5)
(11, 2)
(11, 1)
(12, 9)
(12, 1)
(13, 5)
(13,...
0
votes
0
answers
20
views
erorr dump apache pig
I load 1 data from hdfs:demo2.txt
enter image description here
command:data = LOAD 'hdfs://localhost:9000/demo1/demo2.txt' USING PigStorage(',') AS (name:chararray, age:int, city:chararray);
then I ...
0
votes
1
answer
26
views
How can i find each professional count and age count of a particular country( for eg US) in pig
this is the example. from this i want to find each professional count and each age group count.
filter_data = FILTER customer by loc=='us';
grunt> grp_data1 = GROUP filter_data by age; ...
1
vote
2
answers
122
views
Getting Error while running pig script on google Dataproc Cluster, all the parameter are defined correctly
This is my pig script -
fs -cp -f gs://$codepath/db_password.sh file://$dataprochome/db_password.sh;
fs -cp -f gs://$codepath/jdbc_daily_load_tables.py file://$dataprochome/jdbc_daily_load_tables.py;
...
1
vote
0
answers
85
views
What happens if some hadoop map-reduce Job or some hdfs command is running and in the between kerberos ticket expires?
Lets say we have kerberos secured Hadoop Cluster. ( For example Kdc is configured to provide kerberos tickets with validity of 2 hour and renewal period of 3 hours after this so total 5 hours. )
...
3
votes
1
answer
356
views
Error in submitting a pig job to Google Dataproc with properties file
I'm new to Dataproc and trying to submit a pig job to google dataproc via gcloud
gcloud config set project PROJECT
gcloud dataproc jobs submit pig --cluster=cluster-workaround --region=us-...
0
votes
1
answer
34
views
What is the best way to add BigDecimals in Hadoop?
I need to add BigDecimals in Hadoop. I'm currently using Apache Pig BigDecimalWritable but Pig seems to be completely outdated.
<dependency>
<groupId>org.apache.pig</groupId>
<...
0
votes
2
answers
32
views
Splitting string into a new line in apache-pig
I want to split the string in a dataset that is joined by a backslash (/) into a new line.
The example datatset is:
(David Jones / John Smith)
I want the result to be:
(David Jones)
(John Smith)
The ...
0
votes
1
answer
84
views
Having trouble Loading a multilevel JSON file into Apache Pig and saving it as csv
I'm super new to apache pig and was trying to load a multilevel json into pig and save it as csv.
Json file I have - home/vikaspattathe/dataset/sample.json
{"_id":{"$oid":"...
0
votes
2
answers
50
views
Apache Pig Sum calculation
I am trying to calculate the total shares but am receiving an error with the code, what am I missing?
grunt> history
1 stockprice = load 'PG/TutorialA/input/stockprice.csv' using PigStorage(',') ...
0
votes
0
answers
91
views
Store Pig result into Hive table using Hcat - ClassNotFoundException: org.datanucleus.NucleusContext
I want to load the simple json data into hive table without using Json Serde.
So I have tried to Load the json file in Pig and store the result into hive table.
A = LOAD '/data/pig_files/test.json' ...
0
votes
0
answers
41
views
Pig - System cannot find the file specified
I have error in pig latin while i run the basic command "Dump Students" for the file
>Students = LOAD 'C:\\Users\\avtar\\OneDrive\\Desktop\\student.txt' USING PigStorage(',') as (id:int, ...
0
votes
1
answer
90
views
Is there a pig / hadoop property that can be set for PIG_HEAPSIZE?
spark, hadoop, tez, etc. all have a list of properties that can be manually configured. example:
yarn.nodemanager.resource.memory-mb
or
spark.executor.memory
or
pig.exec.reducers.bytes.per.reducer,
...
0
votes
1
answer
315
views
ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2998: Unhandled internal error. org/python/google/common/collect/Lists
I'm just getting started with Pig and I'm facing lots of issues with running my first program. Any help is much appreciated.
I've tried resolving using these:
ERROR org.apache.pig.tools.grunt.Grunt - ...
0
votes
1
answer
59
views
Merging multiple files in Pig
I have several files (around 10 files) which I would like to merge together in Pig:
Student01.txt
Student02.txt
...
Student10.txt
I am aware that I could merge two datasets together by:
data = ...
0
votes
1
answer
59
views
Write a pig sequence of statements to calculate the number of ratings records and the average rating score by gender
this is what i have done so far.
A = LOAD '.mike/users.txt' USING PigStorage() AS (userID: int, age: int, gender: chararray, occupation: chararray, zip: int);
B = LOAD '.mike/data.txt' USING ...
0
votes
1
answer
52
views
PIG mapreduce output and HIVE
I have a file called test.txt with the records as below (disregard the dots):
"1" "a" "x"
"2" "b" "y"
"3" "c" "z&...
0
votes
1
answer
165
views
Sum of Salary in Apache Pig
Create employee and dept tables for the available files emp1.csv and
dept.csv.
Colnames: Emp: Empno, name, sal, did, branch, dno
Dept: deptno, name, loc
Retrieve total salaries to be paid for the ...
0
votes
1
answer
205
views
How to execute pig script and save the result in another file?
I have a "solution.pig" file which contain all load, join and dump queries. I need to run them by typing "solution.pig" in grunt> and save all the result in other file. How can ...
0
votes
1
answer
110
views
Apache Pig : Counting frequencies of "character"
Assume there is a text file named abalone_data with 3 attributes : name, gender and length
Kn,M,0.89
Un,F,0.77
An,M,0.89
Az,I,0.55
Au,M,0.72
with M is male, F is female and I is infant.
The question ...
0
votes
1
answer
88
views
Error 1070 Pig Could not resolve PigSorage using imports:
I'm trying to read a file with pig and I have the error indicated in the title.
data = LOAD '/user/cloudera/pigexample/commands' USING PigSorage('\n') as
(command:chararray);
DUMPdata;
The file ...
2
votes
4
answers
87
views
Use REPLACE in multicondition RegExp in Pig Latin
I have this set of data:
dump data;
This is a sample output: (this dataset is almost a million rows long).
("0",60,0,1,"Fri")
("1",47,0,1,"Mon")
("1",...
0
votes
0
answers
53
views
How to remove the "," comma which i am getting before a column value after joining data from 2 files
I am trying to run a pig script which picks data from 2 separate strings (string1, string2). And then using tokenize to separate them and later
Input:
String1 ="cow moon";
string2 ="...
0
votes
2
answers
214
views
Load Text Files and store it in Dataframe using Pyspark
I am migrating pig script to pyspark and I am new to Pyspark so I am stuck at data loading.
My pig script looks like:
Bag1 = LOAD '/refined/em/em_results/202112/' USING PigStorage('\u1') AS
(...
0
votes
1
answer
98
views
how to extract single element from a bag in pig?
My pig statement generates the following output:
({(10)},5201)
({(20),(20),(20)},3334)
({(30),(30),(30),(30)},4632)
({(40),(40)},3101)
({(50),(50)},3801)
({(60),(60),(60)},3959)
But I want to store ...
0
votes
1
answer
171
views
Why am I not getting the NULL values when using FILTER to remove CSV Headers in PIG?
I have this data below in a .csv file:
Needed_values,TEMP,Desc
,022.3,NewYork
3,022.30,India
,027.0,Australia
,027.00,Russia
1,027.1,Austria
,027.10,Norway
,036.2,Hungary
,036.20,Lithunia
2,785.52,...
0
votes
0
answers
115
views
Any way to validate pig script before running in hdfs cluster?
I'm new to pig script. The problem I'm facing is inability to validate how syntactically correct my script is. I have to upload it to hdfs cluster and run there just to realize I missed ';' in the end ...
0
votes
1
answer
58
views
Convert chararray to float
I am new to pig programming. I have one txt file and comma (,) as a delimiter. In amount columns i.e; amt_IN and amy_OUT are of type chararray with data $830.03 and $1392.54 respectively.
I need these ...
0
votes
1
answer
407
views
Pig Installation error: ERROR pig.Main: ERROR 2998: Unhandled internal error
ERROR pig.Main: ERROR 2998: Unhandled internal error. com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V
WARN pig.Main: There is no log file to write to.
ERROR ...
-1
votes
2
answers
37
views
Is it possible to use/query data using Pig/Tableau or some-other tool from HDFS which was inserted/loaded using a HIVE Managed table?
Is it possible to use or query data using Pig or Drill or Tableau or some other tool from HDFS which was inserted/loaded using a HIVE Managed table; or is it only applicable with data in HDFS which ...
1
vote
2
answers
311
views
HIVE/PIG JOIN Based on SUBSTRING match
I have a requirement where I need to JOIN a tweets table with person names, like filtering the tweets if it contains any person name. I have following data:
Tweets Table: (70 million records stored as ...
0
votes
1
answer
385
views
How to solve problem of DUMP not working in Apache Pig
I have Apache Pig version 0.17.0 (r1797386) .
I executed the following code. I have hadoop 2.9.2 on Ubuntu 18.04.
While run pig in mapreduce mode it gives following messages:
21/11/12 09:47:37 INFO ...
0
votes
1
answer
985
views
LOAD error mismatched input '-' expecting RIGHT_PAREN, column with hyphen
My code:
book = LOAD '/user/samanthafox/books.csv' USING PigStorage (',') AS (ISBN:chararray, Book-Title:chararray,Book-Author:chararray,Year-Of-Publication:chararray,Publisher:chararray);
dump book ;...
1
vote
1
answer
235
views
Best practices when running Hadoop MapReduce jobs/Hive scripts/Pig scripts etc
I am interested in understanding how ETL jobs like Hadoop MapReduce jobs/Spark Jobs/Hive scripts/Pig scripts are usually deployed in an on premises production/development environment.
Are they always ...
0
votes
1
answer
64
views
Hive Pig anti join
I am trying to merge two datasets: Events and mortality by patientid. Events have all the patientid including the patientid in mortality, but didn't have the timestamp and label (always equal 1) in ...
0
votes
1
answer
67
views
APACHE PIG - error Projected field [Units_Sold] does not exist in schema: group:chararray,D2:bag{:tuple(Item_Type:chararray,Units_Sold:int)}
Good afternoon - I have a Sales Dataset and am trying to see which Item has the most units sold.
Here is my code:
Country:chararray,
Item_Type:chararray,
Sales_Channel:...
0
votes
2
answers
114
views
data is converting to binary format while loading data into monet db using Apache pig
I am using MonetDb-Pig layer to load the csv data into Monet db. Internally it is using Binarybulkload commands to load the data but after loading data into table, the csv file values are not not ...
0
votes
0
answers
107
views
replace empty fields with string in Pig
I have the following table loaded onto Pig using the CSVloader:
list = LOAD '/user/department_list.csv' USING CSVLoader(',') AS (employee_name:chararray, city:chararray, department:chararray, ...
0
votes
0
answers
65
views
Load json files with different schema for a same element in Pig
I'm trying to load json files, and when I load a specific field, pig seems to recognize it for a string or a bag.
The Json element "program" in the file can be:
"program":"...
0
votes
0
answers
282
views
Pig: Invalid field projection - does not exist in schema - using BagConcat
I'm trying to filter by a field in the schema that I've created, but I can't figure out how to write the field name appropriately. Here's the schema description:
B: {datafu.pig.bags.bagconcat_data2_39:...
0
votes
2
answers
217
views
I need to filter 2 conditions from the same column in PIG HADOOP
I am new to pig.
What am I trying to achieve is getting 2 values from 1 table and find if a certain id has both of these items.
For example
Product Table
Product Name
Tofu
Ikura
Sushi
Order table
...
-1
votes
1
answer
71
views
Pig Latin unable to dump
I am unable to use the dump funtion for the alias 'TMP'. I had no issue when loading and dumping the files. I have no issue using the describe 'function' for both tables as well as the alias 'TMP'. I'...
0
votes
1
answer
235
views
Apache Pig Group by and Filter if multiple values exist?
I am trying to group multiple rows with the same IDs, and then check for each tuple in the group if it contains both values, for example:
(10461 , 55 )
(10435 , 17 )
(10435 , 11 )
(10435 , 72 )
(10437 ...
1
vote
1
answer
55
views
Using bicond to get ID?
I have this snippet of table called joinTbl:
PRODUCT_ID PRODUCT_NAME ORDER_ID PRODUCT_ID CUSTOMER_ID SALESPERSON_ID UNIT_PRICE
11 CAKE 10946 11 ...