11

Im new to Bigdata and currently learning Hive. I understood the concept of InputFormat & OutputFormat in Hive as part of SerDe. I also understood that 'Stored as' is used to store a file in a particular format just like InputFormat. But I don't understand what is the significant difference between using the 'InputFormat, OutputFormat' & 'Stored as'.

Any help is appreciated.

1 Answer 1

26

Hive has a lot of options of how to store the data. You can either use external storage where Hive would just wrap some data from other place or you can create standalone table from start in hive warehouse. Input and Output formats allows you to specify the original data structure of these two types of tables or how the data will be physically stored. From your client side you will keep working with a table using sql, but on the low level it would be either text file or sequence file or hbase table or some other data structure.

InputFormat and OutputFormat - allows you to describe you the original data structure so that Hive could properly map it to the table view

SerDe - represents the class which performs actual translation of data from table view to the low level input-output format structures and opposite

Generally your process would be like this: HDFS files --> InputFileFormat --> Deserializer --> Row object --> Serializer --> OutputFileFormat --> HDFS files

Stored as - specifies such storage format which includes Input and Output formats for you new tables in Hive

These attributes can really affect the performance, the overall size, data schema evolution support or enable such features as ACID. You can follow the steps described in this article to see things are working on the low level and to get some general information about most commonly used formats - https://oyermolenko.blog/2017/02/16/structuring-hadoop-data-through-hive-and-sql

5
  • I've gone thru the information you gave. Im clear now. Appreciate for your time.
    – Metadata
    Feb 24, 2017 at 6:54
  • Such a clear answer - wish official documentations explained it in this way ! Many thanks Apr 11, 2018 at 19:26
  • 1
    @AbhinandanDubey Thank you for your feedback. I 100% agree with you regarding documentation and unfortunately this problem exists in all Big Data solutions.
    – Alex
    Apr 13, 2018 at 9:27
  • @Alex Can I have different InputFormat and OutputFormat? For e.g. XML Input format and JSON output format ?
    – Bala
    Nov 20, 2018 at 22:07
  • 2
    @Bala Don't think it would work. Input and output format targeting single entity - table. You cannot read XML data and write Json data within the same file. Both formats should keep synced logic for working with the file
    – Alex
    Nov 21, 2018 at 22:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.