1

I am a rising senior CS major at a large state university and I'm working as an intern at a large publicly traded technology company in their data science department. I've learned in school about data structures and algorithms (Maps, Trees, Graphs, Sorting Algorithms, Seaching Algorithms, MapReduce, etc.) and I have some experience through personal projects with MySQL and SQL queries.

My project for this internship is to create a dashboard for displaying analytics gathered from a Hadoop database. I'm struggling to understand how this data is structured and queried. I'm pretty sure all of the data in Hadoop is coming from the production Oracle Relational DB that runs their platform. I guess my core question is why are Hadoop and distributed processing required to gather analytics from a database that is already in a structured format? What does the data look like stored in Hadoop? Are there tables like MySQL, or JSON docs like MongoDB? I will be querying Hadoop through Druid but I'm not sure what is even in this database.

The engineers I have been working with have been great at explaining things to me, especially questions about their specific implementation, but they only have a certain amount of time to dedicate to helping an intern and I want to take the initiative to learn some of this on my own.

As a side note, it is incredible how different it is working on a school project than a project at a company with millions of active users and petabytes of sensitive information.

2 Answers 2

0

Hadoop is not a database, and it therefore has no such thing as tables or any inherit structure of relations or documents.

You can place a schema over stored files of various formats like CSV, JSON, Avro, Parquet, etc using Hive, Presto, SparkSQL, for example, but these are all tools that read from the Hadoop FileSystem, and not part of Hadoop itself. Tables and databases at that level are only metadata, and not entirely representative as what the raw data looks like

Hadoop is simply capable of storing more data than an Oracle database, and is free, however for fast analytics, it's recommended to compute statistics within Hadoop frameworks in distributed fashion, then load back into a indexed system (such as Druid) or just any actual database

0

I get your question. Basically you are trying to understand what and how are the data present in Hadoop and why not Traditional database but the data from a Traditional Database in Hadoop.

Few main points to understand when it comes to Hadoop,

1. Hadoop is not only for structured data, it can be used for semi structured and unstructured data as well. Mostly for the purpose of Analysis of data.

2. Hadoop is a framework and has different components present in it. The majorly used components to query structured data from HDFS is Hive and Impala.

3. As far as structured data are concerned, Hadoop has HDFS and Hive Metastore for storing the data in a structured way. HDFS stores only data files (such as text, avro, parquet, json, etc.,) not metadata (such as column name, count of rows, etc.,). On the other hand Hive Metastore is basically traditional database such as MySQL, Postgres, etc., and this carries only the metadata. So metastore knows where the data of a table is stored in HDFS i.e., the HDFS file path. For more on this point - you can read one of my posts HERE

4. Why Hadoop? Hadoop is designed to store big amount of data with high availability due to its distributed nature. Also, Hadoop is meant for WRITE once and READ many times - meaning it is more for analytics and reporting purposes not for transactional purposes like how the traditional databases are being used. More importantly, its open-source!

Hope this helps you in getting a baseline!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.