What is Big Data?

  • Big Data is one of the hottest topics in the tech world.
  • This is not only related to computers, it comes under a blanket term called IT, which is now part of almost all other technologies and fields of studies and business.
  • Big data is not a Big Deal. Sure pretty big deal to confuse all.
  • Here let me try to explain what big data is.
What is Big Data?
  • All the data categorized or not categorized, present in user nodes are collectively called Big Data.
  • All this data can be used to get different results using different types of analysis.
  • Not necessary that all analysis uses all the data.
  • Different analysis uses different parts of big data to produce the results & predictions necessary.
How big is Big Data?
  • Collecting of Hugh data is Big Data, but many researchers agree that big data cannot be manipulated using normal spreadsheets and regular tools of database management.
  • It requires tools like HADOOP, so that all data can be analyzed at one go.
  • For analyzing data, people use to create different sets based on one or more common fields, so that analysis can be easy.
  • Now we have tools that can analysis data of Hugh volume, these tools will categorize that data even as they are analyzing it.
Big Data Concepts
Experts say that the Big Data concepts are three Vs
  1. Volume
  2. Velocity
  3. Variety
Some others add few more to the list
  1. Visualization
  2. Veracity (Reliability)
  3. Variability
  4. Value

Comments

  1. Anonymous11 July 2016 at 21:03

    Nice post...

    ReplyDelete
  2. Peter22 July 2016 at 11:38

    a single place with good collection of topics. nice for a person who starts to read whats is big data.

    But why there is nothing related to gpdb! Expecting that too with similar write-up.

    Altogether nice approach.

    Regards,

    ReplyDelete
    Replies
    1. nmarshal23 July 2016 at 22:06

      Thanks, As expected gpdb related topic will be published soon...

      Delete

Post a Comment

Popular posts from this blog

GP - Kerberos errors and resolutions

Issue:1. Ticket granting has expired with the error message, ERROR: psql: GSSAPI continuation error: Unspecified GSS failure. The Minor code may also display the error message, GSSAPI continuation error: Ticket expired. Solution: Create a new ticket on the Master server using the following command: kinit -k -t /home/gpadmin/gpdb-kerberos.keytab db_user Note: -t is the keytab file and db_user is the user. Issue:2. The library required to run the kinit command does not exist in the LD_LIBRARY_PATH  set in the Greenplum binaries, producing the following error message: ERROR: kinit: relocation error: kinit: symbol krb5_get_init_creds_opt_set_fast_ccache_name, version krb5_3_MIT not defined in file libkrb5.so.3 with link time reference Solution: Unset the LD_LIBRARY_PATH. IMPORTANT NOTE: The above fix will cause the following error when you try to run Postgres commands:  ERROR: psql: error while loading shared libraries: libpq.so.5: cannot open shared object ...
Read more

How to set Optimizer at database level in greenplum

How can I turn ON optimizer at Database level? You can set the optimier ON at database level using below command; ALTER DATABASE databasename SET OPTIMIZER = ON; Once this is done, kindly do reconnect to Database to see the change. How do I check if optimizer is set (ON/OFF) in a given database? You can use the show command as below to check psql -d databasename -c 'show optimizer'
Read more

GP - SQL Joins

Image
Available Join Types? Row Elimination Minimize Data Movement Join Implementation Seen in Query Plans Join Types: Inner Join Resulting data set is obtained from joining two tables on a common column Each row in left table is compared with each row in right table Matching rows will be present in result set Most commonly used Also know has Simple Join or EQUI-Join Left Outer Join Returns all rows from the left table even if there is no matching rows in right table. Will also get the matching rows Rows that are not matched in the right table will not be included in the result set Right Outer Join This is similar to left outer join, just need to flip the join Full Outer Join Will return all rows from both the tables where there is a match We will receive a NULL for rows which don’t match Cross Join Which is also called has Cartesian products Result set will have all rows on the left table paired with all rows on the right table Exam...
Read more