Big Data

Every day, billion of gigabytes of high-velocity data are created in a variety of forms, such as social media posts, information gathered in sensors and medical devices, videos and transaction records.

Not a Big Data – Not everything is actually a big data problem. There are lots of cases where you can use traditional systems to store, manage, and process your data. For example order details for a purchase at a store or information about a person’s bank details

What is a Big Data? – All orders across hundreds of branches nationwide or  All bank transactions made on the Wisconsin state area during the year

Definition of Big Data – A data set of terabytes or more to be ‘big data’.  It’s data which can’t comfortably be processed on a single machine. Big Data is not just size of the data, but also data is created very fast and data from different sources is in different formats. Most data is not worthless but actually does have a lot of value.

Big data is high volume, high velocity, and/or high variety information assets that require new forms of processing to enable enhanced decision making, insight discovery and process.

The 3 V’s of Big Data
VolumeThe size of data. Storing the data is only one part of the equation; you also need to be able to read and process it efficiently. Storing a terabyte of data on a SAN (storage area networks) isn’t so hard, but streaming the data from the SAN across the network to some central processor can take a long time, and processing it can be extremely slow.

Hadoop, which stores data in a distributed way across multiple machines

VarietyThe data is often coming from lots of different sources and in many different formats. Lots of the data you want to store is what we tend to call ‘structured data’, and ‘unstructured data’, or ‘semi-structured data’.  ‘unstructured data’, mean the data arrives in lots of different formats. For example, a bank might have a list of your credit card and account transactions, scans of your checks. All of that data is in a variety of different formats, and it can be hard to store and reconcile it all using traditional systems.

VelocityThe speed at which the data is being generated, and the speed at which it needs to be made available for processing.  For example, E-Commerce Web site, what products you’ve looked at in the past, site could recommend similar products the next time you visit our site. Just spending five minutes looking at a particular item, site could maybe send you an email informing you when that item is on sale.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: