Big Data and its importance in today's world

Big Data

Big data refers to huge the volume of data (structured or unstructured) that cannot be stored and processed using the traditional approach (i.e. using computers processors or devices) within a given time frame.

How much data is called Big Data?

Data in Gigabits, Terabytes, Petabytes or Exabytes or anything that is larger than these in size. Even a small amount of data can be referred to as Big data depending on the context it is being used.

Example. If you try to attach a document to an email that is of 100 MB, we would not be able to do so. As the email system would not be able to support an attachment of this size, therefore this 100MB size of attachment with respect to email can be termed as Big Data.

Let’s take an example of real-world scenarios.

The popular networking sites such as Facebook, Twitter, Instagram, Linked, YouTube etc each receives a huge volume of data on a daily basis. Facebook receiving 100TB of data each day, Twitter processes around 400 Million tweets each day, Linked receiving tons of TB of data each day, On YouTube each minute around 48 hours of new videos are uploaded.

But as the number of users is increasing day by day, storing and processing of this data becomes challenging. Since this data holds a lot of valuable information, this data needs to be processed in a short span of time. By using this valuable information, companies can boost their sales and generate more revenue. But using the traditional computing system, we would not be able to accomplish this task within a given stipulated time. Therefore, we can term this data as Big Data.

How is Big Data classified?

Basically, Big Data is classified in 3 categories.

1.     Structured Data – The data that have a proper format associated with it is called Structured Data.

·       Example – Data present within the databases (College Student database etc), Excel Spreadsheet etc.

2.     Semi-Structured Data – The data that does not have a proper format associated with it is called Semi Structured Data.

·       Example – Data present within the Emails, Log files, Word Documents etc.

3.     Unstructured Data – The data that does not have any format associated with it is called Unstructured data.

·       Example – The image files, audio files, video files etc.

Characteristics of Big Data

·       Volume – The amount of data that is being generated.

·       Velocity – The speed at which the data is being generated.

·       Variety – The different types of data being generated.

·       Veracity – The quality or the value of the data that is being generated.

Other important characteristics.

·       Variability – The inconsistency in the data that is being generated.

·       Value – The utility of the data being generated.

·       Virality – The speed through which data can be transmitted over a network.

How is Big Data Stored and processed?

The traditional approach of storing Big data.

In a traditional approach, the data being generated in an organization such as Stock Markets, Banks, Hospitals etc. are                                                                                                                                      given as an input to ETL System (ETL – Extract, Transform, Load these database functions are combined into one system and used to pull out data from one system and transfer it to others). ETL system would convert this data to a proper format and loading it into the database. Now, the end-users can perform analytics and generate a report from this data.

But as this data grows, it becomes a very challenging task to manage and process this data using Traditional approach

Drawbacks of using Traditional Approach

·       Expensive System – It requires huge investments in establishing or upgrading the system. Therefore, not being feasible for small and mid-size companies.

·       Scalability – It becomes a challenging task to expand the system when the data grows.

·       Time-Consuming – Traditional approach the system takes a large amount of time to process and extract valuable information from the data.

Applications of Big Data in Various fields.

·       Healthcare – With the advancement in technology and development of health tracking devices, every day tons of activities of individuals are monitored and analysed. There is no scope for human intervention in handling these data. Such data are termed as Big data and are processed to give a valuable outcome.

·       Education – As there is an increase in demand due to the development in technology, education institutions and colleges collect data from various sources and processes to determine the on-going demand for a particular profession/degree and develops the curriculum as per the needs. This helping the industry to meet the demand by availing the people with that particular domain knowledge.

·       Insurance – Insurance companies collect the data on “Determinants of health” such as food habits, TV consumption, marital status, purchasing habits etc and processes the data to determine the age expectancy of the individual and also determine the premium for their health policies.

·       Information Technology – The organizations applying the principles of data along with the machine learning and machine intelligence, the IT department can predict the potential issue and help them in avoiding or overcoming them. Thus, Big Data plays an important role in Information Technology.

Benefits of Big Data Processing

Improved Customer Services – With Big Data the companies can analyse the user habits patterns from various social media sites and target their users more precisely. Thus, increasing the customer satisfaction rate.

Early Identification of Risk of Product/Services – By replacing the traditional feedback form with Big Data systems, the organizations can detect early about the change in demand and hence make changes in their strategies.

Internet of Things (IoT)

The Internet of Things (IoT) alludes to the ever-developing system of physical articles that highlight an IP address for web network, and the correspondence that happens between these items and other Internet-empowered gadgets and frameworks. In direct words, the Internet of Things (IoT) is an environment of associated physical items that are available through the web. It is likewise alluded to as Machine-to-Machine (M2M), Skynet, or Internet of Everything. There is a heterogeneous blend of correspondence advances, which should be adjusted to address the necessities of IoT applications, for example, vitality productivity, speed, security, and dependability. In this unique situation, it is conceivable that the degree of decent variety will be scaled to a number reasonable network innovations that address the requirements of the IoT applications, are embraced by the market, they have just ended up being useful, stick by solid innovation collusion. Instances of measures in these classifications incorporate wired and remote advancements like Ethernet, WI-FI, Bluetooth, GPRS, GSM, and ZigBee.

Characteristics of Internet of Things

1-Interconnectivity: as to the IoT, anything can be interconnected with the worldwide data and correspondence framework.

 2- Dynamic changes: The condition of gadgets change powerfully, e.g., dozing and awakening, associated as well as detached just as the setting of gadgets including area and speed. Also, the number of gadgets can change progressively.

 3- Things-related administrations: The IoT is equipped for thing-related administrations inside the requirements of things, for example, security assurance and semantic consistency between physical things and their related virtual things. To give thing-related administrations inside the requirements of things, both the innovations in the physical world and data world will change.

 4- Heterogeneity: The gadgets in the IoT are heterogeneous as dependent on various equipment stages and systems. They can cooperate with different gadgets or administration stages through various systems.

 5- Availability: Connectivity empowers organize openness and similarity. Openness is jumping on a system while similarity gives the normal capacity to expend and create information.

 6- Gigantic scope: The number of gadgets that should be overseen and that speak with one another will be in any event a request to extent bigger than the gadgets associated with the present Internet.

Challenges faced by IoT

1- Scalability
2- Security
3- Technical Requirments
4- Technological Standardisation
5- Software Complexity

Several solutions were proposed to solve the above-mentioned problems most of them worked, for example- 

 1- Beating similarity issues is a critical IoT obstacle, however, rising organizations are beginning to empower expanded interoperability through open-source advancement.
 2- Governments and industry bodies need to set measures and guidelines for the different ventures to guarantee that information isn't abused.
 3- IoT needs solid confirmation techniques, encoded information, and a stage that can follow inconsistencies in a system.

Advantages of IoT-

 1-Conveyance- Better correspondence is conceivable over a system of interconnected gadgets, making the correspondence of gadgets progressively straightforward, which decreases wasteful aspects. Procedures, where machines need to speak with one another, are made increasingly proficient and produce better, quicker outcomes. The ideal model for this is a device at an assembling or creation unit.

 2- Financially viable- As referenced in the point above, correspondence between electronic gadgets is made simpler as a result of IoT. This helps individuals in everyday aspects. Moving information bundles over an associated arrange set aside time and cash. Similar information that would set aside a long effort to move should now be possible a lot quicker, on account of IoT.

 3- Access data -You can without much of a stretch access information and data that is sitting a long way from your area, continuously. This is conceivable given the system of gadgets, an individual can get to any data sitting from any piece of the globe. This makes it exceptionally advantageous for individuals to approach their work, regardless of whether they are not truly present.

 4- Mechanization -Mechanization is the need of great importance to oversee ordinary undertakings without human intervention. Computerizing undertakings in business help support the nature of administrations and decrease the need for human intercession.

Disadvantages of IoT

 1- Employment- With each undertaking being mechanized, the requirement for human work will decrease radically. This will directly affect employability. As we head into the fate of IoT, there will be an obvious decrease in the recruiting procedure of experts.

 2- Intricacy- A different system that interfaces different gadgets is the thing that we call IoT. A solitary proviso can influence the whole framework. This is by a long shot the most entangled part of the web of things that can have an enormous impact.

 3- Protection and security -In the present tech-driven world, every single gadget that an individual uses is associated using the web. This expands the danger of any spillage of information that may be significant. This is a significant disadvantage of sharing data, as private data probably won't be sheltered and could be hacked by outsiders without any problem.

 4- Trustworthiness- We may not see it, however, we are seeing a significant move in innovation and its execution in regular day to day existences. There is no uncertainty that innovation is overwhelming our way of life, mirroring a human's trustworthiness on innovation. In the event of a bug in the framework, there are high odds of each related gadget getting tainted.

Conclusion-

Web of Things is another level of the Internet and it is a key research point for analysts, software engineering, and data innovation territory. Because of its exceptionally different zone of use and the heterogeneous blend of different interchanges and implanted innovation in its design it favours a lot for its user.