Big Data

Big Data is a term referring to extremely large and complex datasets generated by both humans and machines. They cannot be easily managed or analyzed using traditional data processing tools, especially ordinary spreadsheets. The term encompasses:

  • Structured data, such as an inventory database or a list of financial transactions.
  • Unstructured data, for example various types of posts or videos on social media.
  • Mixed data, such as those used to train large language models for artificial intelligence. Such data can include virtually anything—from corporate spreadsheets to literary works.

In today’s world, data has become capital. Many of the largest global companies constantly analyze data to improve efficiency and develop new initiatives, and many of their latest products are now based on data. The emergence of large datasets is closely tied to the advancement of computer technology. The rapid growth of computing power and storage capacity has led to the gradual accumulation of increasingly vast amounts of data. However, size alone does not define Big Data, which is why the use of the letter “V” was proposed—derived from the first letters of the words that describe it:

  • Volume – It refers to the amount of digital data that is collected and stored. Currently, it is growing at an increasingly rapid pace. It is difficult to precisely determine the size that qualifies as Big Data, since what was considered a large dataset 10 years ago may no longer meet today’s standards. For example, in 2008 CERN, after extracting 1% of its data, had around 25 PB of data to process annually.
  • Velocity - It is the speed at which data is received, processed, and utilized for further actions.
  • Variety - It is the availability of many types of data. New partially or completely unstructured types of data, such as audio or video, are emerging, which require prior processing to obtain and manage metadata.
  • Veracity – The accuracy of the collected data and the assessment of how reliable it is.
  • Value – Big Data contains extensive and in-depth information. Hidden within are insights that can bring benefits. The values they can provide are internal, such as the identification and optimization of various processes, or external, for example customer profiling.