Data

Definition of data

ISO 14721 defines data as

A reinterpretable representation of information in a formalized manner suitable for communication, interpretation, or processing. Examples of data include a sequence of bits, a table of numbers, the characters on a page, the recording of sounds made by a person speaking, or a moon rock specimen.

Two types of data

Two (three) types of data (a)

data that describes things: master data
data that describes events: transactional data
Some authors (Vayghan et al 2007) also add historical data to the list.

Two types of data (b)

Qualitative data
Quantitative data

(See also level of measurement)

Structured and unstructured data

Data might also be divided into

Structured data (Relational databases etc.)
Semi-structured data (i.e. PDF, XML)
Unstructered data (like raw video and audio)

Because of the complexity of unstructured data, Data profiling tools usually focus on structured and semi-structured data.

Challenges with increasing data volumes

The ever incresing volume of collected data causes challanges such as

Age of data: old data might not represent the a factual truth
Storage Cost: the cost to store data increases. Contrary to popular opninion, storage prices don't get cheaper. Especially SANs are sometimes prohibitively expensive and are easily the most costly factor in a data center.
Complexity: Managing lots of data is complicated and puts even more pressure on its cost
Data sprawl: The size of data is increased also because it is copied between organizations, divisions, test, integration and prod systems etc.

TODO

TXR for data munging.

Visualize data with statistical graphics.

In order to get a value from data, it needs to be moved from one location to another.

The four W:

What is this data: Data dictionary
Where is my data: Data catalog
Where did it come from: Data lineage
Who knows about the data: Data Stewardship

Regulations that require data hygiene, such as

BCBS 239 (Principles for effective risk data aggregation and risk reporting)
GDPR (General Data Protection Regulation)

DSVGO = Datenschutz-Grundverordnung, rules for processing data related to persons in the EU. Compare with the the CCPA, Califrona Consumer Privacy Act in the USA.

Storage

DAS: Direct-attached storage, connected through a HBA (Host Bus Adaptor)

NAS: Network-attached storage. Access files (rather than blocks)

Protocols: NFS, SMB, proprietary

SAN: Access blocks (rather than files)

Protocols: SCSI, iSCSI, Fibre Channel, Infiniband