Real-time container throughput data

List of the contents of this article:

• 1, Mall Data Warehouse ETL System Architecture Design
• 2, One of the ETL learning notes: What is ETL?
• 3、 What is a big data collection platform?

Shopping Mall Data Warehouse ETL System Architecture Design

ETL is Data Extract, Transform, Loading (Loa The abbreviated word of d) refers to extracting data from various heterogeneous data sources, and converting and integrating data from different data sources to obtain consistent data, and then load it into the data warehouse.

ETL refers to extracting data from the source system, converting data into a standard format, and loading data into the target data storage area, usually a data warehouse. ETL architecture diagram Design manager provides a graphical mapping environment that allows developers to define the mapping relationship, conversion and processing process from the source to the target.

In the process of realizing the supermarket data warehouse, you need to have more professional skills, with the ability of data architecture design and development, data mining and statistical analysis.

Offline data warehouse is one of the core tools of the data platform, which mainly prepares data for T+1 data reports.

ETL is the abbreviation of the three initials of Extraction-Transformation-Loading in English, which means data extraction, conversion and loading in Chinese.ETL plays a crucial role in making data warehouse systems. Compared with traditional database technology, ETL is not based on mathematical theory, but mainly for practical engineering applications.

One of the ETL learning notes: What is ETL?

1. ETL tool refers to a tool used to merge, clean, convert and export data from different data sources. ETL is the abbreviation of Extract, Transform and Load in English.

2. ETL, the abbreviation of Extraction-Transformation-Loading, the Chinese name is data extraction, conversion and loading.

3. First of all, let's understand the most basic definition: Well, some people simply call ETL data extraction. At least before learning, the leader told me that you need to make a data extraction tool.

4. ETL refers to the process of obtaining the original big data stream, then parsing it, and generating a set of available output data. Extract (E) data from the data source, and then convert it into available data through various aggregations, functions, combinations and other transformations (T).

5. ETL is the abbreviation of Extract-Transform-Load in English, which is used to describe the process of extracting, transform and loading data from the source to the destination.The term ETL is more commonly used in data warehouses, but its objects are not limited to data warehouses.

6. Most of the pure BI developers naturally choose mature ETL tools for development. Of course, there are also those who write program scripts as soon as they come up. The masters of such BI developers are basically programmers.

What is a big data collection platform?

1. The NLPIR big data semantic intelligent analysis platform is based on the comprehensive needs of Chinese data mining, integrating the research results of network accurate collection, natural language understanding, text mining and semantic search, and is a shared development platform for the whole technical chain of Internet content processing.

2. Big data refers to a collection of data that cannot be captured, managed and processed by conventional software tools within a certain period of time.

3. The big data platform is to calculate the increasing amount of data generated by today's society. A platform for the purpose of storage, operation and display. Is it to allow developers to either run the written programs in the cloud, or use the services provided in the cloud, or both.

4. Big data collection, that is, the collection of structured and unstructured massive data from various sources. Database acquisition: Sqoop and ETL are popular, and traditional relational databases MySQL and Oracle still act as data storage methods for many enterprises.