DEDAO App is a knowledge service product produced by the Internet knowledge service platform “DEDAO”, aiming to build a general education university serving lifelong learners. Online, DEDAO App provides meditation practitioners with a set of multi-thinking model courses dedicated from all walks of life. DEDAO App organizes experts from different industries to conduct collective learning.

CHALLENGE

• The pressure of maintaining the data pipeline is increasing for the backend team, and their productivity is reaching a bottleneck.The traditional SQL method used for data ETL requires an average of one person to maintain 20-30 data pipelines, which is a huge burden on the team.
• Batch processing causes data delays, and the dual-write method is unable to ensure data quality.For instance, the algorithm engine data is obtained through regular batch processing every day, causing currently unavailable products to be recommended to users and significantly affecting customer experience.
• With an increasing number of business requirements, the demand for data real-time processing is also increasing.Traditional SQL methods and regular batch processing methods cannot meet the requirements for rapidly growing data demands and real-time data processing needs.

TAPDATA SOLUTION

• Using Tapdata’s low-code approach to batch ETL work.Based on the advantages of Tapdata’s low-code, visual, and easy-to-deploy features, the backend team can use configuration to complete heterogeneous data real-time synchronization, cross-database data real-time aggregation, and other work in batches, and new employees can quickly get started.
• Tapdata CDC log parsing ensures data real-time synchronization and accuracy.Tapdata uses its self-developed CDC log parsing technology, which is more user-friendly and reliable than the traditional method of double-writing data to Kafka for consumption, and can also solve the problem of not being able to complete it within a limited time due to large data volumes under batch processing.
• Preparing common data models for common requirements and automatically updating data in real-time.With Tapdata’s data development and modeling capabilities, common data integration requirements can be pre-sorted and modeled, enabling quick response to business data needs.

CUSTOMER BENEFITS

• Saved 2-3 manpower costs, and only 0.2 manpower is needed to maintain all data services during regular times, significantly reducing costs and increasing efficiency.
• Data quality has greatly improved, supporting the accuracy and real-time recommendation of products by algorithms, leading to an increase in both user experience and performance.
• Business requirements are responded to more timely, and data is upgraded from daily batch processing to real-time updates, allowing efficient integration of heterogeneous and cross-system data.