The share of online purchases is constantly increasing, driven by changing paradigms and the proliferation of internet stores. The trend of selling products on marketplaces, which has seen significant growth in recent years, is slowing down. Consumers increasingly opt for niche online stores, and the number of such stores is growing each year.
When considering an online store as a fully functional business, with a primary focus on sales growth, there arises a need to enhance efficiency, both operationally and in marketing.
Especially during the transformation stage from an in-house store to a mid-size business, numerous questions arise regarding data-driven decision-making, as data sources (applications, services) are highly diverse.
On the one hand, accessing all available or accessible data is easily achievable (Google Analytics, Facebook Ads, Google Ads, Stripe, Paypal) — all these systems offer their mechanisms for data access and report generation.
On the other hand, all these systems are very different, and consolidating all necessary metrics or charts into a unified report becomes a challenging task without considerable manual effort.
It is precisely at this juncture that the need arises for consolidating data from various sources in one place for subsequent analysis, visualization, or reporting.
What is ETL?
ETL (Extract, Transform, Load) is the process of extracting/gathering data from one source, transforming it, and loading it into a target destination. In other words, it is the process of moving data from several different sources (inputs) to one place, such as a data warehouse. First and foremost, this is necessary for the standardization of data access, as each source has its interface for data access (API), complicating the construction of reports or analyses. Having a single source with all the data significantly simplifies the report-building process.
The ETL process is quite complex to implement, but at a fundamental level, it consists of three stages:
-
E - Extraction:
The stage of obtaining data from a source, which can be a database, API, or text files.
-
T - Transformation:
The stage of changing data includes cleaning it from invalid information, altering the structure, and transforming the data into a format more suitable for analysis.
-
L - Loading:
The stage of moving prepared data to the target destination is usually a database such as Progress, Google BigQuery, Snowflake, etc.
As a result, the data warehouse contains high-quality and homogeneous data from various sources in one place, facilitating further analysis and utilization of information (especially for e-commerce where the number of data sources can be extensive).
Data Warehouse
A Data Warehouse is a centralized repository where prepared information from various sources is stored for utilization. The primary purpose of creating your data warehouse is to preserve information from different sources for subsequent use in business intelligence (BI) systems, internal reporting systems, or operational support.
Key requirements in the context of e-commerce include:
-
Relevance:
Ensuring continuous updating and synchronization of data to keep the information in the data warehouse always current and accurate.
-
Completeness:
Storing all necessary data from various sources to have a comprehensive set of information for further analysis and decision-making.
-
Convenience of Selection:
Establishing a sensible and easily accessible structure for the repository, enabling efficient queries (SQL) and obtaining necessary data for diverse business needs.
A dedicated data warehouse in e-commerce serves as a central point for analysis and interaction with data, enabling businesses to gain valuable insights, optimize strategies, and refine operational processes.
Shopify: What data can be extracted?
As Shopify is a cloud platform, direct access to data is not possible. However, Shopify provides a high-quality API for data access. The Shopify API is convenient for processing and integrating data into a data warehouse.
Using the Shopify API, you can extract various data, including:
- Order Data
OrdersOrder ItemsDraft ordersRefundsFulfillmentsCustomers
- Product Data
ProductsProduct VariantsGallery
- Transaction Data
Transactions
- Payouts Data
PayoutsPayout Transactions
- User Data (only for Shopify Plus)
- Inventory Data
Inventory LevelsInventory ItemsLocations
- Checkouts
CheckoutsShopping Carts
However, it's essential to note that some data is not available through the API, which may pose challenges for comprehensive analysis and reporting. Such data includes:
- Historical Inventory: Changes in the quantity of products over specific dates;
- Inventory Transfers: Data about the movement of products between warehouses;
- Purchase Orders: Information about orders from suppliers;
- Web Analytics: Data about visitors, page views, and other web analytics.
To compensate for these missing data points, integrating data from other systems, such as Google Analytics and Google Search Console, into the warehouse is recommended. This allows for a more comprehensive understanding of customer interactions with the website, search engine queries, visibility, and more.
ETL for Shopify
When considering the ETL process for Shopify, both positive aspects and some challenges arise during data processing from this platform.
Positive Aspects:
Quality of Shopify API:
The high-quality Shopify API enables the retrieval of a large amount of well-structured data. This includes information about products, orders, customers, and more, providing an excellent foundation for further analysis and business analytics.
Webhooks Support:
Shopify supports webhooks, allowing the ETL system to react to any changes in data in real time. This means that data can be updated almost in real-time, ensuring data freshness.
Challenges:
Unstructured Data (raw JSON):
Some data, such as meta fields and tags, is presented in an unstructured format (raw JSON). This significantly complicates the processing and analysis of this data, as it lacks a clear structure. Using these data requires a transformation stage that parses the fields and branches them into respective tables.
Role of the Transformation Stage:
Since some data arrives in an unstructured format, the transformation stage plays a crucial role in processing this data.
In conclusion, when using the ETL process in conjunction with Shopify, it is essential to consider both positive and challenging aspects to ensure effective processing and analysis of e-commerce business data.
Overview of ETL Systems
Currently, several ETL systems support Shopify ETL at a relatively high level. However, it's important to note that none of these systems fully leverage the capabilities of the Shopify API. This can lead to limitations in data extraction and processing special fields, such as metafields or tags, in a format suitable for report building.
Regarding the processing of special fields like meta fields or tags, most ETL systems may not fully comprehend their essence. This complicates report building and analysis since these additional fields often contain crucial contextual information about products, orders, customers, and third-party apps.
It's crucial to consider these limitations when choosing an ETL tool for Shopify. With large data volumes and complex structures, selecting a tool that best suits the unique needs of your business and ensures full compatibility with the features of the Shopify API is important.
Summary
Creating your data warehouse through the ETL process in modern e-commerce is a critically important step for future growth and making informed data-driven decisions. As your store grows, the urgency of this process becomes increasingly relevant.
In conclusion, establishing your data warehouse is not only a step forward but also a strategic decision to ensure success and resilience in the face of rapidly changing demands and trends. It's an investment in opportunities for improvement, innovation, and the overall development of the company.
