Let’s say you are working on a dope new service. It is starting to grow. Then someone important asks “How many X are our users making?” or “How many times are users doing Y?”. In engineering terms they are asking you to synthesize the data produced by your service into a format that can be used to make business decisions.
Initially you may be able to answer these questions by writing scripts or running one off queries, but eventually you will be getting too many data requests that are too complicated. You will get sick of maintaining and running a bunch of complicated scripts and will decide to build a reporting pipeline.
Be careful adding a reporting pipeline as an afterthought. If not done carefully it can be overly fragile and can even slow down the development of the entire service. If it produces inaccurate data the consumers can lose trust. The principles I introduce here will help you design your reporting pipeline so that it will scale with your system with minimal added cost.
I will not be discussing here how to determine what data should be produced by your reporting pipeline or what technologies may serve you best when building your pipeline.
Reporting Facts definition:
“Reporting facts” are groupings of information related to a single occurrence of an action within your service. An action within your service could be “A user created an account” and the fact could include when the account was created and the username.
Principles
Reporting facts must be separate from application logic
In Scenario A you will be thinking: “Ah this piece of data needed for reporting is already produced by Service X. When it makes a call to Service Y, I can just have it also send that data to the reporting pipeline”. This saves time in the short run but will cause problems later.
By sending the same data to Service Y and your reporting pipeline (still talking about Scenario A), you tightly couple the implementation of Service X to your reporting pipeline. If Service X wants to change the information it sends to Service Y, it needs to also consider the needs of the reporting pipeline.
Save yourself some time by creating separate objects for use specifically by the reporting pipeline (illustrated by Scenario B). It may look redundant at first, but when you change the implementation details the reporting considerations will already be separated.
Reporting facts must be produced by the service doing the work
Initially you may only have one source of reporting data. Over time there will be more and more services publishing data to your pipeline. Each service must manage their own special cases when producing reporting facts. This allows the reporting pipeline to focus on its purpose-to produce data for making business decisions.
Let’s say there are two ways to create a new account, by entering your information or by logging in through Google. If you enter your information manually then the account creation service will receive an account creation date in the DD-MM-YYYY format. If you login through Google then the account creation service will receive an account creation date in the MM-DD-YYYY format. Who should be in charge of standardizing these date formats, the reporting pipeline or the account creation service?
The answer is that the date standardization should be handled by the account creation service. The account creation service is most aware of the information needed to decide which format to use depending on the account creation method.
Reporting facts must be saved for auditing
You need a record of the unaltered reporting facts sent to your reporting pipeline. Then when someone complains “the average browse time is wrong” you can first inspect the materialized reporting facts and manually perform the averaging. If your calculation matches what you are reporting then your reporting pipeline is operating correctly but is not receiving the correct data. If your calculation doesn’t match then your reporting pipeline is not operating correctly.
You must validate reporting facts
When you ingest reporting facts into your reporting pipeline you must validate that the facts are properly formed. There will always be errors that lead to bad data being passed to the reporting pipeline. Maybe in one edge case a service fails to populate the timestamp when the customer made a purchase.
If you erroneously ingest invalid reporting facts and publish them to consumers, it will undermine their trust which is hard to earn back.
Parting Thoughts
Even though reporting requirements may be an afterthought when building your service, you must design it as a first class application.
When you are moving from design to implementation you may want to learn more about tools used to process large amounts of data. I recommend Designing Data Intensive Applications for learning more about data processing architecture.
Please share any principles you follow when building reporting pipelines!
Leave a Reply