Please tell me if you have ever worked for a company that collected ANY type of data and not heard the term “System of Record”. If that term is new to you, then make sure you get out from underneath that rock!
This week’s reading brought about an interesting topic regarding overlapping systems and/or applications. Let me bring you to the most difficult and daunting task we had to take on last week while trying to finalize our EOY adoption goals for our SaaS product. Coming up with baseline numbers the first question someone always asks is, what and where is this single source of truth for that data? To be honest, if you are part of a seasoned and complex organization, it does not exist. More often that not, an enterprise has adopted two, three, and even six database management systems. This makes it all the more complex to find the information you are looking for.
Take our EOY product adoption goals for example:
- Task 1: find out how many customers we have (active)
- Task 2: find out how many customers that are active have our product (active)
- Task 3: find out how many customers that are active and have our product active and in what region
- Task 4: find out how many customers are active that do not have our product (active)
- Task 5: find out how many customers are active that have never had our product
…and so forth, so you get the idea. The difficulty lies in attempting to gather and cross-reference this data. Big data is complex for a reason right? Gathering and then applying requires plenty of resources and knowledge!
During this exercise, we went through the following systems to glean information on each task:
- Adobe Analytics
- Core system reports
- Manual excel files
…and probably two other programs that I cannot even remember the name. You tell me, how long do you think it would take to define a baseline having to decipher and dedupe records from each of the above sources? What is more, how would you define the single source of record or the majority of record from those nine sources? Would the analytics information be the master record and all other sources would need to be matched up to that, or would Versus be the master record? We went in circles for two weeks attempting to come to an agreement for an initial gathering of information.
But wait, it did not stop there! At one point we had agreed Vertica would be the master data file, but upon opening the same requirements in Insights, realized that Insights had five more fields of relevant data that Vertica did not. What?! Should we have set up a secondary master file of truth? How about grouping the most relevant sources with the data they provided? Oh! This is so complicated – just give me one database management tool to work from!
To be frank, there is no right or wrong answer. After a two week deliberation, our enterprise agreed that the answer had to be one that everyone understood and agreed to use in relation to reporting progress on EOY adoption goals. Our decision was also based largely in part on the “derived from” technique. We ended up identifying Oracle as our data system of record since the common characteristics of the other sources were that their data was derived from the information in Oracle. If we needed to confirm the information in Oracle with employees, we would use Salesforce as our CRM system of record. We ended up with a viable solution and had to analyze the data from each source to come to a thoughtful decision.
Below is a process map from this week’s article. I would say we followed this aside from limiting the use of our other sources.
When you are a part of an enterprise that large, it is difficult to limit or just stop the use of other applications or systems. Ideally, they are there for a reason. Although some capabilities may overlap, most of the time these systems and applications all have new contributing capabilities that can be used for different purposes.
James, G. (2005, October 25) Use Enterprise Architecture to Control Overlapping Applications. (ID Number: G00131279). Gartner Database.
Faculty Development and Instructional Design Center. (2005). Responsible Conduct in Data Management: Data Analysis. Retrieved from https://ori.hhs.gov/education/products/n_illinois_u/datamanagement/datopic.html