Put simply, there are two types of data stacks – those that fuel organisational growth and help your business to succeed, and those that cause it to grind to a halt, causing your business to fail.
Having worked in a range of companies, from 20-person start-ups to large enterprises, I’ve used both. I want to help you make sure you only use the type that make your business succeed.
How to tell if your data stack isn’t working for you
There are some tell-tale signs that your current data setup isn’t working for you. If the following sounds familiar, then you could have a stack that isn’t helping you:
- Engineering data from a simple source is complicated and takes a long time.
- Once the data is engineered, the task is marked as complete yet the people who need it don’t have access to the data.
- The stability of the data stack keeps having issues, for instance errors and failed pipelines that still aren’t resolved.
- The cost of the data stack is spiralling out of control as volumes increase.
If any of the above sound familiar, I have some ideas that may help.
An effective data solution must quickly put the information you need into the hands of the people who need it. The more complicated the data stack, the longer it takes to provide answers to questions.
Simplifying your data system reduces the time needed to complete the end-to-end analytics that are key to generating the data needed, the data that will add value to the business.
You do not need a complicated solution when your company’s database can operate on a desktop computer.
By this I mean, if you have a manageable database, something that can be operated on a regular desktop computer, you don’t need a data solution that can support flights to mars.
I’ve seen organisations build complicated solutions, which at the time far exceeded the organisation’s data volumes and business maturity.
This isn’t future proofing, it’s overcomplicating.
Design overkill actually prevents the data stack from functioning efficiently and providing the necessary analytics at speed, turning it into an overly costly and potentially wasteful and damaging exercise.
What solution should you use?
Data solutions fall into two main camps.
Solution 1 – Built for large volumes of real-time data processing
This is a data stack that has been built to scale to meet the demands of what are usually large companies with equally large data volumes that have real-time data availability demands. It uses the best-in-class technology with the latest tools.
- Who needs this? Very few huge companies that need real-time data at their fingertips for their customers. Think Uber, Amazon and Airbnb. Companies of this type and size are few and far between. Data for these organisations cannot be run and processed on a desktop computer.
- Technologies Used - Their data solutions require a lot of different technologies. The resources needed that are as rare as unicorns, experienced and expensive. You’ll need an experienced Full Stack Engineer who has experience working in Data.
- Cost – Astronomical to build and not sustainable for small and medium businesses to run, from both a resource and cloud-cost perspective.
Solution 2 - Built for speed of analytics with lower volumes of data
This is a simple and elegant solution that consists of 3 items:
- Cloud-based data pipeline orchestration tool
- Data Storage Solution
- Data Reporting and Access Solution
- Who needs this? - Most small and medium sized companies. Where the time taken to deliver an answer can make a huge impact on the company’s growth. This is the optimum solution for any company of a size that means their data can exist on a desktop computer.
- Technologies Used – SQL, Python and a cloud orchestration tool. Experience of these tools is commonly found with many Data Analysts and Data Engineers.
- Cost - Affordable to set up and maintain, and easy to run.
Which solution do I need?
When deciding the solution you need, consider the current size and volumes of your business’ data requirements, and match your solution to this.
It’s not a good idea to go jump straight to Solution 1 before using Solution 2.
Solution 2 will ensure you understand and see first-hand how data can fuel your organisation’s decision making.
I’ve seen Solution 1 used many times in start-ups that caused challenges in how they used their data to drive decision making. Ultimately, it slowed decision making and hindered rather than helped the businesses.
If your business isn’t dependent on real-time data to make business decisions, then Solution 2 is very likely the right solution for you.
What next?
If you’re not sure where your business sits, or if you’ve been advised to go for Solution 1 and aren’t sure if it’s the right thing for you, get in touch. I’m happy to advise on the best way forward for your specific needs.
Email Pratik@indatawetrust.co.uk to discuss your organisations data requirements and how we can help you on your data journey.
Coming up
Look out for my next blog posts, where I’ll be taking you through the skills you need to build our your data team.