This is a series of three blog posts on data management. We're deliberately taking an unconventional approach, starting "backwards" by focusing on value generation. While there's inherent value in simply possessing data, I believe the true value is unlocked when users can actively utilize that data.
So we will start with just that: Data applications
The order will be:
- Data applications
- Data platform and management
- Data ingestion
Data Applications
So you have your data in a data platform, what now? Where is the value and how do I extract it?
A data application can be many things, a few things we have come across in the energy sector:
- Power-bi report/dashboard
- Customer facing component covering their month to month cost and usage.
- Cost simulation software
- Inventory of sensors
- Sales recommendations
- Predictions
- Anomaly detection
- Alert and monitoring
One thing these applications all have in common is the involvement of setup and development inherent to building them. They not only leverage data from your data platform but can also require dedicated infrastructure, integrations with other services, and data transformation that extends beyond typical layered structures (like the medallion architecture). Furthermore, they often generate new data that can be valuable to others.
These applications can drive massive cost savings and business value, but if architected poorly, they quickly turn into expensive technical debt. To ensure you can actually deliver these applications without letting costs or complexity spiral out of control, you need to factor them into your data platform investment from day one.
And the most critical part of that investment isn't just the technology, it's the people building on it!
Dev Experience/Effectiveness/Infra
In the modern data landscape, effectively everyone involved in creating and leveraging data solutions, whether they are data engineers, data scientists, or analysts acts as a developer. Because building data applications is complex, the experience of these developers dictates your success. The more time you put into creating a great Developer Experience (DevEx), the more secure and maintainable your applications will be, and the faster you can get them to market. If you want to scale the value of your data, you have to make life easy for the developers extracting it.
Take some time and think of what type of value your data can bring the organisation. Remember the 80/20 rule, don't try to solve every edge case from day one, but ensure your platform easily covers the highest-impact use cases.
A few rules can get you far, these are some that I think are great.
Self Service
Ensuring developers have the autonomy and tools to get things done themselves. Think structured checkpoints or automated 'toll gates' for quality and compliance, not manual 'gatekeeping' that creates unnecessary blockers.
Ownership
Ownership of the thing that is being built. Clearly defined ownership for every project in production guarantees accountability and ensures that each one is actively managed. This includes the crucial tasks of altering ownership when necessary and proactively retiring projects that don't fit organizational needs or provide value any longer.
Golden Path
Build processes and tools that makes it easy to build great things, you shouldn't need to know everything but it should be easy to discover and to do the right thing.
- Infrastructure
- Deployment
- Developer envs
- Getting the data
- Authentication and security
- Best practices
Dog Fooding
Try the things you build. It is easy to say this is how it should be done but hard to know when the process or tools need to change if you never use them.
High Quality
Aming for high quality from day one, through automated testing, observability, and peer reviews, means developers spend less time firefighting production bugs and more time innovating.
Conclusion
Building high-value data applications is the ultimate goal, but as we've seen, it requires a foundation that prioritizes developer experience and scalable infrastructure. You can't build these applications efficiently if your underlying platform is a mess. In the next post of this series, we will take one step further back and explore Data Platform and Management—the engine that makes all these applications possible