Consulting, Technology Ecosystem and Setting Expectations

Question

8 years back when I started my consulting journey, there were 2 major technology ecosystems to deal with. Depending on the problem evaluated it against business, application, development and operational considerations, was able to provide a solution which worked for most of my customers.

4 years back, things started changing. I was putting together the technology road map for the SBU I was working for. There were more than 2 technology ecosystems to deal with. One of the major question I had to answer at that point was if I choose a technology today and start developing, Can this survive the test of time? Can we avoid rewrite at least for next 5 years?

Fast forward to 2016, technology is changing so fast, 5 years seems to be a very long time. If you are starting to develop a product/framework/platform, you need to be prepared at least one or more of your components may change as you start developing and your ecosystem should be ready for that. This requires a different mindset with the business teams, architecture groups, development teams and your ops teams. As an architecture consultant, it is very important to communicate and bring everyone to same page. Having the expectations set right and being ready will avoid the frustrations towards the later part of the journey!

Happy Learning!

Data Infrastructure, Data Pipeline and Analytics – Reading List – Sep 12, 2016

Three incremental, manageable steps to building a “data first” data lake
Applications have always dictated the data. That has made sense historically, and to some extent, continues to be the case. But an “applications first” approach creates data silos that are causing operational problems and preventing organizations from getting the full value from their business intelligence initiatives.
http://www.networkworld.com/article/3098937/analytics/three-incremental-manageable-steps-to-building-a-data-first-data-lake.html

Azure SQL Data Warehouse: Introduction
Azure SQL Data Warehouse is a fully-managed and scalable cloud service.
https://www.simple-talk.com/cloud/azure-sql-data-warehouse/

The Informed Data Lake: Beyond Metadata
Historically, the volume and extent of data that an enterprise could store, assemble, analyze and act upon exceeded the capacity of their computing resources and was too expensive. The solution was to model some extract of a portion of the available data into a data model or schema, presupposing what was “important,” and then fit the incoming data into that structure.
https://hiredbrains.wordpress.com/2016/05/13/the-informed-data-lake-beyond-metadata/

Real Time Streaming with Spring xd, Apache Geode (Gemfire), and Greenplum
Spring xd is a unified, distributed, and extensible service for data ingestion, real time analytics, batch processing, and data export.
http://zdatainc.com/2016/01/real-time-streaming-with-spring-xd-apache-geode-gemfire-and-greenplum-earthquake-data-demo/

Data Orchestration using Hortonworks DataFlow (HDF)
Hortonworks Dataflow (HDF), powered by Apache NiFi, is the first integrated platform that solves the real time complexity and challenges of collecting and transporting data from a multitude of sources be they big or small, fast or slow, always connected or intermittently available
http://zdatainc.com/2016/02/hello-nifi-data-orchestration-using-hortonworks-dataflow-hdf/

Happy Learning!

Data Infrastructure, Data Pipeline and Analytics – Useful Links – September, 2016

Some of the Interesting links i have read in the last couple of weeks around Data, In-Memory Databases, Pipeline Development and Analytics.
In Search of Database Nirvana
An excellent post providing an in-depth look at the possibilities and the challenges for companies that long for a single query engine to rule them all.
https://www.oreilly.com/ideas/in-search-of-database-nirvana
http://www.slideshare.net/RohitJain0813/in-search-of-database-nirvana-the-challenges-of-delivering-hybrid-transactionanalytical-processing
Aerospike Vs Cassandra Comparison
Comparison on Aerospike with Apache Cassandra. Cassandra is a columnar NoSQL database that is great for ingesting and analyzing hundreds of terabytes of data stored on rotational disks. Aerospike is an in-memory, NoSQL database, a key-value store that can run purely in RAM and is also optimized for storing data in Flash (SSDs).
http://www.aerospike.com/when-to-use-aerospike-vs-cassandra/
An overview of Apache Streaming Technologies
A very good comparison comparing technologies around simple event processors, stream processors, and complex event processors.
https://databaseline.wordpress.com/2016/03/12/an-overview-of-apache-streaming-technologies/
Flow based Programming
Flow-Based Programming defines applications using the metaphor of a “data factory”. It views an application not as a single, sequential process, which starts at a point in time, and then does one thing at a time until it is finished, but as a network of asynchronous processes communicating by means of streams of structured data chunks, called “information packets” (IPs).
http://www.jpaulmorrison.com/fbp/introduction.html
Hadoop Deployment Cheat Sheet
If you are using, or planning to use the Hadoop framework for big data and Business Intelligence (BI) this document can help you navigate some of the technology and terminology, and guide you in setting up and configuring the system.
http://jethro.io/hadoop-deployment-cheat-sheet/
Amazon Redshift for Custom Analytics
Experience summary on building Custom Analytics on top of Redshift
https://www.alooma.com/blog/custom-analytics-amazon-redshift
Building Analytics at 500px
Experience summary on how they have built the ecosystem
https://medium.com/@samson_hu/building-analytics-at-500px-92e9a7005c83#.pdkk7xrui
How Artificial Intelligence Will Kickstart the Internet of Things?
IoT will produce a tsunami of big data, with the rapid expansion of devices and sensors connected to the Internet of Things continues, the sheer volume of data being created by them will increase to an astronomical level. This data will hold extremely valuable insights into what’s working well or what’s not.
https://datafloq.com/read/Artificial-Intelligence-Kickstart-Internet-Things/1776

Happy Learning!

AI, Deep Learning, Machine Learning and NLP

Watched this presentation by Frank Chen (Andreessen Horowitz – Tech Topics) today. An Excellent overview about AI, Deep Learning, Neural Networks and Machine Learning.

Frank talked about how one person has built a self driving car using the open source ecosystem available right now. Very inspiring.

Must Watch…

On a side note: May be it is time, we use these technologies to solve the traffic problems at Belandur and Agara, which can make atleast lives of 100 thousand people better and productive, save the $ spent in burning fuel!

Happy Learning!

Is it time to start looking out for a new job?

Career Decision - Next ExitI recently happened to read this post from the Spend Matters blog. Blog Post talks about three questions one should ask themselves regularly to decide whether it is time to look for a new job.

1. Do you feel you are personally learning and developing in your current role? Are you gaining new skills, developing your capability, becoming more expert – with the caveat that it really helps if these are in some sense transferable skills and knowledge.

2. Are you progressing in your career, moving forward from a seniority and / or financial point of view, with a trajectory that is heading in the right direction? (This assumes you have some ambition – not everyone does have, I realize).

3. Do you enjoy what you are doing – is it a good place to work, with decent “hygiene factors” and a bit more – which might be anything from pleasant colleagues and working environment, a commute that is manageable, technology that works, social events …

Excellent set of questions. You can find the original post here. 

This one is my favorite on this topic. Probably tied to all the 3 questions above.

To be happy and be fulfilled at work, people want to feel they are advancing, getting things done, and making an impact. But it’s not enough to simply to receive a pat on the back and a word of encouragement. Rather, we respond much more positively to feedback from the work itself. When we have achieved a goal like closing a sale, writing code that passes the test harness and is pushed to production, releasing a new feature that a million users touch every day, our happiness at work blooms.

Source: Managing for Progress

An Excellent book on this topic : The Progress Principle

Teresa Amabile’s talk at Google

In the end… it is not about the nice office buildings, additional perks etc. It is about the job itself. It is about the people you interact with on a daily basis and deal with.

The Successful demos at the end of every 2 weeks, Production Releases, a good solution to a complex problem, providing a solution using a new technology stack, number of first calls to potential prospects, a good sales pitch to a new prospect, a new customer win on a regular basis, hiring a good candidate, coming up with a new product offering… (A few items from my list). All of these contribute to the small wins part. If you dont have them as part of your day to day job or you dont see the number of small wins…. may be it is time!

Happy Learning!

Scaling data operations with in-memory OLTP

Data has become the center of our universe in modern digital world. Applications are designed to store and collect more and more data. Companies are looking to integrate and analyse the data to generate insights and take actions.

Data is a precious thing and will last longer than the systems themselves ~ Tim Berners-Lee

Can an existing relational database scale with high ingestion rates, improved read performance?Database

In-Memory OLTP seems to be the direction forward. This is considering your existing technology investments. Of course if the company is open to change technology there would be more options.

Found couple of very good articles posts related to SQL Server in-memory OLTP. Looks like SQL Server 2016 has fixes to most of the issues with in-memory OLTP.

I just think it is an amazing technology and if we can use it in the right way, will definitely yield great results for your customers.

Introducing SQL Server In-Memory OLTP
https://msdn.microsoft.com/en-in/library/dn133186.aspx
https://www.simple-talk.com/sql/learn-sql-server/introducing-sql-server-in-memory-oltp/
http://blog.sqlauthority.com/2014/08/08/sql-server-introduction-to-sql-server-2014-in-memory-oltp/

The Use Cases for SQL Server 2014 In-Memory OLTP
http://sqlturbo.com/the-use-cases-for-sql-server-2014-in-memory-oltp/

SQL Server In-Memory OLTP Internals Overview
https://msdn.microsoft.com/en-us/library/dn720242.aspx

The Promise – and the Pitfalls – of In-Memory OLTP
https://www.simple-talk.com/sql/performance/the-promise—and-the-pitfalls—of-in-memory-oltp/
https://msdn.microsoft.com/en-us/library/dn246937.aspx

SQL Server 2016 : In-Memory OLTP Enhancements
http://sqlperformance.com/2015/11/sql-server-2016/in-memory-oltp-enhancements

Speeding up Business Analytics Using In-Memory Technology
https://blogs.technet.microsoft.com/dataplatforminsider/2015/12/08/speeding-up-business-analytics-using-in-memory-technology/

Dynamic Data Masking in SQL Server 2016
http://www.codeproject.com/Articles/1084808/Dynamic-Data-Masking-in-SQL-Server
https://blogs.technet.microsoft.com/dataplatforminsider/2016/01/25/use-dynamic-data-masking-to-obfuscate-your-sensitive-data/

Happy Learning!

Software Architecture, Customer Success

Happened to Watch couple of good videos last week on Software Architecture, Design and Customer Success.

How the World Wide Web just happened – Tim Berners-Lee
https://www.youtube.com/watch?v=yF5-6AcohQw
Great Session. Talks about the importance of being in the right place and the right time.

Mary Poppendieck (Poppendieck.LLC) – The New New Software Development Game: Containers, Micro Services
http://m.ustream.tv/recorded/61477219?rmalang=de_DE
Complexity grows non-linearly with Software size. Software size continues to grow so software complexity will continue to grow even faster. She explains what can we do about the complexity?

A summary of this talk is available here
http://highscalability.com/blog/2015/4/27/how-can-we-build-better-complex-systems-containers-microserv.html

Zen and the art of Customer Relationships
https://www.youtube.com/watch?v=G_2UP4-J7Vc
I loved the Zen and the Art of Customer Relationships presentation from Zen Desk. Awesome Presentation!
Pointers for building long lasting relationships

  1. Don’t overestimate your importance in your customers life
  2. Consider the entire customer experience
  3. Recognize the right relationships and adapt
  4. Be something actual humans can relate to
  5. Be Transparent
  6. Empower your best people to do what’s best
  7. Put a face to your customers

Framework to Build a Killer Customer Success Scorecard
https://www.youtube.com/watch?v=lhx06h8RZ3Q
Another Fantastic presentation from the trenches. A good overview around how to define Customer Success and what are the metrics to monitor (Customer, Financial, Practice and Inter-team)

Building the Customer Success Management Team
https://www.youtube.com/watch?v=XIx5HhfG56w
Happy Learning!