Scrum Confessions – Part 1

Agile development and scrum are no-brainer-choices in almost every IT department in Europe. Over the past 2-3 years it became almost a religion within the most tech communities, every executive and every proficient IT professional starts setting the scene by focusing the importance of applying the scrum process properly and effectively for success.

I do think that scrum has changed the way software is built ,  supported the development processes  in a very positive way and it opened a new era of engineering enabling the alignment of business and engineering much better than it used to be.  On the other hand, having gone through some scrum transitions in several working environments and based on my observations from the management balcony until now I know that there exist some typical and chronic problems around scrum that bother the most of the stakeholders.

Now, I don’t mean the problems that are  related to the suboptimal implementation of the scrum process or not understanding the philosophy. Indeed, probably the first thing  you would hear when your project fails, delivers late or actually gets way to expensive to finance would be: “there must be something wrong with your people and the way there are doing scrum”. However, I believe ,scrum process and roles have weak points and fundamental issues causing productivity and quality disadvantages.
I think many executives and computer scientists working with scrum do know that it cannot be efficiency or the people every time.  We know this funny feeling that there might indeed be some stuff fishy and not mature about the beautiful scrum methodology. Somehow, we usually prefer not to talk about these thoughts and experiences. It is an unwritten law not to question the process and always praise the practices thought to us by the gurus of the Scrumentalists. So I decided to write a series of blog articles which will reflect my observations and criticism about scrum framework, its deficits and risks that every IT manager should consider.

Confession 1 : The product owner will always fail

We expect product owners to act like product managers and drive the business while being fully aware of the customer expectations and competitive market out there. Easy said than done, this means that a PO needs to have strong external orientation, time and skills to drive the KPIs and strategic decisions related to the product. They are also need to have project management skills and fully control the product backlog and prioritize tasks for the team, write the stories and be responsible for translating end user requirements in a way that the team can understand and correctly implement. She has to judge operational and emergency issues against new feature development and act as a leader to keep everybody focused and energized. Which actually requires that she has to be technically  quite knowledgable  about the software development technologies, be internally oriented, have at least some engineering skills and do lots of things that is outside the scope of product management or business analysis functions of waterfall days.

There is hardly anyone on earth who can juggle so many balls in so many disciplines. The consequences of such illusional expectations for the companies are really serious and it is a huge problem for many executives. Because we want convincing market analysis about the products we decide to invest in, see the KPIs and reports that enable us to measure success and of course(!) there must be somebody who is accountable for the optimal delivery of new features while fixing the right bugs and giving the necessary support for operations. The bill is mostly charged to the product owner in cases of failure or problems. “There must be something wrong with her” or “we need to improve how the role is defined and understood” or “ it is a product owner who did not understand/do this and that” bla bla bla. It is an endless and useless conversation! I have known many POs with extraordinary skills and amazing performance but the PO defined above does not and will not exist.

So what could be a solution? : Personally I believe mid and large size companies need to separate the business and technical process side of the product management and install a separate set of professionals who are accountable for the product KPIs, market competitiveness, benchmarking and external analysis of opportunities and risks for the business. These people do not have to cuddle with the scrum team on a daily basis and they do not need to know all the user stories and technical details about the product. Call them product managers, business analysts or anything you want. POs on the other hand can focus on the internal and more technical aspects of software development and backlog management; they get the product delivered fulfilling the defined KPIs by her partner with the help of her team.

Confession 2:  Scrum=software and processes without documentation & lots of risks

Agile manifesto says “we value working software over documentation”. What happens in reality is that over 90% of all scrum projects are not or very poorly documented. There is no incentive, recognition, rules or any process for documentation in the framework. What is even worse is that the framework turns developers into professionals who do not value documentation and find the related tasks boring and inferior. And it hurts, it hurts you a lot

  • When you have attrition in your core team and lose the valuable know-how
  • When your services team have no documented and structured data about what peculiarities and operational requirements a new piece of software on production has
  • When a developer needs to maintain the code he has not implemented and it costs way too much for the company
  • When you need an overview of your services or product value chain and you don’t know what processes are mapped to your running software in what way. (This will bother you a lot in case of ERP,SOX or CMMI implementations, mergers, attempts to scale the business)

There have been discussions in the communities about the need of a technical writer in the scrum teams to tackle this problem. My reaction to this as a manager would be “what???!!”. Seriously I don’t get why managers have to be satisfied with the output of what scrum delivers to them and find additional ways of obtaining the product attributes they require for their business.

Management does not have to care what possibilities scrum has. It needs product with proper functionality, the required user manuals and documentation for internal operations, it is part of the order and it needs to come in the package. Instead, a manager will likely to hear comments like “if you need documentation that will decrease out velocity”, “ you have to invest more into this project, much much more” “ we have higher priorities than this” or “it is not part of feature development of our scrum process” from the team.

I believe ,again the problem is not about management’s understanding or the teams. Scrum fails to provide the practices and incentives for defining and delivering the documentation with the required amount of detail and quality to an enterprise. It needs to be tackled and implemented in each environment and become a mandatory part of the software engineering process.(If you think you can commit such a sinn against the scrum religion)

Coming up next soon : Scrum Confessions – Part 2 with issues about scrum master role, software architecture problems and application lifecycle challenges..

Some Thoughts On Testers And Testing Teams

Recently I heard the dilemma of being a tester from a friend: “If you find many bugs and quality flaws , the developers hate you. If you don’t find them you are anyhow useless….”  There must be surely truth to it. It has been a known and old issue for many testing departments not to have the same visibility and recognition as developers within their organization and their projects.

Agile practices brought also many additional questions and challenges to software quality management field in the last years. There has been many discussions questioning the future of testers and QA departments.  It is mainly due to the blurring boundaries between old fashioned roles. The days of DBA, sysadmin, developer and tester siloes are gone, there are various success stories of companies that eliminated the matrix structures and clear scoping of the engineer tasks : “Facebook has no testers..”, “ XYZ Company is doing dev-ops and people have full access to everything”…

Yet the need for teams and individuals who are skilled to perform various complex tasks and willing to be accountable for the quality of the outcome is bigger than ever. Today’s successful teams consist of engineers who are continuously developing their technical mastery and are able to apply the systems thinking to their work at the same time.

So what do you need and do not need as an IT manager? The question is actually not whether you need tests or not. The question is also not whether there should be dedicated engineers who master in testing techniques. The main question should be whether you need a separate testing team or discipline within a certain environment. It is of course possible to produce high quality software without a testing team in many scenarios. However, in many other cases the optimal software development and ALM processes can only be implemented through a separate testing team. There is no one-size-fits-all solution.  Every company, every development organization needs to find its own unique setup for success.

How do you decide?  First of all, take a look at the behaviors and competencies of the engineers you have in the team. And decide which efficiency and what level of testing excellence is required for your product.  Good testers are quality specialists and offer unique advantages to the QA process. They work and think differently than the rest of the engineer team. They are trained to find and report problems.  Naturally, an experienced testing engineer would very likely to have the techniques, personality and experience to find, report and present bugs faster, better and more effective than most of the other development engineers. If you go for a solution with a testing team, you need to make sure that they are more than bug documenters or quality advisors. They need to have certain authority and decision making power over certain project execution situations, spring breakers and change management.  Here the key is that the testers and their lead understand the business priorities, defined quality metrics and the stakeholder needs in the whole system so that they can find the right balance of business needs and risks. Making the right call to fix or backlog a bug, escalating an emergency to halt a sprint, getting other decision makers’ buy-in in various critical situations are vital skills for testing professionals. So if you decide for a tester team, get good people on board, empower them and support them get the credibility and trust from their peers and cross-functional teams.

If your decision is go for a one-team approach without a testing team, make the quality metrics top priority for everybody and hold certain individuals accountable in the scrum teams. Ensure that the product owner is also held accountable for overall quality of the outcome. Train your engineers to know/understand how critical each error is and what types of errors should be avoided, develop strategies and integrate in to the development process. A developer is focused to do things right and get the code out whereas the testing-mode requires to look at the “bad” in the system in a systematical way,  to focus on to find out what is wrong and be determined to break the system.  It is a conflict for a developer to try to write solid, quality code and to fearlessly attempt to break and take things apart almost in a destructive way while testing. Give your teams opportunities to practice and reflect on different modes of working in a software project; improve the approach with their feedback after each sprint if necessary.

No matter what your decision is, best practices and agreed processes may help you to find your way to plan and get your software out with the desired quality. And you have to invest a lot of energy and priority in automation: do not end any meeting or project regarding testing without having talked about the status of test automation efforts. But at the end of the day, the main impact on the quality of the software does not originate from processes or methods. What matters by far the most is your people, the team, what capabilities they have and how they work together. The development and testing leads should motivate the engineers and get their commitment for quality by clear messaging about the importance of quality, impact of testing to the project and the company.  Track and recognize the engineers’ contributions within the community, make sure you have their support for the project and there is always someone to ask for help and to discuss about issues/problems.

Quality must be everybody’s business involved in the whole value chain starting from user request to the end of life of the software. And in many organizations there is a separate testing and QA team to drive and support the quality management discipline in a professional manner. But the bottom-line is that every team member, every manager and developer carries the responsibility of the quality of the product they ship, with all its success and other consequences. If this awareness and corresponding culture can be created, it would be much easier to get the support and creativity to find the processes and organizational setup that work for your teams.

Recommended reading :

Beatiful Testing, Does the existence of seperate QA Team damage quality?, end of the tester, Systems Thinking

About The Human Factor In IT Services

Services is a very broad sector, it can be basically defined as everything produced except manufacturing and farming. IT services is one of the most complex and biggest subsets of the industry. Defining meaningful categories , standards and controls is pretty difficult given the diversity of the services being offered and the factors influencing them.

So what is the most challenging parameter in the services equations? It is the human factor. In no other industry in the world, the human is so important and critical for the effectiveness and efficiency of the business. No other industry depends so much on the people’s education, experience and behaviour (and ethics). This fact can be a blessing or a curse for a services organization; performance evaluation and management is much more difficult than any manufacturing sector. You can never standardize and have a control on the output as you do with processes and machines, the same “input” will not always produce the same results with services delivered by humans, your business will always have higher variability of  risks.

If the conditions or the environment of the human changes, the criteria for evaluation and success will change too.Hence the service organization goals defined are also subject to change at all times, leading to more uncertainty. I personally think that the longest time horizon that can realistically be set and controlled in IT services is  around 1 year, for greater enterprises maybe some more. Any objective defined for 3 or more years in advance makes as much sense as predicting the weather for that point in the future.

For the purchaser, the huge scope of services and the dependencies on the human skills and capabilities increase the risk of the services spending in many ways. How can they assess the value and quality of the services being delivered? Standardization and quality control may be difficult or not feasible at all. The diversity and dynamic nature of the sector usually require the customization of the service to fit to the exact needs of the purchasing company;  a standardized approach is not always the right call for the receivers of the service. Besides, it is usually not the purchasing department that can determine if the service was carried out properly. And one needs to purchase a certain type of service at least once to make a solid evaluation of the buying decision.

The services company has the responsibility to consider all the intangible and human related factors as well as the challenges of its customers (i.e purcahser) to  reach and maintain a sustainable success it its industry. Given the high level of risk and uncertainty for both sides, the most important task is ensuring that the all the relevant information is communicated efficiently, especially when it comes to the requirements, the service cost and bidirectional feedback. Another vital task is the management of the skills and capabilities of the organisation and each individual involved. Investment in the processes, assets,staff and synergies is the “only” way to differentiate a service organisation from the others.

Even the investments are made and an optimal portfolio of skills  is built, services organisations are exposed to risks of uncertainties and intangibility. The management of these risks should be done via demand management. One cannot create an inventory of services to serve the customers in times of higher demand. There is always an uncertainity that the demand may be below or above the existing capacity. (and it is usually so). Demand can experience a huge change overnight whereas building services capacity with skilled human labor is a time-consuming and expensive investment. Meeting and creating the demand is the key to survival in services. But how? Well, if I knew the exact answer, I would probably be an entrepreneur now. 🙂 On the other hand, it is for sure very important to build a good CRM system to understand the chaninging needs and behaviours of customers, build a relationship based on trust and opennes. Use of IT frameworks and service delivery methods also helps both the purchaser and the service provider to agree on expectations and be able to fix short to mid term plans. For instance; SLAs, integration of frameworks like ITIL or MOF, ability to comply to SOX or TQM requirements make life easier to manage scope and quality in this huge heterogenous landscape.

Services is one of the strongest rising trends of our century and it is definetely not a bubble. There is no single industry where the services spending is not continously growing. Especially in IT, the purchasing and delivery of services is quite complex, diverse and difficult to manage. Human factor is the driving force but yet it introduces intangibility, uncertainity and variability to the business. To thrive in the hugely comptetive market, services organisations have to invest a significant amount in the processes and their people, be flexible and strong enough to survive the changing environements of the service domain and fluctiating demand. They build excellent relationships to customers and know their needs and develop the capacity to forecast the demand. In times of fluctiating demand there has to be a plan B to overcome the challanges financially and morally in the organisation. This might include many organisational skills like hiring talent fast, getting the workforce provide overtime without loss of quality, building financial buffers to bridge low demand periods while keeping the capacity. And finally, a services organisation must have the ability to acquire new skills and offer new services continously, the comfort zone of services for cash cows should not be the only domain they show their presence. There is no sustainable growth possible  in IT services without  intelligent risk taking.

(Special thanks to my former MBA classmates Sara Al Murad, Annette Barth, Gianmarco Dionisio and Peter Ippy who worked with me on service chains & purchasing and inspired me with our findings and discussions)

%d bloggers like this: