Management

I realised recently that it is many years since I saw what used to be called a Run Book or System Log Book. This was a file – as in a plastic binder – with sheets of paper or printouts in it about a given system. Yes, this was a while back. It would often also have diagrams {occasionally drawn by hand on scraps of paper – so that would be the database ERD then}, hand-written notes and often the printed stuff would have scribbles against it.

{BTW I asked a colleague if he remembered these and when he said he did, what he used to call them – “err, documentation???”. Lol}

In my last post touching on my case for Data Engineers, my friend Greg Rahn provided a humorous quote about data from Andy Todd:

“Data matures like wine, applications like fish”

Which, near as I can tell, came from an Open Source Developer’s Conference in Brisbane, 2009 at which Andy talked about, of all things, “Change in Database Schemas and Source Code“.

I’ve dropped Andy an email to see if his presentation is online anywhere, since it touches the topic that is near and dear to my heart.

In this post, though, I’d like to address some of the humor behind the quote — the implication that data gets better as it ages, while applications get worse (and start to smell like stinky fish).

Has “database” become a dirty word within your organization lately? If you’re someone who has been a data technologist for the better part of your career, you may be wondering why the technologies you work with everyday seem to be acquiring such a bad rap. From NoSQL to No DB the current influx of brogrammers seem to take extreme pride in describing how they’re able to write code while avoiding any kind of database technology whatsoever.

The impetus for this post actually started with something I read on the ‘Net the other day about Command Query Responsibility Segregation (CQRS), and how I was initially excited about the concept.

Martin Fowler has a nice, gentle introduction the topic here.

Last week I read an interesting article about how cloud computing is changing the role of the enterprise architect and it got me thinking about the bad rap many architects are getting in the brave new agile, cloud, big data world.

From what I’ve been reading, there’s been a bit of a straw man argument going on — enterprise architects are often described as uber-control freaks who attempt to dictate software architectures in a repressive way to implementation teams. Mention the term “reference architecture” and you’ll often raise the hackles of the new developer-led world.

To be sure, there are many enterprise architects who match that description. And they’re the ones who give architects a bad name, just like undisciplined developers can give agile a bad rap too.

You know how it goes. You get a call/mail/text with something along the lines of “I need to know all the details of customer orders placed on Tuesday 7th by customers based in Botswana – and I need it ASAP, by end of play today at the latest”. So you skip lunch, drop that task you have been trying to get around to doing all week and work out how to resolve the issue that has just been dropped on you. It takes a lot of effort and you finally get it sorted out around an hour after you told your girlfriend/boyfriend/cat you would be leaving the office that day – and mail it off to the requestor. You might even call them to let them know it is done, but oddly they don’t answer.

Next day, you see the guy who wanted this urgent request and ask if it was what they wanted “Oh, I have not looked at it yet – but thanks for doing it.”

I was involved in a discussion recently with Debra Lilley which version of Oracle to use. You can see her blog about it here (and she would love any further feedback from others). Oracle now has a policy that it will release the quarterly PSUs for a given point release for 12 months once that point release is superseded. ie once 11.2.0.3 came out, Oracle will only guarantee to provide PSUs for 11.2.0.2 for 12 months. See “My Oracle Support” note ID 742060.1. However, an older Terminal release such as 11.1.0.7 is not superseded and is supported until 2015 – and will get the quarterly PSU updates. This left the customer with an issue. Should they start doing their development on the latest and theoretically greatest version of Oracle and be forced to do a point upgrade “soon” to keep getting the PSUs, or use an older version of Oracle and avoid the need to upgrade?

If you manage people, it helps if they don’t dislike you. Sadly, this can be the default starting opinion for some people who have never been managers (we all know someone who “has never had a decent manager, they are all bloody idiots”). Frozen dairy products might be a route to easing this situation.

I mention this as we in the UK are having an unusually warm start to autumn, an Indian Summer as we call it. I used to work in a place that had an on-site cafe and a nice area outside to sit. If the weather was warm and I knew my team was not facing some crisis, I would occasionally pop my head around the door and announce “Team Ice-Cream!”. Anyone who wished could come down with me and I would buy them an ice-cream of their choice and we would sit out in the sun for 15 minutes and talk rubbish.

I’ve done similar in other situations. Taking the guys to the pub is the obvious one and it usually is appreciated, but in some ways it is less successful. I think this is because people will come to the pub because they want a pint and will put up with any idiot willing to provide a pint of Fosters (why is it so many of the “all managers are idiots” brigade drink some brand of nasty lager?). People will come for a tea/coffee or an ice-cream only if they are at least ambivalent to the provider. If you really dislike someone, who cares about an ice-cream? The serious malcontents will stay away and this helps identify people who really are not happy with you {so you can beat them mercilessly of course – or, if you’ve progressed beyond the school-yard, put some thought into why they are unhappy and what to do about it}.

By the way, this is very different to everyone going to the pub/restaurant in the evening and spending hours telling people what “you really think” and trying to impress Jessica the new trainee/intern. Such team building events generally need much more planning.

It’s a cheap bribe, should you resort to such shallow tactics to make people like you? Well, it’s only a cheap bribe as I said above. The trick to it is that it has to be {almost} spontaneous, such that the team are not expecting it, and not all the time. I’m not sure the teams I have done this for have always appreciated that I made special effort to do this either after a hard period of work or when there had been some malcontent within the team (people fall out, it impacts the rest of the team). The way I look at it, it also has to be a team thing and not an individual thing as the sitting around talking rubbish is a key part to the team being a team. Even if it is just over a cup of nasty coffee in the basement – that particular company’s canteen was not the best.

Oh, I should mention that I have access to a wife that makes wonderful cakes. Left-over cake is a brilliant “team ice-cream” substitute, it is both “cheap” so not a bribe but also appreciated as someone put effort in. My wife in this case. I Never claim I made the cake. well, not often.

TeddyBear Picnic Cake

That’s the carrot. What about the stick? When it comes down to it, you are there to guide the team and the individuals in it and get the best you can out of them. Not being disliked is important but you are not there to be their friend either. If someone transgresses you need to correct them.

In my opinion one of the very worst things a manager can do is dress down a member of their staff in public. That is not correcting them, that is either an attempt to humiliate them or an attempt by the boss to scape-goat the blame to a subordinate. Neither is morally correct and both are highly likely to engender considerable dislike or even hatred.

I distinctly remember one situation where I was in a team meeting and the boss’s managers came in and wanted to know why a recent change had gone so badly wrong. The manager’s response was immediate, he picked one of the team and said something like “It was him, he didn’t test the change properly”. It was so obvious that the sub-text was “it was not my fault”. In reality the sacrificed staff member was not at fault but the boss sure as heck was. A manager gets paid more as a boss and part of the reason is that you take both the credit and the blame for your team’s efforts. This action by that boss did not make us scared of failing and thus work harder, it made us distrust the man and demoralised us.

Sadly it is something I’ve seen a lot over the years and never by what I would call a good manager. I just don’t understand why these people think a public dressing down is going to inspire the target or the audience to work more effectively.

If I’m in the situation where, in a meeting or discussion, it becomes obvious one of my guys has screwed up we discuss how to sort it out as a team. Then after the meeting, the transgressor and I have a private conversation. This has several benefits:

• I am not publicly humiliating them or scoring points in front of a crowd.
• Neither of us is playing to the crowd and so are more likely to be honest.
• Things can be said that stay private. I’ve had team members mess things up because they have more important issues on their mind that they are uncomfortable with the team knowing about. I’ve had to tell a guy this is chance #last and the next step is disciplinary.
• This never happens, but there is a very small theoretical chance I could have misunderstood and, in fact, it’s my fault. You look a right idiot if you attempt to dress someone down in public and it turns out to be you.

As I said, that last point has never happened to me {yes, this is an outrageous lie }. I’ve experienced that last point from the other side as well. In a large meeting I had a board member pushing me as to why we had not finished a project on the date I promised. I kept giving vague answers about “other things coming up” and it would be done by a new, given date. She would not let it go though so eventually I had to say “It is late because you told me to do other stuff as top priority, I raised this project and you told me to delay it. So it is late because you changed the priorities. That would make it your responsibility.” She was very angry but it had been her choice to do this publicly.

All this boils down to – Reward the team in public. Chastise the individual in private.

During conference season, it can be a challenge to come up with abstracts that you can feel passionate about, while making sure to craft them in a way that is both attractive to selection committees and the audience you feel like you want to reach. I often find that tri-purpose (satisfying myself, a committee, and the potential audience) to be daunting and occasionally conflicting — leading to abstract paralysis.

Starting today, I’m going to work harder at it. If you’ve been to any of my presentations in the recent past, you know that I like to spend more time on what I think are technical “culture” issues rather than examples of how to implement or interpret the technical features of the latest software release. It’s an area that I’m passionate about, and it’s one that I feel is drastically underrepresented and underserved at most technical conferences.

The biggest challenge I have with those kinds of presentations is making them selectable and attractive — for the topics mostly concern our ability to collaborate and communicate effectively in support of our business and mission objectives. And in that case, we all feel (myself included) like we’re from Lake Wobegon.

To me, no where is this more apparent than in the discussions about the Agile movement in software development, testing and production operations. Fellow Oak Table member Martin Widlake has some excellent examples of these issues in his 2 recent blog posts on the subject:

“Friday Philosophy – Why Doesn’t Agile Work?” and “In Defense of Agile Development (and their Ilk)”

(I especially like “Ilk”)

In a small, forgotten corner of the Internet, I belong to a Yahoo! Group (yes, they still exist!) on Agile Databases, which has as its description:

Discussion about Database management with regards to Extreme Programming practices and principles.

You can visit the group here.

In a recent discussion, there was a post from Scott Ambler that I found myself violently agreeing with:

A question was asked about coordinating and scheduling changes made by database and ETL teams with the development teams in order to reduce confusion and churn during development.

Question / Comment: While one or more code iterations are taking place in parallel, the data design and ETL are working on their iteration of the db schema and data, which will be consumed by later code iterations.

Scott’s Comment / Answer: Better yet, this could occur in a “whole team” manner where data-experienced people are embedded in the actual team.  This can improve productivity by reducing overall overhead.  Unfortunately this can be difficult in many companies due to the organizational complexities resulting from the cultural impedance mismatch between data and development professionals.

(Emphasis mine)

I feel like I’ve have the privilege of working in places where those organizational complexities and cultural impedance mismatches were overcome and I’d love to talk about what I think made that happen.

Now just to write some compelling abstracts on the subject — ideas welcome!

In my previous post I asked the question “why doesn’t Agile work?”. I’m not sure the nuance of the question came over correctly.

I’d just like to highlight that the question I asked was “Why does agile not work”. It was not “Why is Agile rubbish“. I’ve said a few times in the past couple of weeks that I like the ideology of Agile and I am (and have been for years and years) a strong proponent of prototyping, cyclic development, test driven design and many other things that are part of the Agile or XP methodologies.

That distinction in the title is a really important distinction and one I’d hoped I’d made clear in my post. Looking back at my post though, I think it is clear I failed . I highlighted reasons why I think Agile does not work and in my head I was thinking “if we avoid these, Agile could work” – but when you write something down it does not matter what is in your head if it does not reach the paper.

I’m actually frustrated that in the last few years I have not seen Agile really succeed and also that this must be the normal situation, going on the response you get when the topic of Agile comes up with fellow technicians and comments on my own blog.

However, on that post about Agile two people who’s opinion I deeply respect came back at me to say “Agile does work!”. Cary Millsap, who many of you will have heard of as the “Method R” guy and the person behind Oracle Flexible Architecture. And Mike Cox, who most of you won’t have heard of but Mike taught me a lot about sensible development back in the 90′s. He’s one of the best developers I have ever had the pleasure of working with and I know he has had great success with Agile and RED. I’m not sure if they read my post as “Agile is Rubbish” or they are, like me, simply frustrated that it can work but so often does not.

So I’ve been thinking about this a lot this weekend and I was helped by Cary’s paper on the topic that he mentioned in his comment. I’d highly recommend downloading it as it is an excellent description of not only why Agile can help but describes how and some of the pitfalls {I’d started my own post on that, but go read Cary’s}. I should add, you can see Cary present his case for Agile at the UKOUG conference this year.

So where does this bring me to? Well, I think “Is Agile good or bad” has become almost an “IT religion” topic, people love it or loath it and it is based on what they have seen of the methodology in real life. No, that’s wrong, it is based on what they have seen that has been labelled with that methodology in real life. Or worse, it is based on anecdotal opinion of those around them. The thing is, if you look at what XP is supposed to consist of or what Agile Programming is supposed to consist of, most of us would agree that a great deal of it makes sense in many situations. I’d disagree with some of the details in Cary’s paper but overall I’m in strong agreement. Sadly, What Agile and XP is supposed to be is not well matched by what you see on the ground in most cases. So even if these methodologies are right for the situation, what has been implemented is not the methodology but probably more a slap-dash process that simply jettisons documentation, design and proper testing. This whole thread sprung from my lamenting the demise of database design and several of the comments highlighted that the introduction of Agile seemed to equate, at least in part, with the demise of design. As MIke and Cary say, and as I think anyone who has successfully utilized Agile would say, Design is an integral part of Agile and XP methodology.

Agile can and does work. But many things can and do work, such as taking regular exercise to keep healthy or regularly maintaining your house to keep it weathertight. Like Agile, both take effort but the overall benefit is greater than the cost. And like Agile, do it wrong and you can make things worse. If your window frames are starting to rot and you just slap a new layer of top-coat on them all you will do is seal in the damp and rot and hide the problem – until the glass falls out. Going for a regular 5 mile run is good for you – but not if you are 10 stone (60KG) overweight and have not run in years. A 5 mile run is also not a good idea if you want to be a long-jumper. Right training (methodology) for the right aim. Also, just like keeping healthy, house maintenance or anything that takes effort but works, proponents tend towards extremism – probably as a reaction to the constant {perceived} pig-headedness of critics or the failure of people to just do what now seems so sensible to them {think reformed smokers}. I’ll have to buy Cary and Mike pints to make up for that jibe now, and promise them it was not aimed at them personally…

Sadly, the reality is, Agile does not work 90% of the time it is tried. So, does that mean Agile is actually rubbish? Or at least, not fit for purpose, because many companies are not able to use it? Companies are there to achieve something and the IT systems are part of achieving that something. If Agile cannot aid that IT department then Agile is the wrong way for that department and company.

*sigh* I’ve gone on and on about this and still not got to my own main point, which is this.

- Can we identify reasons for Agile and XP Failing.
- Having identified the Reasons, can we fix them in simple ways?
- Can we create some simple guidelines as to when a project should be more Agile and when it should be more Up-Front design.

I’d love to know people’s opinions on those three points above.

Why doesn’t Agile Development Methodology seem to work?

I’m going say right here at the start that I like much of what is in Agile, for many, many years I’ve used aspects of Rapid Application Development {which Agile seems to have borrowed extensively from} to great success. However, after my post last week on database design, many of the comments were quite negative about Agile – and I had not even mentioned it in my post!

To nail my flag to the post though, I have not seen an Agile-managed project yet that gave me confidence that Agile itself was really helping to produce a better product, a product more quickly and most certainly not a final system that was going to be easy to maintain. Bring up the topic of Agile with other experienced IT people and I would estimate 90% of the feedback is negative.

That last point about ongoing maintenance of the system is the killer one for me. On the last few projects I have been on where the culture was Agile-fixated I just constantly had this little voice in my head going:

“How is anyone going to know why you did that in six months? You’ve just bolted that onto the side of the design like a kludge and it really is a kludge. When you just said in the standup meeting that we will address that issue ‘later’, is that the same “later” that accounts for the other half-dozen issues that seem to have been forgotten?”.

From what I can determine after the fact, that voice turns out to be reason screaming out against insanity. A major reason Agile fails is that it is implemented in a way that has no consideration for post-implementation.

Agile, as it is often implemented, is all about a headlong rush to get the job done super-quick. Ignore all distractions, work harder, be completely focused and be smarter. It really does seem to be the attitude by those who impose Agile that by being Agile your staff will magically come up with more innovative solutions and will adapt to any change in requirements just because they work under an agile methodology. Go Agile, increase their IQ by 10 points and their work capacity by 25%. Well, it doesn’t work like that. Some people can in fact think on their feet and pull solutions out of thin air, but they can do that irrespective of the methodology. People who are more completer-finishers, who need a while to change direction but boy do they produce good stuff, have you just demoralized and hamstrung them?Agile does not suit the way all people work and to succeed those people it does not suit need to be considered.

The other thing that seems to be a constant theme under Agile is utterly knackered {sorry, UK slang, knackered means tired, worn out and a bit broken} staff. Every scrum is a mad panic to shove it all out of the door and people stop doing other things to cope. Like helping outside the group or keeping an eye on that dodgy process they just adopted as it needed doing. Agile fails when it is used to beat up team. Also, I believe Agile fails when those ‘distractions’ are ignored by everyone and work that does not fall neatly into a scrum is not done.

I suppose it does not help that my role has usually been one that is more Production Support than development and Agile is incompatible with production support. Take the idea of the scrum, where you have x days to analyse, plan, design, unit test and integrate the 6 things you will do in this round. On average I only spend 50% of my time dealing with urgent production issues, so I get allocated several tasks. Guess what, if I end up spending 75% of my time that week on urgent production issues, and urgent production issues have to take priority, I can screw up the scrum all on my own. No, I can’t pass my tasks onto others in the team as (a) they are all fully assigned to their tasks and (b) passing over a task takes extra time. Agile fails when it is used for the wrong teams and work type.

I’ve come to the conclusion that on most projects Agile has some beneficial impact in getting tasks done, as it forces people to justify what they have done each and every day, encourages communication and gives the developers a reason to ignore anything else that could be distracting them as it is not in the scrum. Probably any methodology would help with all of that.

My final issue with Agile is the idiot fanatics. At one customer site I spent a while at, they had an Agile Coach come around to help the team to become more agile. I thought this was a little odd as this team was actually doing a reasonable job with Agile, they had increased productivity and had managed to avoid the worst of the potential negative impacts. This man came along and patronisingly told us we were doing OK, but it was hard for us to raise our game like this, we just needed help to see the core values of Agile and, once we did, once we really believed in it, productivity would go up 500% {That is a direct quote, he actually said “productivity will go up by 500%”}. He was sparkly-eyed and animated and full of the granite confidence of the seriously self-deluded. I think he managed to put back the benefits of Agile by 50%, such was the level of “inspiration” he gave us. Agile fails when it is implemented like a religion. It’s just a methodolgy guys.

I find it all quite depressing as I strongly suspect that, if you had a good team in a positive environment, doing a focused job, Agile could reap great rewards. I’m assured by some of my friends that this is the case. {update – it took my good friend Mike less than an hour to chime in with a comment. I think I hit a nerve}.