Archive for the ‘Developer’ Category

Goodbye and see you soon!

October 22, 2011

Jayway – it lasted more than a decade. All the colleagues, all the fun, all the experiences. It just had to end.

We started in 2k and we just didn’t get it. The world of IT was about to change. All pretty much went downhill after the glorious, but delirious, nineties. It was four of us trapped in an attic with only a sliver of the sky to see – if we jumped really high.

I had signed on with a clause in my contract – I couldn’t be a manager. This suited me fine. I was a coder acutely aware of my flaws as a person. And I knew I tended to forget them (and still do). I knew one other thing. The choice for me was to go at it alone or to team up. The latter seemed so much more promising. Having colleagues seemed better. The sheer speed of a group on the go was way better if I wanted to learn faster.

It was to be – despite the first interviews, which were unbelievable bad. In this process my nephew (here’s looking at you Niklas) did two things right. One; He pointed me to a sentence in the national rag Computer Sweden talking about a company focused solely on Java programming. Two; He argued if they’re that immature, imagine how much you can affect! I hope I did.

Coffee
To this day I can’t pass the café without thinking about two of the founding members of Jayway I met there. High on caffeine I talked a couple of holes in their brains. The third founder was still on vacation when I signed the agreement which was high risk. But then again I never was the one for safety. Never has been.

That first year was one big struggle to make ends meet. It was a close call, but somehow we managed to survive. To this day I don’t truly know how. What we did get right though, was the team thing and two words – fun and competence. In the first year we got those right and kept on going.

Fast forward
11 years later, some 1300 hours of internal exchange of knowledge – above beyond the call of duty – we were touching 150 colleagues. Sure there were days where it rained and the smiles hardened by work. Who hasn’t been there? But there were so many hours and days of sheer joy. In short, it was great.

But, despite of all of those moments, all of those faces, I realized I needed to take the next step. For me it was to work with a team towards a common goal. Maybe dispersed geographically, but still hooked up mentally trying to live a common dream – a team. So there had to be an end. I resigned.

The people
It felt truly odd. I was going to be ex Jayway.
The next step was a total surprise to me – loads of heart and best wishes. I guess I just hadn’t given it much thought, but it hit my heart deep. A week later I still struggle to get my head around it.

Jaywayers! Thank you for all the years of fun and great work!

/Björn

Øredev 2009 Panel Video Books

November 9, 2009

The panel of Øredev 2009 proved to be a great group of people ready to take the panel format further.
As the moderator I wanted something other than your daddy’s discussion – something edgy or just plain edutainment.
I certainly got what I asked 🙂

The folks on the stage were:
James Bach – Author of Lessons Learned in Software Testing
Ola Bini – Thoughtworker, core developer of JRuby and creator of Ioke
Stu Halloway – Author of Programming Clojure
Scott Hanselman – Principal Programmer at MS and general spreader of good info for developers
Oren Eini (Ayende Rahien) – NHibernate Profiler, NHibernate, Castle, Rhino Mocks
Chris Hughes – AT&T iPhone Hacker

Scott Hanselman did a great job of setting up a live feed together with the Øredev people.
Here’s a good page from Scott, http://bit.ly/3C2UIi, on the event with video and everything.

Oh, and the non-computer related books that the panel recommended:
* The Gathering Storm (12th book of The Wheel Of Time) – Neil Jordan
* Anathem – Neal Stephenson +1
* Gödel Escher Bach – Douglas Hofstadter
* Replay – Ken Grimwood
* Flow – Mihaly Csikszentmihaly

Thanks guys!
/Björn

Originally published at Jayway Team Blog.

PMI, Scrum Shock Therapy

May 19, 2009

I’ve just uploaded the slides from the PMI conference in Amsterdam here.
Some 150 people turned up for this morning show, a Pecha Kucha with group discussions, about various aspects on Scrum. After some initial wrestling with the audio we got under way. We were four to speak – Juliet Andrew, Gabrielle Benefield, Petri Haapio and me. If you haven’t heard of this format it is a sort of PowerPoint in “burst mode”, i.e. 20 slides and each will flip over automatically after 20 seconds.
Difficult, but I would like to do it more 🙂

After that we broke up into separate groups for discussion. There were many good questions and I wished I’ve had more time on hand. Anyway, thanks to the those who attended and for your great feedback.

/Björn

Originally published at Jayway Team Blog.

Scrum Shock Therapy, Part 2

December 17, 2008

So we have a recipy for the team. But this is just not enough. We need to make sure that management and everyone else is on board. The second part of the Scrum Shock Therapy presents a bootstrapping recipe on how you can do this.

The first part of this series can be found here: Scrum Shock Therapy, Part 1.

The Management Recipe

The checklist above is only a part of what it takes to go through Shock Therapy. I often find myself at new companies and need to work with a bigger picture than focusing solely on the team. As is evident from the polls and surveys above, we need to address management as well. The word shock is perhaps more applicable here. If it has learnt anything from the past it is that you have to “drive” development – delegate, follow up, check, adjust, hand out, recheck…
Telling them to trust the very group that has not delivered before is asking for a leap of faith.

Setting up a recipe for this group is tricky. You have to adapt to management style, the corporate culture and so forth and the things you ask for are softer. These can be very hard indeed to change! In plain English; The management has to take some more real tough decisions for a Scrum to work efficiently. If you need to kill a document standard or process step then do so! If this is impossible (for now) then go for “barely enough”.

The Management

  • Attend the Scrum training session with everyone else
  • Hands off during 3 iterations
  • Attend:
    • Some Daily Scrums – be quiet
    • All Sprint Reviews
  • Start to work on waste – now!
  • Management by walking, asking and listening, i.e. practice facilitation.
  • Make the first step easy for the team

When learning something new you have to do it several times just to make sure you are not totally confused. I have found that three times a charm. Try your best to get management to abide by this “rule”. After the three iterations everyone has better picture to judge the value of the effort and Scrum.
Eliminating waste is vital on any type of project or endeavor. In a transition it makes even more sense, you have to make space to learn something new.

Finally, management has to ask the people that will perform these miracles what they need to do just this. Have they? Have you?

The Management Exit

  • One successful team
  • Removed impediments and studied results
  • Change in perception at “ground level”
  • A good agile reason to change the rule

Having one successful team is sort of self-evident. This takes patience when the ride is anything but smooth. Being a part of this process is important. Management will have the best ability to change ambition level, removing impediments etc. Get cracking and be that good example.

If there is a change at ground level on how they perceive management, then you’re probably on to something.

Is this recipe the final version – probably not? Your mileage will vary, adapt to your management.

The Organization Recipe

Many of our customers are just starting out doing Scrum. Therefore we typically come into play as an external effect. It is important for us to understand how the agile effort has started and its nature within our client’s organization. We can basically divide this efforts into to two major types – “bottom-up” and “top-down”. The typical traits for a bottom-up style is one where the programmers et al have decided to go agile. The management might be aware of this and even letting it run. Then again this might be a stealth and an under the cover job.

I have found the bottom-up approach to be the most common case. Programmers usually like to work in an agile way. In these type of setups my work usually revolves around questions like “How do we fit this to a waterfall context?” etcetera. This can be tough work, but I find it to be the more easy of the two – the people who do the day to day grinding are already on board.

The top-down effort of introducing Scrum comes with its own set of problems. Any methodology or decision that involves change tend to meet resistance from the organization. We need both experience and sideways effort to the transition.

The Top-Down Scrum Organization

  • Get a Scrum Sensei
  • Inject a Agile Senior Programmer into the teams
  • Make sure there is organizational transparency

The Scrum Sensei is an experienced Scrum Master who’s been there before. Much in agile and Scrum is common sense, but there is no need in making the simple mistakes. With a battle-trained mentor the whole organization get a sounding board (Swedish: bollplank) helping out to implement Scrum. This part is important to the Shock Therapy where he or she will be the contract owner enforcing the startup and approving exit. In some sense this person could act as a “bad cop” in a good-bad ScrumMaster setup.

Or in the words of Nanny McPhee – a children’s movie where a nanny answers the seven ne’er-do-well children on how long she will stay:
– When you need me, but do not want me, then I will stay. When you want me, but do not need me, then I have to go.
The Scrum Sensei needs to work in similar way to bootstrap the team and management.

A top-down effort always risks resistance from the organization. My experience tells me that a side way force is needed for translation on the factory floor. Moreover, such a person has the experience on a technical level to do the hands on tools and practices that is a vital part of any agile projects. A seasoned programmer with agile experience knows what works and how to translate the management vision into bits and bytes.

“Self-evident” sometimes reads “evident to myself and no one else”. If we want to affect the organization we also have to make sure that practices, experiences and results are easy to come by. Walking down a corridor should yield interesting information about ongoing projects. If there is a pilot project doing Scrum then visibility is high on the wish list. Not only does it mean commitment for the Scrum team, but also a pull factor for other teams to get on board. Organizational transparency comes in many flavors. Apply liberally!

Summing it up

There is no denying that projects have problems getting started with Scrum. But more often than not I believe that we can do something about it. Why not try some nice Shock Therapy?
It might be the sweetest hard deal around!

/Björn Granvik, Jayway

MySpace Therapy:
http://jeffsutherland.com/scrum/2008/09/shock-therapy-bootstrapping.html

Scrum Poll on Nokia Test Practice:
http://tinyurl.com/68vapj

VersionOne Report: State of Agile Development Survey:
http://www.versionone.com/agilesurvey/

Situational Leadership:
http://en.wikipedia.org/wiki/Situational_leadership_theory

Nanny McPhee:
http://www.imdb.com/title/tt0396752/

Originally published at Jayway Team Blog.

Questions from the Øredev session Shock Therapy

December 9, 2008

The Øredev conference was a real success. I’m involved in it ,so that probably makes me biased:-). However, I struggled for the first 10 minutes of my presentation to get my slides on the projector. A tip for you Mac users out there: Plug the adapter (dmi to vga) into the VGA cable first! I plugged a lone adapter into the computer first. It couldn’t find any projector and gladly gave up on me… Sigh. After that bumpy start I got going, but had to keep a brisk pace. Sorry, for that to those of you who where listening.

I asked for comments/questions. As it turns out I got mostly questions on Scrum in general and not so much on my topic – how to bootstrap Scrum. I’m not sure what to make of this.
Anyway, here are my takes on those questions. I hope they’re of some value.

Dear Björn,

How do you motivate a team to produce more without paying more for IT (salaries)?
Hourly salary seems retarded for agile teams.
Thanks!

Dear anonymous,

I’ll first look at the non-monetary part, mostly Scrum, and later on salaries and bonuses.

Self organizing
In many organizations people enjoy very limited control over their own situation. Being able to self organize is important. We humans just tend to like this.

Pick your tasks and choose your implementation
Again, if I can use my competence and decide (as far as possible) on how to implement a certain goal, I will enjoy my work the more. Using this approach I’ve even been able to rehire people to very problematic situations. Very powerful.

Deliver
When I ask programmers what they are most proud of, I usually hear words like “idea, made by me, real users”. It might not be the hardest and certainly not the biggest project they worked on. They used their competence, created something and delivered! In Scrum you get the delivery feeling at every sprint. If you break it down into activities no bigger than a day, then you will get that nice “flow” and people will be able to say “I did” every morning.

Good Colleagues
I’ve found that Agile makes sense if for no other reason than you can keep and hire the good people. SmartFriends(tm) is a great way to work!

I’m sure I could pick more facets of Scrum and Agile on how to motivate your team. But let’s move on to the Money. This part of my answer doesn’t have so much to do with Scrum or Agile, but rather my beliefs. So you have to be the judge here – what would work for you and your situation?

Powerful, way too powerful
Salaries, bonuses etc affect you – all the way to the bank and your holiday and…
In short: Once you start using money in different forms as your primary means of rewards, you will get a powerful ally in driving your people. Perhaps to powerful.

Take bonuses at a consultancy firm (a real example from life). You have all the good intentions so you set up a bonus system so that everyone will benefit if they have a client. More hours with customers, bigger bonus. Easy. This way you can lower your costs in bad times. Nice, if you’re just counting beans.

Effect: No one wants to come in do monthly meetings if it’s going to cost them money.
Counter effect: Introduce a threshold so that there is room for a bonus even though it doesn’t mean 100% with customer.

Another effect: You just removed your ability to make strategic decisions like “put someone on the big corporate account” that pays less, but means more hours. Who wants to loose money working for that customer if they have to take a cut money-wise?
Counter effect: You introduce another “rule” specifying that a lower price per hour will not affect their bonus.

Yet another effect: Holidays suddenly never cross the monthly boundaries. That would put your people below the threshold for the bonus on two separate salaries.
Counter effect: …haven’t got the faintest here what they did here.

I hope my example makes sense in your situation. My point is that money matters. When used to drive people you will get side-effects.
For me this is not good enough. I don’t want just to persuade people’s wallets – I want their hearts and minds. It has to be fun and engaging.

Here’s my short take: First make sure you have decent salaries etc. When this in place, aim for those things that make our working day worth while. Set interesting goals, get good colleagues, the right work to do, empower people and so on.

How to ensure creativity and quality in Scrum?
/Fredrik

Hi Fredrik,
I liked the way you wrote this on a green note (meaning “good” when voting at the conference). Thanks 🙂

Creativity
I believe there doesn’t have to be a contradiction between setting a goal and using proper frames (time, resources etc) vs. creativity – at least as long as we have the right to work to our own judgement within these constraints. I often find that this last part is missing. You get all the obligations, but no mandate.

In Scrum, one part of the creativity is built right into the process: Understanding the different goals. In the product backlog you should enter business value. This way the team has direct contact with the user’s intentions and can suggest alternative solutions that might be better, i.e. be creative.

Quality
If we’re talking IT, I would like to use automatic testing, continuous integration and so on to ensure consistent quality. Add to this an annoying email whenever something breaks and you have a good start 🙂

Moreover, the tight feedback loop using sprint reviews and such should pick up on issues like usability etc. Basically make sure you have a product owner that can spend time on the project.

Also make sure that your “definition of done” reflects your quality goals. It has to be very clear what “done” means within your organisation.

Hur gör man för att bryta ner gigantiska projekt över flera plattformar i lagom stora bitar för att köra Scrum?
[How to break down into gigantic projects over several platforms into decent chunks so that Scrum can be used.]
Sven Nilsson, SAAB

Hi Sven,

Your question seems to revolve around two hefty issues: Breaking down the project into several teams that are likely to depend on each others and breaking down the actual work to fit into a sprint. Perhaps “platform” has a special meaning within your organization and how you work. I would have to know a bit more to answer you on that point.

Teams and dependencies
This is tough on several levels. First let’s do the pure Scrum answer – Scrum of Scrums. This basically means that we organize our teams so that one person from each group “steps up” and forms a team that work across several teams. Dependencies etc can be resolved in this group, set up an encompassing backlog and work much in the same fashion as a Scrum Team. Jeff Sutherland, co-fonder of Scrum, has run this set up on 500+ persons. I would love to work on such an outfit.

However, and this is a big however, this is a major change for most companies. The ones I’ve met (medium size and up) just aren’t geared to handle such an approach right now. The number of issues and obstacles with such a transformation can be huge. If you find yourself in such a situation then I suggest you adapt piece by piece and continuously strive to be ever more agile. This is hard work and will mean small improvements upon each other, but maybe no hyper productive state.

This is where I chicken out and suggest you should get someone with experience. Every company is different and needs their set of actions.
Hell, buy me lunch and maybe I can point you in a good direction 🙂

Breaking up the work into chunks
This takes creativity.

First: Try to slice the work so that you will touch base with most layers/parts. This way we can have a better understanding of what it means to deliver the whole system. Therefore make the slice as thin as you can while at the same time deliver something “valuable”. This might mean you need to mock other components (code that isolates different components with dummy answers). This can be very beneficial since it becomes some sort of contract with other teams.

If the chunk still is to big then you might have to resort to cutting up the chunk into parts by level in your architecture. This of course has the drawback that we don’t know what it will take to implement the parts we didn’t do. Business value might be less, but we might be able to “prove” that we’re on the right track. What is “proof” to your product owner?
If you can, stay away of doing tasks in your product backlog. In worst case people will be task-driven: “I’ve changed the registry, can I go home now?”

If we want to/have to promise the customer a delivery date. How can we do that when we don’t analyse the whole project?
We are focusing on the next sprint?

How can we tell management how many resources we will need in one month?
/Patrik Johansson, Ericsson

Hi Patrik,

Several questions and perhaps the most common ones. Let’s dive into them.

Analyze the whole project?
Why we can’t analyze the whole project? Because, this is blatant lie for anything bigger than a trivial project. I apologize for my frankness, but I’ve never seen a project with a single version of the Gantt schema, a single time plan, with a known set of resources…
They always change. Always.

If you and your management can’t agree that things change then you might have to go dualistic – outwards project manager old style, inwards ScrumMaster. It’s not easy, but in time you will gain some victories, err deliveries, and you can move the agile thinking further up. Hard work, but it’s better than doing waterfall all the way.

Your question is still valid. How do we commit on delivery if most things might change? The customer still needs to plan an ad campaign.
One trick is to reinterpret the “commit to delivery date” into agile terms. Get a stable backlog and learn the team’s velocities. This way you can commit to the same “distance”. Things will change, but (as always) it’s down to making things fit into the available time.

For this work we need some iterations under our belt to know our velocity and hashing out what the product backlog should contain. Therefore, mix the project plan’s you have do at the start with some “real work” like coding. Inspect and adapt. After while I think you will able to commit to management.

Focus on the next sprint?
You shouldn’t just focus on the next sprint. The closest work (sprint) is fine grained and the most well understood. The further away (further down on the product backlog), the more coarse grained should the user stories be. No point in being specific, when they’re far into the future.

Resources?
Which resources can you get? Go for full time members. Make sure can keep them. Look at the product backlog and calculate what you can get done with the people (velocity etc) that you got. The important part is to couple the resources with the goals. All too often your resources get slashed and the goal knocked into orbit – and not on the same day.
Get them in synch so that a change in one of them will affect the other.

I hope this helped.

Regards,
Björn

Originally published at Jayway Team Blog.

Where to listen to Scrum Shock Therapy

November 18, 2008

Just a quick note on Scrum Shock Therapy. I will give a session on this subject at the Øredev conference on Tuesday at around 10 o’clock. If you can make it I will talk about the whole shebang – recipes for the team, management and organisation.

I’d love to get your feedback. See you in the halls and the Way Group booth.
Or you can just give your comments about the session here on this blog.

Developer's conference in Malmö, Sweden

Developer

//Björn

Originally published at Jayway Team Blog.

Scrum Shock Therapy, Part 1

November 17, 2008

Scrum consists of a straightforward process, half a bunch of roles and a few artifacts. Sounds simple enough, but according to an online poll 3 out of 4 projects that call themselves Scrum fail to implement even the simplest parts. To make matters worse, most of the mistakes are on the simple side of things. Is there a way to bootstrap the project? Might there be a sweet hard deal that we can use?

Scrum and Agile development has started to rock the world within IT in various ways. A simple, but powerful process that seems to deliver. Developers and customer alike are happy. What’s not to like about it? Still, there are signs of problems. Projects that claim to use Scrum do not reap the benefits, management looses patience with this new method and team members struggle to adapt an agile approach to their everyday work life. Why is this? And more importantly, what can we do about it?

The Therapy: Get off to a good start by directing the team with a careful set of good practices and a strict agreement that leaves little or no choice. The team can then, over a couple of iterations, gradually take command themselves.

Easy, but still so hard

Why then is Scrum so hard? Well, first of all it is fail-fast, meaning that any problems will be quickly apparent. There are no artifacts or process steps to hide behind. This is never easy. To this list we can add even more pitfalls that are just plain hard to deal with whatever method we are using.

Still, it does not explain why there seems to be so many simple mistakes with such a simple framework. One of the best examples is the failure to keep backlog that is prioritized and up to date.
– We’re doing Scrum, but we’re too busy to work on the product backlog!

That one has always short-circuited my brain. What can be more important than to decide what is important? This pattern of falling short and missing basic parts of Scrum even has a name – “ScrumButt”. If you look in the sentence above you see why.

Nokia created a test to make sure the projects are “Scrum enough”. These plain questions form a lithmus test that can be used to analyze most agile projects.

The Nokia Test
Iterative Development?

  • Iterations – time boxed, less than 4 weeks.
  • Software features – tested and working at the end of each iteration.
  • Iteration must start before specification is complete.

Scrum (in Nokia’s opinion)?

  • You know who the product owner is.
  • There is a product backlog prioritized by business value.
  • The product backlog has estimates created by the team.
  • The team generates burndown charts and knows their velocity.
  • No project managers (or anyone else) disrupting the work of the team.

The questions seem to be on the easy side. It turns out that this might not be the case.
One online poll, Nokia Test by Practice , about the ability to achieve the Nokia Test level gave some interesting answers. In essence, only one out of four give themselves a full mark. Half of the groups report that management interferes, not directs but interferes.

Informal queries by Jeff Sutherland among his audiences around the world give an even worse number – something like 1 out of 10 pass the Nokia Test. Hand on your heart: Does your project live up to this?

A survey by VersionOne tries to dig deeper into the current state of the Agile affairs. Under the headline “Top barriers to adopting Agile” we read the top four answers:

  1. Organisational culture, 45%
  2. General resistance to change, 44%
  3. Lack of people with experience, 42%
  4. Lack of management support, 32%

Take a long look at these barriers. No technical stuff. Nothing specific to agile. Just people.

What do we know?

Learn to walk before you run. Greek Proverb.

We could be a bit closer to today’s reality than these ancient words of wisdom. The Situational Leadership offers an approach to management that has been around since the 60’s. These theories say that there is no single leadership style that is optimal for all people in every situation. Situational leadership talks about four different leadership styles that need to be matched with the follower. Basically, when leading someone who is passionate, but have no experience, you should adapt and give clear directives. As the follower evolves his or her ability to execute, so should the leader using less directives and more encouragement. In the end the follower is able to perform with only a stated goal and little or no directives.

If this holds true for people, could it not be true for teams as well?

The Shock

So we have problems making Scrum and Agile efforts to run. What is the conclusion? Could it be that we adapt our leadership style to the group? Make them walk before running?

This is counter intuitive, at least to me: To go agile, we first need to control. We should not let our teams and managements start off without some kind of directives. Give them a good deal, some practices that make Scrum work. Otherwise, we are just repeating the same old mistakes as the next-door agile project. Set up an agreement with both management and team so that they can evolve and experience Scrum for real. Just reading a book (or article) or listening to some good speaker alone does not help you out there in real life. You have to experience it.

Shocking as it may feel: We need to exercise some control over the Scrum project start so that it can evolve and self organize! But, and this is a big but, this has to be done with compassion and genuine involvement, otherwise we stand the risk of destroying what we are trying to build.

The Therapy

So what can we do? We need to work the team, management and often the whole organization.

The Team Recipe

So how do we get started? Here’s a recipe for the team as it used at MySpace:
The Team

  • Scrum training session for everyone
  • Sprint 1 week long
  • Definition of Done:
    • Feature Complete
    • Code Complete
    • No known defects
    • Approved by the Product Owner
    • Production Ready
  • Story Points
  • Physical Task Board
  • All-in-one Sprint planning meeting.
  • No Multi-tasking, work in priority order.

Scott Downey, Agile Coach at MySpace

The idea of this recipe is to set a good practice right from the start. Instead of debating the details on how to work in an agile way, we can focus on getting going.

Everyone has to have a common Scrum understanding to start from, hence the training session. One-week iteration sounds really tight for any reasonably sized projects. The thinking behind this is to get as quick iterations as possible – the faster that the group gets going the better. Short sprints help in this aspect. We learn faster with more turnarounds.
This in turn affects the meetings where MySpace has elected to combine them all, except the morning meeting, into one single occasion. The time is just too short to in a weekly sprint.

MySpace’s definition of done is a good and basic list. However, as a programmer I lack for instance continuous integration, pair programming and so on. These are not mandatory to have, but I see them as enablers for Scrum and supporting your agile initiative.

The next two on the list – Story Points and Physical Task Board – are really effective. If you haven’t tried them out then please do. They work. Finally we have something that is a no-brainer to me even though I have great problems avoiding it, “No Multi-tasking, work in priority order”. Context-switching is expensive, just avoid it. The priorities are in order, you just have to follow them. In the Reference box you find a link to Scott Downey’s description of their therapy.

When the team matures, they should evolve and handle things more on their own and no longer need directives. At MySpace the group can “prove” that they are ready for prime time by reaching the following goals.

The Team Exit

  • Hyper-Productive (>240%)
  • Three successful Sprints consecutively
  • Good business reason to change the rule

The Result?

So how does this work out? At MySpace all groups achieved exit. All, but one, improved after that. One group even achieved a whopping 1,650% improvement after just four months (16 sprints). At Jayway one of the teams used such a bootstrap technique, primarily on the technical side, and reached 800% after 3 months.

The second part of this blog can be found here: Scrum Shock Therapy, Part 2 where I will look into how to handle management and organisation with some recipes.

/Björn Granvik, Jayway

Originally published at Jayway Team Blog.

Interview: Dr Jeff Sutherland, father of Scrum

February 2, 2008

Jeff Sutherland has piercing blue eyes – friendly, but probing – and a frequent smile. They are a bit like a distant uncle who wants to know what you have made of yourself.

After several CTO assignments at various companies, Jeff Sutherland found himself at Easel corporation where he created Scrum together with Ken Schwaber. Ken later presented this methodology at the OOPSLA conference in 1995.

A dozen years later I find myself in search for a good spot where I can concentrate on my interview. Jeff and I sit down at a café at the Øredev conference. My computer has been acting up going blue screen on me twice that day. Time is short and I wonder if it’s an omen.

Photo of Dr Jeff Sutherland.

Photo of Dr Jeff Sutherland.

Dr Jeff Sutherland. Photo by Mai Skou Nielsen, Trifork

Q: I hope I’ve done my reading. As far as I understand you’ve been a fighter pilot in Vietnam, an assistant professor in math and medicine and the CTO at nine different high tech companies. Moreover, you travel the world to teach Scrum. Does it take a resumé like this to invent a process like Scrum?

The key part of what makes Scrum work is “complex adaptive systems” and you need some understanding of this. Also you need to understand systems theory and how to optimise a system as a whole.

You do not have to be smart, but you have to thoughtful.

Q: How did Scrum start and from where does it draw its heritage?
Scrum and Lean both derive from complex adaptive system theory. Constraint theory is a narrow subset of complex adaptive systems thinking. Lean implementations of these theories have been created at companies like Toyota and Honda. Honda was actually more aggressive at this in the nineties, but then Toyota was more successful in the market. They also were more formalised and better known.

Q: Why does it work?
The first reason for all of this is important – super intelligent performance out of ordinary developers. In order to do that we need we need to set up a simple framework so that people will self-organize. The second reason is that there is no central point of control. The manager needs to back off. That is critical.

The third principle is behaviour that emerges in the team based on self-organizing in order to get max production.

The fourth reason is that the production is far higher for the group than any single developer. Groups actually achieve more than any lone bright developer.

These are the basic principles of a complex adaptive system. Any organization is a complex adaptive system. Software production is an organizational phenomenon. It is an empirical process that must absorb significant change in requirements, technologies, priorities, and people.

Q: You talk about managers, why is it important for them to back off?
A company needs to stop sub-optimizing locally. That is the thing that Toyota figured out, largely due to the efforts of Edward Deming and other Americans who trained the Japanese in WWII industry processes. Most managers just care about local performance. Managers have to work as a team to look across the whole spectrum. It is difficult for them. They’re paid and given bonuses on a single level, not the company as a whole.

For instance, look at the General Motors manufacturing plant in Fremont, California. GM shut down that plant 1982 due to several problems, among them quality. Two years later it reopened as a joint venture between Toyota and GM. Toyota needed a plant in the US in order to avoid import restrictions. In an agreement with the labour union they offered reemployment to the former employees.

Two years later the NUMMI plant was producing over 200 000 cars per year and was twice as productive as the typical GM plant. The thinking they applied is the same as “lean production” where Toyota makes the team responsible to figure out the best way to solve the problems and achieve the goals. NUMMI later won several awards for quality.

“Your battery is running low”, declares my machine solemnly. Jeff and I make a run for a power outlet and find one on the second floor. My power chord is too short so we stack up real close to the wall. It will do. More than that actually, it feels like we really are working together on this interview.

Q: Some parts of Scrum are frightening, like making the group totally responsible for an implementation. How come people in management buy this?

The best answer I can give you, is the discussion I had with my first CEO when I introduced the ideas of Scrum. We had a critical project with a hard deadline. It was a question of survival.

The CEO expected a Gantt chart [ed: a timeline specifying activities and their dependencies, showing the progress of a project].

  • How many Gantt charts have you received that actually worked? I asked the CEO.
  • None, was the short answer. In my 25 years I haven’t received a single one that was correct. I asked him if a 100% failure rate wouldn’t make it foolish of me to give him one.
  • What are you going to give me, he asked?
  • Working software. Decide for your self to if we’re on track. In return I made him promise not to disrupt the team before the deadline. At this point he started to worry.

I said that at the end of the month he could change everything he wanted, but not during that month.

  • If you do this, you will get significant more code with better quality.
  • I can live with that. Let’s do it.

Scrum was born in that minute.

It’s a basic contract between management and team. The team delivers twice as much stuff with twice as good quality (minimum) and the management supports the team in their efforts to self-organize and self-manage. If either party does not fulfil this contract then it will fail.

Q: Doesn’t Scrum fail?

Scrum itself is a set of rules that causes self-organization if certain things are put in place. It’s like the rules of soccer. Does soccer fail?
With Scrum you will fail significantly less.

The Standish Group \[ed: a company that specializes in IT studies\] reports that project in the range 3-6 million dollars with a traditional project plan fail more than 85% of the time. Scrum does a lot better than that!

Q: What are your top three tips on using Scrum?

Bas Vodde was the Lead trainer for Scum Master at Nokia Siemens Networks. He has the shortest set of questions to identify a dysfunctional Scrum team.

You must have iterations of 30 days or less with working software tested at system level at end of each iteration. You need an agile specification, just enough for programmers to start for that iteration.

You need a product owner who orders activities by business value of the feature set. The team estimates the product backlog.

And lastly, the team needs to monitor progress with a burn down chart and know its velocity at the end of the iteration.

Q: Finally, what is in the future for Scrum?

Hyper productivity on an economic and marketing level where companies consistently out perform their competitors. Scrum is a disruptive technology that will alter markets. Companies doing Scrum well will grow rapidly in both size and profitability and dominate their waterfall competitors. There is already a brain drain going on where the better developers are fleeing from waterfall companies to Agile companies. This will hasten the demise of traditional projects.

Q: Is this a shameless plug?
Oh no, we have achieved it in several departments at my company \[PatientKeeper\]. It is doable. Other companies I work with right here in Scandinavia are doing it as well.

Going away from this fast paced interview, I still have the feeling that I could have done more with my life. If Jeff can do it, then so can I …I hope

Originally published in JayView.

Interview: Martin Fowler – man in the know

February 2, 2007

I am in search of an empty room at the Øredev conference. Normally this is an easy task, but I’ve got Martin Fowler on my tail. My mind is still blank. What on earth can I ask him that he hasn’t already written himself?

Finally, an empty room, well almost. Another speaker, Erik Dörnenburg, is sitting half way into his screen and mutters.
– What’s up, I ask.
– I’ve updated my machine and my demo doesn’t work. I’ve got 45 minutes until the presentation.
We sat down next to him. Do not disturb a developer while he’s coding…

So I got a man who has coined phrases like Dependency Injection and POJO in front of me. What next? Martin is easily recognizable both in accent and appearance, a frequent and brilliant speaker.
He has an excellent web site, http://www.martinfowler.com, which contains loads about his work. Articles and references abound. That is when it suddenly hits me – who is he as a programmer and person?

Q: When was the last time you coded?
Well, I do code my own website. But it’s been a while since I had any paying customers. I’ve been pairing quite recently though. A real delivery? That was some time ago. I’m actually afraid to lose contact with code, but I have smart people around me.

Q: But what makes you tick?
I enjoy trying to figure out new techniques – to organize knowledge. I see myself as a conduit [ledning] for transferring knowledge, to process what is out there and make some kind of structure out of it. Brian Foot actually described me as an “intellectual jackal with a good taste in carrion” [intellektuell schackal med god smak för kadaver].
I look around for interesting stuff and try to make sense of it.
The “Refactoring” is a good example. I figured out how to describe it and wrote a book that came out when it could make a difference and move the area forward.
I also enjoy writing a lot, that’s a big thing. I’m better now at speaking, but that’s not what makes me tick.

Q: You’ve written quite a few books – how do they compare?
Out of the five, “Uml Distilled” sold more copies than the others put together. Usually you can’t make a living out of your books, I guess I could though.
All of the books had their good sides, but I would have to say that it was fun to write with Kent Beck [red: wrote “Extreme Programming”, created JUnit etc]. We were in tune and could support each other through the dull bits.
I would have to say though that I’m proudest of “Refactoring”. It’s an important technique and didn’t get the attention it should have received – the book helped.

Q: How did you start out?
I was an independent consultant for many years. Giving talks was a good way of getting jobs. Articles same thing – it got my name known.
Also, I write something because I don’t understand a certain area or technology. It’s a good way to learn.
Erik is now on the phone with California. We calculate that time is roughly 6:30 there – in the morning.

Q: Then what? How come you started working for Thoughtworks (TW)?
I’ve been there for six years and done a lot of consulting. I never wanted to work for a company, but there was something about TW that made me interested.
Get the work done and tons of bright people. But more importantly is that it is a sort of social experiment. A notion that good people makes a difference.
I hope we can affect IT, which is a difficult and skilled exercise at best.

Q: What is the most difficult part of being a celebrity?
I’m not an extrovert person. I’m not good at the “person to person”. I get emails with questions like “I got a problem on…what is the magic trick”. They worked for months on it and I can only point to a book. That clearly wasn’t an answer they liked. It’s frustrating.
However, celebrity is also a nice thing – it opens a few doors. I can email people like Rod Johnson [red CEO of Interface 21 that created the Spring framework] if I have a question about something. And he will answer.
People tend to think I’m an ingenious programmer. I’m not. I’m pretty good, but not necessarily that great.
Erik suddenly spits out:
– F—!…ok the demo will be shorter.

Q: Looking forward, what’s next?
Oh, there is tons of stuff to write about. The design patterns area for instance. I’m also interested in DSL [domain specific languages] and agile development. But in agile there are too many writers and I don’t like competition. There are too many smart people in agile development.
My strategy is to look for topics that no one has written about. Basically I don’t foretell the future.

Q: What are your top three pieces of advice to a programmer?
My first advice must be to learn to collaborate with the user or purchaser. The really good ideas usually come from them. You don’t have to be an expert to do this. This I found to be a good general advice.
Secondly, it would be “continuous learning”. It’s like running up a downwards-moving escalator – you have to keep running.
The third one is difficult…
“Buy lots of books by good authors” would be it.
Erik suddenly releases a big:
– Yes!
I saw Erik’s demo some twenty minutes later – it was really good.
As for Martin, our discussions continued well into the debate panel and beyond. He would frequently forget his back pain and sip into some extra energy pack. I wonder how he did that.

Originally published in JayView.

Neo – a netbase

February 1, 2007

Neo is a network-oriented database for semi-structured information. Too complicated, let us try again. Neo handles data in networks – nodes, relationships and properties – instead of tables. This means entirely new solutions for data that is diffi cult to handle in static tables. It could mean we can go agile all the way into the persistence layer.

The relational database represents one of the most important developments in the history of computer science. Upon its arrival some 30 years ago, it revolutionized the way the industry views data management and today it is practically ubiquitous.

In fact, it is so taken for granted that we, as an industry, have stopped thinking. Could there be a better way to represent and store our data? In some cases the answer is – yes, absolutely. The relational database is showing its age. Some telltale signs:

  • The mismatch between relational data and object oriented programming.
  • Schema evolution – updating your data model when the domain model changes – is just so much manual labor.
  • Semi-structured or network-oriented data is diffi cult to handle.

The Neo Database

Neo is a database that is built with a different philosophy. Where the relational database squeezes all data into static tables, Neo uses a fl exible and dynamic network model. This model allows data to evolve more naturally as business requirements change. There’s no need for “alter table…” on your production databases after you introduce a new version of the business layer and no need to rewire and migrate your O/R mapping confi gurations. The network will evolve along with your business logic. This spells agility.

Neo is an embedded persistence engine, which means it’s a small, lightweight and non-intrusive Java library that is easy to include in your development environment. It has been designed for performance and scalability and has been proven to handle large networks of data (100+ millions of nodes, relationships and properties). Neo is a newly founded open source project, but the software is robust. It has been in commercial production in a highly demanding 24/7 environment for almost four years and has full support for enterprise-grade features such as distributed ACID transactions, confi gurable isolation levels and full transaction recovery. But so much for sweet talk, let’s cut to some code!

Model and Code

Representation

In the Neo model, everything is represented by nodes, relationships and properties. A relationship connects two nodes and has a well-defi ned, mandatory type. Properties are key-value pairs that are attached to both nodes and relationships. When you combine nodes, relationships between them and properties on both nodes and relationships they form a node space – a coherent network representing your business domain data.
This may sound fancy, but it’s all very intuitive. Here is how a simple social network
might be modeled:

Figure 1

Figure 1: An example of a social network from a somewhat famous movie. Note the different type on the relation between Agent Smith and his creator The Architect.

Note how all nodes have integer identifi ers and how all relationships have a type (KNOWS or CODED_BY). In this example, all nodes have a “name” property. But some nodes have other properties, for example, an “age” property (node 1) or a “last name” property (node 3). There’s no overall schema that forces all nodes to look the same. This allows Neo to capture so-called semi-structured information: information that has a small amount of mandatory attributes but many optional attributes. Furthermore, the relationships have properties as well. In this example, all relationships have an “age” property to describe how long two people have known each other and some relationships have a “disclosure” property to describe whether the acquaintance is secret.
Working with nodes and relationships is easy. The basic operations are as follows:

Figure 2

This is an intuitive representation of a network and probably similar to many other implementations that want to represent a network of data in an object-oriented language. It’s worth noting, however, that relationships in this model are full-blown objects and not just implicit associations between nodes. If you have another look at the social network example, you’ll see that there’s more information in the relationships between nodes than in the nodes themselves. The value of a network is in the connections between the nodes and Neo’s model captures that.

Creating a Node Space

And now, finally some code. Here’s how we would create the Matrix social network
from figure 1:

Transaction tx = Transaction.begin();
EmbeddedNeo neo = ... // Get factory
// Create Thomas ’Neo’ Anderson
Node mrAnderson = neo.createNode();
mrAnderson.setProperty( ”name”, ”Thomas Anderson” );
mrAnderson.setProperty( ”age”, 29 );
// Create Morpheus
Node morpheus = neo.createNode();
morpheus.setProperty( ”name”, ”Morpheus” );
morpheus.setProperty( ”rank”, ”Captain” );
morpheus.setProperty( ”occupation”, ”Total bad ass” );
// Create a relationship representing that they know each other
mrAnderson.createRelationshipTo( morpheus,
   MatrixRelationshipTypes.KNOWS );
// Create Trinity, Cypher, Agent Smith, Architect similarly
...
tx.commit();

As you can see in the code above: It is rather easy to construct the node space for our Matrix example. And, of course, our network is made persistent once we commit.

Traversing a Node Space

Now that we know how to represent our domain model in the node space, how do we get information out of it? Unlike a relational database, Neo does not support a declarative query language. Instead, Neo provides an object-oriented traverser framework that allows us to express complex queries in plain Java. Working with the traverser framework is very straight-forward. The core abstraction is, unsurprisingly, the Traverser interface. A Traverser is a Java Iterable that encapsulates a “query” – i.e. a traversal on the node space such as “give me all Morpheus’ friends and his friends’ friends” or “does Trinity know someone who is acquainted with an agent?”. The most complex part of working with a Traverser is instantiating it. Here’s an example of how we would create a Traverser that will return all the (transitive) friends of the “Thomas Anderson” node of the example above:

// Instantiate a traverser that returns all mrAnderson’s friends
Traverser friendsTraverser = mrAnderson.traverse(
    Traverser.Order.BREADTH_FIRST,
    StopEvaluator.END_OF_NETWORK,
    ReturnableEvaluator.ALL_BUT_START_NODE,
    MatrixRelationshipTypes.KNOWS,
    Direction.OUTGOING );

Here we can see that traversers are created by invoking the traverse(...) method on a start node with a number of parameters. The parameters control the traversal and in this example they tell Neo to traverse the network breadth-first (rather than depth-fi rst), to traverse until it has covered all reachable nodes in the network (StopEvaluator.END_OF_NETWORK), to return all nodes except the fi rst (ReturnableEvaluator.ALL_BUT_START_NODE), , and to traverse all OUTGOING relationships of type KNOWS. How would we go about if we wanted to list the output of this traversal? After we’ve created a Traverser, working with it is as easy as working with any Java Iterable:

// Traverse the node space and print out the result
for ( Node friend : friendsTraverser )
{
    System.out.println( friend.getProperty( “name” ) + “ at depth “ +
        friendsTraverser.currentPosition().getDepth() );
}

Running the traversal above on the Matrix example would yield the following out-
put:

$ bin/run-neo-example
Morpheus at depth 1
Trinity at depth 1
Cypher at depth 2
Agent Smith at depth 3
$

As you can see, the Traverser has started at the “Thomas Anderson” node and run through the entire network along the KNOWS relationship type, breadth fi rst, and returned all nodes except the fi rst one. “The Architect” is missing from this output since the relationship connecting him is of a different type, CODED_BY. This is a small, contrived example. But the code would work equally well on a network with hundreds of millions of nodes, relationships and properties. Now, let’s look at a more complex traversal. Going with our example, suppose that we wanted to fi nd all “hackers of the Matrix,” where we defi ne a hacker of the Matrix as any node that you reach through a CODED_BY relationship. How would we create a Traverser that gives us those nodes? First off, we want to traverse both our relationship types (KNOWS and CODED_BY). Secondly, we want to traverse until the end of the network and lastly, we want to return only nodes which we came to through a CODED_BY relationship. Here’s the code:

// Instantiate a traverser that returns all hackers of the Matrix
Traverser hackerTraverser = mrAnderson.traverse(
    Traverser.Order.BREADTH_FIRST,
    StopEvaluator.END_OF_NETWORK,
    new ReturnableEvaluator()
    {
        public boolean isReturnableNode( TraversalPosition pos )
        {
            return pos.getLastRelationshipTraversed().
                isType( MatrixRelationshipTypes.CODED_BY );
        }
 },
 MatrixRelationshipTypes.CODED_BY,
 Direction.OUTGOING,
 MatrixRelationshipTypes.KNOWS,
 Direction.OUTGOING );

Now it’s getting interesting! The ReturnableEvaluator.ALL_BUT_START_NODE constant from the previous example was actually a convenience implementation of the ReturnableEvaluator interface. This interface contains a single method and you can supply a custom implementation of it to the traverser framework. It turns out that this is a simple but powerful way to express complex queries. Setting aside the anonymous inner class cruft surrounding the code in bold, we basically pass in a snippet of code that checks whether we traversed a relationship of type CODED_BY to get to the current node. If this statement is evaluated to “true” then the current node will be included in the set of nodes that is returned from the traverser.
When executed with a simple print loop, the above code prints the following:

$ bin/run-neo-example
The Architect
$

StopEvaluators work the same way. In our experience, writing custom evaluators
is very easy. Even the most advanced applications we have developed with Neo – applications that traverse extremely large and complex networks – are based on
evaluators that are rarely more than a few lines of code.

Conclusion

Neo is not a silver bullet and some areas needs to improve, for instance tools, standardizing the model and a query language. However, if your data is naturally ordered in a network or is semi-structured or you just need to go truly agile, give the Neo database a run for your money. We hope you find it, as we do, to be an elegant and fl exible alternative that is both robust and fast.

Emil Eifrém, Neo Technology
Björn Granvik, Jayway

Links

Neo specification
www.neo4j.org

Originally published in JayView.