2

I've been researching for while on data regarding benefits of CI. But I haven't found any solid data, there few threads discussing this topic example:

but most of them deal with abstract example: "Early warning of broken/incompatible code"

These kind of things we can't measure. Has any body seen measurable data like: "bug count", "build time", "defect resolve time" etc when continuous integration is added to build process

3 Answers 3

3

I found this white paper from collab.net very helpful:

http://www.collab.net/content/building-value-continuous-integration

Hope that gives you enough data to go on.

Cheers!

Steve

0

A metric can be quantitative or qualitative.

It is easier to measure quantitative metrics, like build time. And there is actually a benefit in measuring build time, and I have seen that. For example, you might discover that build time goes above the reasonable limit, what impacts "feedback time", which is essential for productive coding, so you might act on "exceeded limit trigger". For example in this particular case, consider to split your solution into multiple components or do "staged" integration or something else.

It is harder to measure qualitative metrics, like project visibility, team happiness. For example, CI makes things (build/test/release/deployment/etc process/status) visible to everybody and visible earlier. So, CI ROI depends on ROI of increased visibility. And outcome of visibility is difficult to measure but it is possible, and it is qualitative metric. One way of capturing qualitative metric is doing regular surveys. It is separate science to develop right surveys, but for example in this case, you might ask people to rate from 1 (not true) to 5 (completely true) the statement: "Notifications from CI system helped me to do better and earlier decisions."

Hope it helps.

3
  • Hi thanks for replay, but did you read my question, You basically just rephrased my last comment? I agree and as I said most of CI benefits are like qualitative, what I'm looking for is actual quantitative data (whitepapers, research, etc..).
    – PotOfTea
    Nov 6, 2013 at 9:59
  • Yes, I have read. I did not paraphrased your last comment. I tried to explain that it is qualitative data, and it is possible to measure it reliably (difficult but possible). What actually is slightly opposite to the statement you made: "These kind of things we can't measure." I have seen similar measurements in action, and even gave you an example how it could be done in your case.
    – Andrew
    Nov 6, 2013 at 23:39
  • I appreciate you taking the time answering my question, but I fail to see how any of this answer my main questions - about any publicly available surveys, whitepapers etc. regarding "measurable data" or what you mentioned "qualitative metrics". "I have seen similar measurements in action" Could you share any of them?
    – PotOfTea
    Nov 7, 2013 at 10:03
0

After 5 years there is book which specifically deal with this topic - "Accelerate: Building and Scaling High Performing Technology Organizations". Contains research/investigation of devops practices and lean management.

Not the answer you're looking for? Browse other questions tagged or ask your own question.