32 replies to this topic

[carlosfocker]

I have been working for a while developing a design for a game I plan on making. I have made small games before and understand somewhat of the time and effort need to construct similar games but not the size of the purposed game. Also, I currently work in the field on web development and have a firm grasp of software estimation when it comes to web development but am having a hard time translating my knowledge to game development. My biggest problem is I have never taken on a project like this and I don't have any historical data to use. Does anyone know if industry historical data exists on the web for video game projects? I need stuff like feature size to LOC, LOC per function point and stuff relating to effort and velocity like average LOC per staff month. If anyone knows where to find information like this or even has information they would like to share it would be greatly appreciated. If nothing exists I might plan on starting a wiki for information about this subject.

[starstutter]

carlosfocker said:

Does anyone know if industry historical data exists on the web for video game projects?

That's actually a really good question. However, I think you're searching for the wrong things. Look on google for developer blogs. If you want to know "whats in store", you want to find it from an indie perspective. Finding the experiences of a huge company probably aren't going to help you much personally.

This site has a "work log" and these guys are behind some really good indie horror games:
http://www.frictiona...play.php?fid=29

One question though... what the heck is an LOC?
Library of Congress?
LeMoyne-Owen College?
Line of Control?

Plz be specific in the future ;)

[carlosfocker]

LOC = Lines of Code. Its a standard way of measuring size for a project/feature and is helpful when coming up with estimates for future projects. I can't imagine that estimation practices for the general software industry would not translate over to video game development. After all, a video game is just another form of software. Also, the big reason i need to do this is for a budgeting needed by some potential investors.

P.S. In the future I will be more explicit with my acronyms :)

[starstutter]

carlosfocker said:

I can't imagine that estimation practices for the general software industry would not translate over to video game development. After all, a video game is just another form of software.

Well that's not really what I meant. I meant to say that you're comparing a planned personal project with a multimillion dollar behemoth. The development process is an apples and oranges thing. What I was getting as is you should probably focus less on records of companies and instead focus on logs and journals of sucessful indie developers. The best part with that is, not only can you probably get access to a lot of blog writing (blog writing that's actually useful), but chances are you could talk to them personally.

The link to frictional games I gave you, the creaters of the penumbra series are on the forums every day chatting it up with people, I've talked to them as well a while ago. I'm sure they would be more than willing to at least provide you with some work related diaries (which usually contain absurd amounts of technical information) and maybe even talk one-on-one (if they find time).

Quote

Also, the big reason i need to do this is for a budgeting needed by some potential investors.

Well, at least you've been in code development for a while and know a fair amount, and in my eyes thats a substancial amount of promise and potential. Trust me, you're a pretty big farcry from our usual "GiMM3 dA 3n9ine ta mak3 teh MM0!!?!!!"

What's the game about btw? sheer curiosity :)

EDIT: Oh, just to let you know, I unfortunatley don't know the answer to your origional question. I'm telling you this because I know how much it pissed me off on GameDev when people (way too frequently) said "I know the answer, but this is what you should be doing instead". So I'm not withholding or anything :D

[JarkkoL]

Hmh, I think LOC is very bad estimate for anything. You can have a newbie coder doing the same thing as more experienced coder with 10x LOC.

[carlosfocker]

JarkkoL said:

Hmh, I think LOC is very bad estimate for anything. You can have a newbie coder doing the same thing as more experienced coder with 10x LOC.

I agree this is a weakness of using lines of codes. For a team though, LOC can be a good way to estimate general velocity. When used in combination with things like Fuzzy Logic it can be a powerful tool to scoping out future project effort.

Edit: "LOC can be a good way to estimate general velocity. " Not sure why I said this (could be that I was posting at 3am :) ) but I really meant that loc data when used with other data is good low effort way to measure size of a project when your in the budgeting phase. Also, if you have enough historical data you can sometimes map feature size and/or function points to loc to get an round about idea about the general size and effort needed for a project. This is what I'm trying to accomplish. This technique should really only be used very early on in a project. It should not be used as the final plan(planning and estimation are two different things). The use of bottom up approaches, that involve the people actually doing the work, to estimate (When done correctly) seem to be much more effective later on in the development life cycle. In no way do I endorse holding developers to loc per month. I have seen the effects of this and it is ugly.

[carlosfocker]

starstutter said:

Well, at least you've been in code development for a while and know a fair amount, and in my eyes thats a substancial amount of promise and potential. Trust me, you're a pretty big farcry from our usual "GiMM3 dA 3n9ine ta mak3 teh MM0!!?!!!"

Or "I don't know anything about development or making games but I have good ideas for the next big MMO. Anyone interested in helping out?" :)

starstutter said:

Well that's not really what I meant. I meant to say that you're comparing a planned personal project with a multimillion dollar behemoth. The development process is an apples and oranges thing.

I get what your saying here. It's true that publisher based games will have higher skilled, more experienced staff which will throw off my particular estimate. This is the biggest issue with using industry data opposed to your own groups historical data when forming estimates. When you have no data this is a low accuracy/low effort way to get a range to use for preliminary planning. What you gave me is actually a good start. I will try to query those guys for what I need. I just need rough figures so I'm not completely off. (OMG, I just sounded like my boss)

starstutter said:

What's the game about btw? sheer curiosity :)

Sorry but due to a NDA that I signed I really can't say anything about the game right now but as it progresses I will be sure to share. I really dig the site and have gotten a lot of information from here. Thanks for your help.

[TheNut]

It's difficult to compare industry data with your own team because there are to many variables. Experience, senior developers vs junior, team size, project size, etc... Some group of people can produce a product every two years, for others it could take up to 6+ years, but maybe there was 2 years worth of R&D in that. I don't have direct links, but Gamasutra has posted emperical data from time to time. Wikipedia also has some games listed with data that could be useful (although a lot of researching required on your part). There was also a couple sites posted here a while back that provided really in-depth data on indie games. I don't have the link, but I'm sure someone here does; otherwise search "I'm feeling lucky" ;)

Without empirical data however, it's unwise to dish out random numbers. I've seen people do it and time and again management gets really angry. LOC is not only weak in this regard, but it's also an archaic form of project planning. I know some people still use it, but in my opinion they need to get out of that micro management. From experience, incorporating a software development life cycle first is important. Break the project up into milestones, each milestone should have a set of goals (documentation, coding, features, tests, etc...). It's easier to look at something like "Soft Shadow Generation" design and implementation documents and extrapolate a timeline from that then it is to project an estimated LOC with no bearings and then hound on your staff to find out how many lines of code were done today. LOC is useful for other statistics however; I just wouldn't recommend labeling your staff with it. If you're going to use that metric silently to measure velocity, that's fine.

Information is key. The most important project planing comes from your team, so talk with them frequently.

[Sol_HSA]

This is my favorite "lines of code" story:
http://www.folklore....nes_Of_Code.txt

You should never, ever use lines of code as a meter of anything that actually affects the programmers. Tell the programmers, for example, that their bonuses are tied to the lines of code produced, and you'll find programmers writing tools to generate craploads of code. Tell them that fewer lines of code are desirable, and you end up with cryptic code and/or very long lines.

Lines of code are comforting as they are something a layman can understand. Coders write code. Code comes in lines. The line count grows. That must mean something, right? But it doesn't.

For historical data, you could mine the web (and especially gamasutra) for postmortem reports. These sometimes have some numerical data, but that probably doesn't go deep enough to give you insight for the specific project you're working on.

Like TheNut above said, there's too many variables involved. The best you can do is get your hands on experienced people and go through your project plans with them. If they tell you something you're attempting will take a long time, it probably will. =)

[JarkkoL]

carlosfocker said:

For a team though, LOC can be a good way to estimate general velocity.

I completely disagree. Number of LOC tells nothing about the amount of work done and should never be included in anyway your way into your project management. Even though I'm not into scrum myself you should probably look into it if you want to know a better way to measure velocity.

[imerso]

"Lines Of Code" tells you nothing.

One can do a brilliant work in 1 line, other can do stupid work in 1000. And vice-versa, of course.

Just don't use "Lines Of Code" measurement, it will not give you any realistic measurement of "velocity" and/or competence of your team.

Programming may be better measured by project goal achievements, and even this may be questionable.

[carlosfocker]

JarkkoL said:

I completely disagree. Number of LOC tells nothing about the amount of work done and should never be included in anyway your way into your project management. Even though I'm not into scrum myself you should probably look into it if you want to know a better way to measure velocity.

I knew I shouldn't have mentioned this :). This type of argument always seems to happen with this topic. Every point that you all have brought up is valid. That's why their are pro's and con's to each method and a good reason to use multiple approaches when approaching early estimation on a project. Just to let you know, I do have actual (Like in a real office ;)) experience with this stuff. If you want a better idea of what I was trying to convey (sometimes English doesn't always agree with me), check out Steve McConnel's book Software Estimation: Demystifying the Black Art. Please, let us not stray to far from the original question or soon we will start arguing about things like how C#'s curly brackets are much better than VB.NET's end statment.

[carlosfocker]

Sol_HSA said:

For historical data, you could mine the web (and especially gamasutra) for postmortem reports. These sometimes have some numerical data, but that probably doesn't go deep enough to give you insight for the specific project you're working on.

I didn't think of gamasutra.

[JarkkoL]

Well, I have actually managed programmers in several large real world game projects and I can't see how LOC would give you any useful information. Only thing it would do, IMHO, is to put wrong kind of pressure on the team which makes the team to focus on wrong things which doesn't help with the success of a project. I have seen the same happening even with scrum which provides much more sensible way to track progress. Measuring LOC sounds just awfully like trying to manage a team by numbers which, IMHO, fails to recognize that team management effort is truly people management in its heart. But I digress, and what ever floats your boat of course :)

[carlosfocker]

TheNut said:

It's difficult to compare industry data with your own team because there are to many variables. Experience, senior developers vs junior, team size, project size, etc... Some group of people can produce a product every two years, for others it could take up to 6+ years, but maybe there was 2 years worth of R&D in that. I don't have direct links, but Gamasutra has posted emperical data from time to time. Wikipedia also has some games listed with data that could be useful (although a lot of researching required on your part). There was also a couple sites posted here a while back that provided really in-depth data on indie games. I don't have the link, but I'm sure someone here does; otherwise search "I'm feeling lucky" ;)

Without empirical data however, it's unwise to dish out random numbers. I've seen people do it and time and again management gets really angry. LOC is not only weak in this regard, but it's also an archaic form of project planning. I know some people still use it, but in my opinion they need to get out of that micro management. From experience, incorporating a software development life cycle first is important. Break the project up into milestones, each milestone should have a set of goals (documentation, coding, features, tests, etc...). It's easier to look at something like "Soft Shadow Generation" design and implementation documents and extrapolate a timeline from that then it is to project an estimated LOC with no bearings and then hound on your staff to find out how many lines of code were done today. LOC is useful for other statistics however; I just wouldn't recommend labeling your staff with it. If you're going to use that metric silently to measure velocity, that's fine.

Information is key. The most important project planing comes from your team, so talk with them frequently.

I totally agree and that's why software estimation and project planning should be held as two different things. This is line that most management blurs together. The code chop shop senorio is definitely the horror of extreme misunderstanding of using a technique that involves
loc. I have seen people that have gotten paid per line of code. What they created was the biggest pile of crap I ever saw. I feel a good planner will use estimates given to him as a tool not a commitment to create a plan. Planning should take into account inevitable change and plan for it. As you said the use of project milestones with set amount of goals is good tool. The problem is I'm not even to that part of my project. I am merely trying to create a budget, not a project plan.

I look back now on my past post and don't know why I used the word velocity. I don't see a method like Fuzzy Logic as a way to measure performance but rather a low effort/low accuracy way of getting figures early on in a project life cycle to establish budget constraints. It is definitely not the only way. It's important to use many methods. If anyone has suggestion of estimation techniques they used for a game i'm interested in hearing them.

[carlosfocker]

JarkkoL said:

Well, I have actually managed programmers in several large real world game projects and I can't see how LOC would give you any useful information. Only thing it would do, IMHO, is to put wrong kind of pressure on the team which makes the team to focus on wrong things which doesn't help with the success of a project. I have seen the same happening even with scrum which provides much more sensible way to track progress. Measuring LOC sounds just awfully like trying to manage a team by numbers which, IMHO, fails to recognize that team management effort is truly people management in its heart. But I digress, and what ever floats your boat of course :)

Yeah, I definitely didn't convey what i wanted to earlier. I do agree that loc does not measure productivity. The worst thing is telling your team they need to produce loc per day. I'm interested in hearing what techniques were used on your projects to establish estimates early on like in the budgeting phase.

[alphadog]

LOC is a generally useless metric for estimation or otherwise.

Worse yet would be to assume that 10K lines in a business line web app would be "as time- and resource-consuming" as 10K lines in a 3D FPS.

For the "best" estimation technique, IMO, look at SCRUM and Cohn's book. Even then, use extreme caution. The fundamental problem is that anything that involves making a detailed list (LOC, Function Points, even use cases if the idea is to doc all use cases a priori, etc...) implies that the whole project is definable up-front in a detailed way. Also, when you excessively breakdown tasks, and each item is naturally padded in estimate, you end up with this insane number as an estimate. EVen if you add something like "Planning Poker"...

Another tack is to use "historical" data, hopefully yours (meaning your team's), which in your case you don't have. First, using some other team's data is pointless. Even then, how sure/knowledgeable are you on your data and your ability to relate relevant items back from old projects to the new one? What if the last similar project was pretty much the same, except for that new parser which caused all manners of coding and integration headaches, making you slip much longer than anticipated at its start, but can be reused in the new project? Especially in game development, where there often is "new" code, nevermind new technology integrations. Can that new graphics algorithm be backtracked into some data on implementing canned graphics algos?

At the end, you are guesstimating. You can guesstimate with one "technique", or with an average of n "techniques" (say, historical, FPs and Planning Poker), but in the end, it's still a wild guess.

Just quickly guesstimate and throw them a number... You'll end up with more coding time to stick to that guesstimate...

[carlosfocker]

alphadog said:

Worse yet would be to assume that 10K lines in a business line web app would be "as time- and resource-consuming" as 10K lines in a 3D FPS.

This is definitely true and I have already account for that. When using loc it must by a similar system. I understand that using someone else's data beside your own gives you a less than accurate estimate. This is the whole balance of Count/Compute/Judgment. Count what you can, compute the data to usable information and use judgment only when nothing else available. If you have had the chance to use estimation software that has been calibrated by industry data the range you get is usually very wide. But it can be used to perform sanity checks on other estimates I have created. Really this discussion can go on forever due to bias. I for my situation, the better approach to establish budget constraints might be to determine my team structure and determine cost for running that team for a certain length of time, adapt my feature set and schedule as I begin to collect data from the group and use that in future estimations to alter the plan to what I see.

[alphadog]

carlosfocker said:

When using loc it must by a similar system... If you have had the chance to use estimation software that has been calibrated by industry data the range you get is usually very wide. But it can be used to perform sanity checks on other estimates I have created.

More than just a similar system. That's the problem.

It must be a similar system, built by a similar team, with similar technologies, similar skills (at the start and which are developed/developing similarly) and with similar resources. This team must generally experience similar problems and roadblocks at similar times. Likewise, the manager(s) and stakeholder(s) must also have similar understandings of the project at similar times.

Once you jot all that down on your napkin, estimating is very easy...

Joking aside, how often does that level similitude happen? Hint: Start with 'N' and ends with 'ever'.

I'm sorry. I don't mean to bash your ideas, but I've been down that road too. I've built Gantt charts and used MS Project. I've done work trying to use previous projects as a jumping point for new projects. Works okay as long as nobody quit and I have the same team, the economy is the same, the client doesn't suddenly call a gazillion meetings because you disagree on the signed-off design of the login page, "John the Tester" doesn't go through another divorce, our CVS server doesn't go belly-up and our backups turn out to be just as dead because the dog ate the tape, etc...

I spend 10x more time creating them, than actually using them.

Quote

the better approach to establish budget constraints might be to determine my team structure and determine cost for running that team for a certain length of time, adapt my feature set and schedule as I begin to collect data from the group and use that in future estimations to alter the plan to what I see.

I guess what was not clear in your posts is: why are you estimating? For yourself? For others? Who are the others?

[carlosfocker]

alphadog said:

Hint: Start with 'N' and ends with 'ever'. :)

Lol

alphadog said:

I'm sorry. I don't mean to bash your ideas, but I've been down that road too.

It's ok, I enjoy hearing other points of view on topics. I like seeing what has and hasn't worked for other people. For me personally, I have used various techniques such as some I outlined with success and some with not so great success. The big difference between what you are saying and what I am talking about is the difference between planning and estimation. What you talk about later in your reply is more planning and managing the plan for and against change. I am at the area where I need estimated ranges to determine general budget for stakeholders. The plan would obviously change from the initial estimates since the Cone of Uncertainty dictates error of up to 400%. What i am trying to do is more business level planning than actual development level planning.