 |
 | Home | Contact Us | FAQ | Search & Site Map | Link to Us |
 | Sign In | Join | Other 45 Sites in Network |
Java Forum / General / September 2005Opinions on Java test development
I am looking for some expert opinions. We are developing new Java tests for a leading global consulting firm. The tests will assess the skills of J2EE architects and Senior Java Developers. I could use input on the differences in how to best assess various level candidates. We will be looking for members of the test development team once the focus of the test is finalized. ReviewNet has been a leading provider of IT skills testing since 1997 and we have developed over 300 tests. We are determined to make this the best Java suite of assessments possible and would like to get a variety of input to make sure we are not biased in how we develop the assessment. I look forward to hearing from you. I can be reached at mbartie@reviewnet.net Thanks, Michelle D. Bartie ReviewNet Corporation www.reviewnet.net http://expert.reviewnet.net phone: 215-572-9400 ext. 113 Rated Best by Inc. Magazine, and now with over 800 assessments for IT, office skills, call center, and more! http://www.inc.com/magazine/20001201/21124.html >I could use >input on the differences in how to best assess various level >candidates. We will be looking for members of the test development >team once the focus of the test is finalized. If I were devising a test, it would be an open book exam. The student would even be allowed to use google and the newsgroups, just like real life. Why make the test environment any more artificial than it absolutely has to be? You give him X hours to create a working miniature system from specs, perhaps involving some new technology that no student would have ever heard of. For more advanced students you have him write the spec too based on wooly customer needs. What a high level person has to be able to do is come up with something plausible then get feedback on it rather than relying on the customer to articulate his needs. All you really can do with multiple choice is measure familiarity.  SignatureCanadian Mind Products, Roedy Green. http://mindprod.com Again taking new Java programming contracts. > You give him X hours to create a working miniature system from specs, > perhaps involving some new technology that no student would have ever > heard of. I think toy problems are fundamentally different from serious systems. IMO, commenting on code is a far better way to go. Although that isn't amenable to an automated test, so doesn't fit what this company wants to do. Tom Hawtin SignatureUnemployed English Java programmer http://jroller.com/page/tackline/ >IMO, commenting on code is a far better way to go. Although that isn't >amenable to an automated test, so doesn't fit what this company wants to do. I think what the company want to do is foolish. If they truly want to test somebody they should give a test that means something, not that is easy to administer. A new hiree is not all that useful during the first year. After that, they really start to carry weight. You are training them and familiarising them to a great extent to start. Since you want someone who will hang around for the long haul, it pays to measure him properly. Note the ads. Companies are desperate to shorten that time, wanting their hirees to be totally familiar with every tool they will use on the job. SignatureCanadian Mind Products, Roedy Green. http://mindprod.com Again taking new Java programming contracts. >>IMO, commenting on code is a far better way to go. Although that isn't >>amenable to an automated test, so doesn't fit what this company wants to do. [quoted text clipped - 6 lines] > you want someone who will hang around for the long haul, it pays to > measure him properly. Or her. If a new hire takes a year to pull their weight, then I think you have problems. Certainly there is going to be a bit of time (months) when not only is the new employee not productive, but they reduce the productivity of others. The critical factor in getting up to speed is the ability to learn and think for yourself. > Note the ads. Companies are desperate to shorten that time, wanting > their hirees to be totally familiar with every tool they will use on > the job. That's more to do with laziness, incompetence and ego boosting. My first job had loads of languages listed. Apparently it's not uncommon for EJB to be an absolute requirement where EJB is not used, has not been used and there are no plans for its use. Listing packages rather than requiring abilities has got to be an inferior technique to actually testing something (or anything!). Tom Hawtin SignatureUnemployed English Java programmer http://jroller.com/page/tackline/ >>IMO, commenting on code is a far better way to go. Although that >>isn't amenable to an automated test, so doesn't fit what this [quoted text clipped - 3 lines] > to test somebody they should give a test that means something, not > that is easy to administer. I agree. This may not be at all what you're thinking of, but ... A job I applied for several months ago required applicants to do an online Java language test. It was timed, but they didn't tell you it was timed until after you started the test. It didn't allow you to use reference materials, which doesn't represent any real-world development environment I can think of. And the questions ... The questions were basically of the form "Here is some code that no one in their right mind would ever write. It uses obscure syntax, requires intimate understanding of precedence rules, etc etc and anyone who codes like this should immediately be taken out back and shot. Describe what happens when it's run." Certainly this test will weed out a lot of people. But will it identify the types of developers you really want on your team? I guess I find that doubtful. I'll admit that I did poorly on the test. So maybe I'm biased. On the other hand, when I did technical interviews with real live people, the questions were a lot better at reflecting real world needs. Things like explaining scenarios in which you would use threads, inheritance, etc. Signaturemonique Ask smart questions, get good answers: http://www.catb.org/~esr/faqs/smart-questions.html > The questions were basically of the form "Here is some code that no > one in their right mind would ever write. It uses obscure syntax, > requires intimate understanding of precedence rules, etc etc and > anyone who codes like this should immediately be taken out back and > shot. Describe what happens when it's run." I put some similar comments on the SCJP 5.0 beta. Most code looks as if it was written by some who should be forcibly restrained from going anywhere near a keyboard. > On the other hand, when I did technical interviews with real live > people, the questions were a lot better at reflecting real world > needs. Things like explaining scenarios in which you would use > threads, inheritance, etc. Most technical interviewers are pants. It's rare to get asked questions by someone with a clue. Thinking back I can only remember four that were worthwhile. The two best of which were with, I believe, C++ programmers. Tom Hawtin SignatureUnemployed English Java programmer http://jroller.com/page/tackline/ >> On the other hand, when I did technical interviews with real live >> people, the questions were a lot better at reflecting real world [quoted text clipped - 5 lines] > four that were worthwhile. The two best of which were with, I > believe, C++ programmers. I'm not familiary with the expression "are pants." What does that mean? Maybe you and I disagree on what constitutes a good technical interview, but to me a general understanding of programming and engineering principles and methodologies is more important than being able to describe obscure points of a specific language. Signaturemonique Ask smart questions, get good answers: http://www.catb.org/~esr/faqs/smart-questions.html >>Most technical interviewers are pants. It's rare to get asked >>questions by someone with a clue. Thinking back I can only remember [quoted text clipped - 3 lines] > I'm not familiary with the expression "are pants." What does that > mean? According to my dictionary "pants" is british slang for "utterly bad". SignatureThomas >>>Most technical interviewers are pants. It's rare to get asked >>>questions by someone with a clue. Thinking back I can only remember [quoted text clipped - 5 lines] > According to my dictionary "pants" is british slang for "utterly > bad". Any explanation of the etymology? Signaturemonique Ask smart questions, get good answers: http://www.catb.org/~esr/faqs/smart-questions.html >>> I'm not familiary with the expression "are pants." What does that >>> mean? >> According to my dictionary "pants" is british slang for "utterly >> bad". > > Any explanation of the etymology? <http://www.ocf.berkeley.edu/~wrader/slang/p.html> says this: inadequate, displeasing, or of poor quality. Possible origin: underwear, called "pants" in Britain. SignatureThomas > Maybe you and I disagree on what constitutes a good technical > interview, but to me a general understanding of programming and > engineering principles and methodologies is more important than being > able to describe obscure points of a specific language. I don't think that the major thrust of an interview should be obscure points. However, if the candidate knows the obscure that is probably because they have a very good general understanding of the subject and can think. You will find it extremely difficult to find someone who knows the obscure but doesn't have a good general understand of programming and engineering principles. Similarly someone who has developed a very detailed understanding of a system will certainly have come across many of the obscurities. As a side point, obscurities are not necessarily useless, although they may be. Many form traps. If you are successfully going to code review, then to complete the task you need to be sure it does exactly what you expect (not a sufficient skill, but a necessary one). If you are going to solve a problem (banging out the trivial is irrelevant), then you need to understand the bounds and not be afraid to determine what is possible and what is not. Tom Hawtin SignatureUnemployed English Java programmer http://jroller.com/page/tackline/ > > Maybe you and I disagree on what constitutes a good technical > > interview, but to me a general understanding of programming and [quoted text clipped - 9 lines] > developed a very detailed understanding of a system will certainly have > come across many of the obscurities. Actually, I have several counterexamples I have met - people who studied for a cert test, and thus know a whole bunch of trivia, without any real knowledge of how these features would be used. People who could tell me all sorts of cruft about exactly how log4j implements a logger, without being able to tell me a coherent logging policy. Similarly, when pressed on how they would evaluate and adopt a new technology, they tend to founder. They learned their trivia from a few sources, rather than finding out about the trivia during the course of implementing something important. I really try to test prospective hires on their knowledge of the skills I need, and their ability to learn the new stuff I do not yet know I need. This is nontrivial - you have to make a judgement call. I might start this process with a block of really cruddy code, then ask how they would approach fixing it. Some trivia might be nice, but the key is how they think. Scott SignatureScott Ellsworth scott@alodar.nospam.com Java and database consulting for the life sciences > You will find it extremely difficult to find someone who >knows the obscure but doesn't have a good general understand of >programming and engineering principles. Yes, but on the other hand, you may find that person a ditherer who can't get on with real work. He is always fiddling with things that are not that important. If he knows this stuff, it is likely because is always pushing the envelope. This may or may not be the sort of guy you want. If you want him maintaining applications, you don't want an envelope-pusher guy. You want someone happy to write very mainstream pedestrian code. SignatureCanadian Mind Products, Roedy Green. http://mindprod.com Again taking new Java programming contracts. >>You will find it extremely difficult to find someone who >>knows the obscure but doesn't have a good general understand of [quoted text clipped - 7 lines] > envelope-pusher guy. You want someone happy to write very mainstream > pedestrian code. That's a variation on the gold-plating myth. It's a popular one with managers who see a mock-up but don't understand that nothing works. I understand it doesn't really happen in real life. Unless you count witting tests and the such like. So, you'd go for pedestrian, copy & paste guy? Suppose you were developing a product and it was decided that testing should be written down as steps to follow in Word documents with visual checking of the results. Would you prefer the engineers going for that, or the one who spends an hour or two adapting the automated testing framework? The people you have to look out for is those who use inappropriate "technology" to put on their CV. Those people who just have to use OLE or EJB or whatever. Tom Hawtin SignatureUnemployed English Java programmer http://jroller.com/page/tackline/ >The questions were basically of the form "Here is some code that no >one in their right mind would ever write. It uses obscure syntax, >requires intimate understanding of precedence rules, etc etc and >anyone who codes like this should immediately be taken out back and >shot. Describe what happens when it's run." Sometimes, unfortunately, this is a VERY realistic test. :-) There is quite a different skill set required to write code from scratch and to make changes to a massive program written by someone else. SignatureCanadian Mind Products, Roedy Green. http://mindprod.com Again taking new Java programming contracts. Thanks to everyone that commented on my post. I agree that there can be some serious problems with on-line testing. That is we we are looking to take a different approach. We may need to utilize a different format other than mutiple choice. Thanks again for your feedback! Michelle >>The questions were basically of the form "Here is some code that no >>one in their right mind would ever write. It uses obscure syntax, [quoted text clipped - 7 lines] > scratch and to make changes to a massive program written by someone > else. I've only rarely been able to work on a project that involved only my code. I still think that the ability to understand purposely obtuse code is only a small part of the skill set you need to work in that kind of environment. From my perspective, I'd rather hire people who will rewrite that code to make it readable (and still do the same thing) than people who find it relatively easy to follow the disgusting code (and hence don't see anything wrong with it). Signaturemonique Ask smart questions, get good answers: http://www.catb.org/~esr/faqs/smart-questions.html >I could use >input on the differences in how to best assess various level >candidates. What you might be able to do successfully with multiple choice is a bullshit detector. Let's say someone said they had enveloped large scale EJB applications. What you need are questions that anyone who has DONE that would HAVE to know, not arcane stuff, but that someone who had merely read a book ABOUT EJB would not know. These can be very bread and butter things, like "Which of these three queries is of the sort most commonly used in EJB applications?" SignatureCanadian Mind Products, Roedy Green. http://mindprod.com Again taking new Java programming contracts. >>I could use >>input on the differences in how to best assess various level >>candidates. > > What you might be able to do successfully with multiple choice is a > bullshit detector. With, at a guess, 90% of programmers being pretty much worthless, a bullshit detector has to be worth its weight in brass. A good proportion of companies don't even seem to bother to test at all. OTOH, they wont have anyone capable of usefully judging candidates anyway. > Let's say someone said they had enveloped large scale EJB > applications. [quoted text clipped - 4 lines] > like "Which of these three queries is of the sort most commonly used > in EJB applications?" That gets you as far as the one month experience level (possibly repeated dozens of times). It fails to tell you whether the candidate is competent. While certainly not foolproof, awareness of the obscure indicates that the candidate has experience gained through thinking through and solving problems. If someone has lots of time served, yet haven't been aware of what's going on, then they're idiots. Tom Hawtin SignatureUnemployed English Java programmer http://jroller.com/page/tackline/ Free MagazinesGet these publications absolutely FREE for up to 12 months. There are no hidden fees and no obligation. Simply choose a title, complete the application form and submit it. Read more ...
|
Reply to this Message
|
 Sign In
 Join
 My Latest Posts My Monitored Threads My Blog My Photo Gallery My Profile My Homepage Start New Thread Enable EMail Alerts Rate this Thread
|
©2008 Advenet LLC Privacy Policy - Terms of Use
This website includes both content owned or controlled by Advenet as well as content owned or controlled by third parties.