CEO Blog: Making the Most of a Proof-of-Concept

Before many organizations invest in technology, especially when it's a new type of technology like a columnar database or a new BI tool, they will run a quick proof-of-concept. I have seen many, many of these. Some are done well, others are not. Here is a quick list of things to do and not to do when running a POC.

Let's start with things TO DO:

1. Think through how you really expect to use the technology, and design your POC to reflect that. This includes the type of queries, the amount of data, the number of users and concurrency, the type of data, etc. You may not be able to re-create an exact simulation, but the closer you come the better.
2. Make sure you have the right hardware to run the POC. This should be as close as possible to the configuration you would use in production.
3. Make sure you have people assigned to conduct the POC. If training is involved, then to the extent you can have your team gain an understanding of the POC environment, that's great. If not, then at least seek help from the vendor in order to have adequate oversight to ensure the POC yields results. Assume your own staff would ultimately need to increase their knowledge of the tools being tested.
4. Establish success criteria. How long should the query set run? What kind of ad-hoc performance do you expect? What should the concurrency rates be? What should the compression ratios be? How much DBA time should be needed to establish the environment? What should the load speeds be? All of this can be established going in if the right thought is given to the effort on the front end. This will always map to a productive effort.
5. Make sure you are serious about what you will do if the POC is successful. If both sides don't understand that, then the POC can be a great deal of work for both with no real results. For example, if the success criteria is met, will you proceed to negotiating contracts? If met, will the supplier agree to pricing you can afford? Much of this can be worked out up front.

Things NOT TO DO:

1. Embark on the effort with the intent of "playing around with the technology". That can be done, but it's not a POC.
2. Fail to allocate the proper hardware configuration to adequately test the software.
3. Fail to test over a reasonable representation of the system load, i.e. don't test against 20GB of data if the anticipated production will be 15TB.
4. Fail to allocate people to do the testing.
5. Go into the test without a defined success criteria (as specific as possible).
6. Expect the vendor to do everything.
7. Expect the vendor to do nothing (as you will often make assumptions that may lead you down incorrect or inappropriate paths).
8. Allow the "high priority" POC to get stopped and interrupted several times, because it turns out it isn't really a priority. This happens all the time, but it helps both the buyer and the seller to be as honest as possible about this up front.
9. Overtest. It's a POC, not a full development project. Don't assume you are going to run a POC that can be flipped into production at the end, unless this is truly contemplated alongside and with the agreement of the vendor. Otherwise, there will be a significant gap in expectations.
10. Do not treat the person or people from the vendor as servants. Most really like their work and are there to help, but they should treat you with respect and vice versa. This is obvious, but it is amazing how often this is a problem.

All in all, a Proof-of-Concept can be a very valuable exercise in determining whether a new technology is right for you. But it's work on both sides. If done right, it doesn't have to be a ton of work, but it can be very, very rewarding for everyone with just a little planning and thought going in.

Happy testing!

Don