Package Software Evaluation Overview
By Stefan Piechota
Purchasing software can be compared to buying a house or a car. If you do not do your package software evaluation homework upfront and “kick the tires”, you could end up wasting a lot of time and money, or worse, being confronted with an angry mob of users.
Recently, with the acquisition and consolidation of software vendors and packages, it has become even more critical to really “look under the hood” of the application prior to selecting a new software package/vendor. As an example, during a recent vendor evaluation, TSI discovered that a vendor’s claim to have real time inventory available between its supply chain and point-of-sale (POS) applications was false. This vendor had acquired a POS vendor and had not fully integrated the databases so these were operating on batch updates every four hours – not real time!
Due to the importance of the “functional fit” of packaged software to one’s requirements, many times the technology due diligence for packaged software purchases often gets rushed or short-changed. Sometimes the purchase decision is in the hands of a few individuals whose biased opinions come into play. On the flip side, some software vendors may steer you to the great functionally-oriented features of their software and seek to avoid discussing any shortcomings. If you don’t ask they won’t tell!
Choosing a software package requires formal software evaluations by two important stakeholder groups in your company: the business users and the IT staff. These evaluations are usually done in two distinct meetings for each of the stakeholder groups:
- Business Software Evaluation, using a formal Demo script derived from business requirements
- Technology Evaluation
This article will delve into the Technology Evaluation component of the software evaluation and it will give you some examples of the types of questions to ask. Remember, asking the right questions and knowing what to look for can make the difference between success and failure on a project like this.
Background
As TSI has no financial relationship with any software vendor, TSI is one of the few, purely objective evaluators of packaged software. During projects of this nature, one of TSI’s goals is to assist clients in how to “choose and use” technology to increase the chances for a positive ROI and satisfied users. We do this working closely with key subject matter experts to document their current and future needs, conducting unbiased software evaluations and helping our clients select the right (best fit) software package that meets the needs of management, business, and technical users. In the March 2003 edition of Info Source we reviewed the techniques and issues related to performing an effective software evaluation. Here’s an important quote:
“Most articles about the pitfalls of implementation projects highlight the mistakes made DURING implementation. Examples include poor project management, scope creep, uncommitted users or lack of an executive sponsor. All of these are valid, but often we forget that before we started the implementation, somehow we had to choose a package and sign a contract.”
The Technology Evaluation
Typically, a technology evaluation is performed with a formal meeting between the IT staff of the software vendor and the IT staff from your company. IT staff who should attend are as follows: CIO/CTO, Database Administrators, Application specialists, and an Infrastructure specialist. The meeting will last a minimum of 1 hour to upwards of a day, depending on the number of software modules being evaluated.
There are 3 types of documentation that should be sent out prior to the meeting. Let’s discuss these in detail.
First, the prospective software vendor should receive a map of your technology footprint, including your current databases, software installations, operating system, and security implementations. This will let the software vendor better understand your technical environment before any demonstrations are conducted and make them aware of any technical prerequisites they must follow (e.g., must be installed on Oracle database, run on Unix, etc.). The level of detail of this documentation will be driven by the complexity of your current and planned technology infrastructure.
Second, the IT Evaluation Team at your company should obtain and pre-read technical literature from the vendor so that the evaluation meeting can focus on key decision criteria requiring a detailed review at this stage.
Third, it is important to have some structure in a technical evaluation meeting as there are many topics to cover and technical discussions can easily go adrift. Here is a list of topics that should be reviewed in a technology evaluation meeting:
- Overall Architecture/Configuration
- Application
- Release Management
- Client/Server Architecture
- Customization
- Portability
- Batch Processing
- Availability/Reliability
- Monitoring
- Database
- Database Management
- Flex/Customized Fields
- Record Contention
- Scalability/Performance
- Portability
- Batch Processing
- Archiving
- Backup/Recovery
- Availability/Reliability
- Monitoring
- Integration/API
- Security
- Staffing/Support
- Reporting/OLAP/Portal
Generally the above topics are sent out to the vendor in question form well ahead of the meeting. The vendor formally responds prior to the meeting so the IT staff has a chance to review the responses. The vendor may also reference their technical literature for some of their responses.
At TSI, we act as an impartial facilitator and subject matter expert to ensure the meeting is on target and all the necessary questions are asked in an unbiased manner. We use the same checklist with every vendor so there will always be a consistent review. We also act as an advocate for the client, much like a professional home inspector prior to purchasing a house to ensure the foundation and architecture of the house are sound.
Let’s dive down into 3 of the topic areas listed above (Client/Server Architecture, Customization, and Database Management) and see what types of questions to ask in the technical evaluation meeting.
Client/Server Architecture
One of the most important questions to ask up front with a software vendor is whether their application client is one of the following:
- Thin client (e.g., web application through a browser)
- Thick/Fat client (e.g., through an .exe on the user’s desktop)
- Remote client (e.g., connected remotely via Citrix/Terminal Services)
Thin clients are becoming more popular as there is less headache for user installations and they offer access from anywhere around the world over the internet. We all take advantage of thin clients everyday such as online banking or airline reservations. Conversely, security issues, performance, and decreased functionality compared to thick clients are factors to consider against thin clients.
Thick/Fat clients tend to have more functionality (e.g., Microsoft Excel or Word) but require a formal installation on the user’s desktop. Updates/bug fixes have to be formally distributed to the client.
Remote clients rely on remote administration software (e.g., Citrix, Microsoft Terminal Services) that allow the application to run on the remote server but be displayed on the user’s desktop. They have some of the benefits of both thin/thick clients at the expense of performance (versus a thick client) and having to have remote administration software installed (versus a thin client).
The lines between thin, thick and remote clients are progressively blurring. Be sure to get a clear understanding of what the software vendor provides. Keep in mind they may provide more than one client type; for example, the typical user may be exposed to a thin client but the administrator to a thick/remote client.
Discuss the language/tools used to develop the client. These will tend to be Microsoft-based (e.g. .NET) or Open Source-based (e.g. Java). Discuss what physical files are required on the user’s PC. Discuss if any 3rd party controls or applications are required or are prerequisites (e.g. only works with Microsoft Office 2003 or higher). Find out what, if any, propriety languages have been used to develop the software.
On the server side, you’ll probably be faced with one of two options from the software vendors: either Microsoft IIS based or Unix Based (e.g. Linux, Solaris). For this discussion, much will depend on what servers you currently have in-house and what your IT professional staff is trained on. Generally the server discussion is based on the software vendor providing technical specification/white papers on their supported server technology prior to the meeting. For example, this documentation should include details on configuration, security, and fail-over. Keep in mind that software vendors often prefer that their application reside on a server dedicated just to them. This will obviously have a cost impact.
Customization
There may be a need for you to customize or add extensions to the vendor’s software application to suit your current or future business needs. Review version control and how updates and patches are distributed. Have the vendor describe code management and development methods that must be followed to ensure that all customizations could be preserved in the event of an application version upgrade. Finally, have the software vendor describe the programming and development skills that are required to support the application and develop custom extensions to the application.
Database Management
It is important to know the type of database(s) that are supported by the software vendor. Oracle and/or Microsoft SQL Server are usually the front runners. It is important to understand what versions of the database are required by the software vendor, especially if you have one of the databases installed already. If the vendor’s software database requires a higher version, it must be factored into the cost and support equation. It is also important to ask the software vendor if any database extension or optional components of the database are required or recommended. Again this may have a cost and/or support implication.
Review with the software vendor any database management tools or utilities they either include or support or use themselves. Discuss data archival, data failover, data redundancy, and any data integrity validation methods with the software vendor. Database performance tuning is important. Ask them what their largest database is (in gigabytes) and how many rows/datapoints they handle. Ask them how they typically address capacity issues with their current client base. Is it a reactive or proactive process?
Good software vendors have data modeling DDL/DML/UML tools that come with their installation. This is important in setting up various environments (Unit Test, System Test, UAT, QA, Staging, etc) in a competent and quick manner. Your company may have unique fields that are not native to the software vendor’s database. Ask the software vendor if the data model is extensible to add user-defined fields (a quick demonstration might be in order).
There is a good chance you’ll need to populate the database prior to go-live so ask how this task is accomplished. Again, ask what they have done at other clients, especially those who are similar to your business model. This will probably involve discussion of their API, import, and data access utilities/routines.
If your company is involved with an intensive transaction processing you’ll want to ask the software vendor what transaction management product/utilities/methodologies are used. This should include discussion of consistency, failover, lock-outs, recovery.
Conclusion
We hope this whitepaper provides you with a sound basis of the type of questions asked in a technical evaluation. The technical evaluation and a sound software evaluation of the business requirements will strengthen your software vendor selection process and lead to a successful outcome. Please consider calling TSI should your team need any skilled, objective expertise in this area.
Would you like to add yourself or others to our list to receive this whitepaper on a regular basis? Just enter your email address in the Sign Up for TSI Whitepapers section at the top right of this web page.
Click these links to learn more about TSI’s services, TSI’s experienced consulting team, and how they can help your organization achieve optimization in your processes.
