IT Operations

If you're new here, you may want to subscribe to my RSS feed. Thanks for visiting!

In recent months I’ve been using SharePoint more and more to improve IT service delivery.  Specifically we’ve been implementing some of the ITIL concepts in SharePoint, such as a Service Catalog, but even more than that we’ve used it to document information about our customers, issues we’ve faced and the decisions we make.

We’re using a combination of custom lists, document libraries, and many other SharePoint features to create a highly collaborative environment that everyone has access to within their security constraints.

We’re starting to see a significant benefit in terms of improved service delivery due to improved communication.  Essentially having the most current information (and I use information versus data for a reason) available at all times improves the ability of the IT team to deliver service.  We’re eliminating mistakes caused by a lack of information.

So why did I call our “information versus data”?  What SharePoint is allowing us to do is add context to data and turn it into actionable information.  We’re mixing structured data with unstructured textual descriptions into organized dashboard like views which provide more than just some data.  The dashboards are providing the reasons why things are they way they are and what options are available per the desires/agreements with our customers.  This is priceless because it helps to reduce the number of contact points needed to get something done which in turn results in faster response time and higher quality service.

The downside of this is that it takes a lot of business analysis time to figure out just how to mix the structured and unstructured information such that it’s meaningful to those that need it.  Then you need the SharePoint expertise to implement it and since SharePoint is more a platform than a software package, its more work than you would expect it to be once you get into the details.

That said, I think the results are well worth the time and effort needed to make it happen.

Many organizations start an IT project, especially a software implementation or development project, and their first step is to start writing requirements.  We’ll at least I hope they write some requirements using RUP, Agile, Extreme or some other iterative or rapid methodology. While this is a good start from the perspective of the project, there’s something that often gets overlooked and not addressed till way too late in the process.

I’m talking about Service Design. Not to be confused with technical design or architecture but rather the design of how the result of the project (the new app) will be provided as a service to its users/customers.

Service design activities should focus on the operations of the production environment after the new application has gone live and address the operational needs in advance of going live.  The following is a checklist in question format to help you get started with “designing a service” that should help you prepare for going live with a new system.

1. What are the Business Continuity/Disaster Recovery requirements?
   • RTO & RPO at a minimum (see my post entitled Are you prepared for a disaster? for more detail)
2. Have performance requirements been defined?
   •  For example, how quickly should pages/screens be rendered or how many concurrent users should be supported?
3. Does the project plan implement a solution to meet the Business Continuity and/or DR requirements?
4. How will end users receive support? Level 1, 2 & 3?
5. What hours will support be available?
6. What skill sets are needed to provide support?  Do the support resources have the needed skills? What’s the anticipated support call volume over time and are you staffed to handle it?
7. Has a maintenance plan been created that identifies the scheduled tasks that should take place in order to keep the OS, database and application healthy?
8. Has a monitoring plan been designed to identify what OS, database and application layer monitoring needs to take place?
9. Who will be responsible for communicating with software vendors to get patches & updates?
10. Who will have the responsibility of reviewing and approving the installation of patches and updates?
11. Who will have the responsibility of reviewing enhancement requests/bug reports and approving them for development?
12. Who will perform the enhancements/bug fixes?
13. How will enhancements/bug fix requests be tracked and status communicated to those that made the requests?
14. Who will plan the development effort for things like what goes into a patch or update and what goes into a minor or major release?
15. Are development, testing and staging environments needed?
16. Who will update training documentation when it’s required due to development work?
17. How will new users be approved? How will you know to remove user accounts when needed? Who has the authority to make these changes?

Not all systems will require everything in this list.  For example, if you’re managing a public web site end user training for the public isn’t required.

In some cases answers to these questions may create system requirements to help manage the production environment.  For example, your new application may require some custom monitoring that requires scripts to be developed or you may need tools to see who’s logged in before you shut down the system for maintenance.

The point is that you should think of these things up front and address them in your project plan such that when the time comes to go live these issues are addressed and you aren’t caught by surprises. 

When working within a large IT Organization, several hundred to several thousand people, you’ll find (or should find) that IT Governance plays at least two key roles.

1. Reviewing business cases for IT projects and approving those that provide the greatest value to the organization

2. Oversight of IT projects as they execute in order to ensure projects both live up to their business case and only consume the organization’s resources (financial & human) that they’re approved to consume.

For those of you responsible for IT Governance, specifically for the 2^nd point above, I’m proposing an alternate approach for how an organization determines when a project manager should go through a governance review or “gate”. In my experience I’ve seen gate reviews tied to methodology phases, such as Requirements, Design, Development, and Implementation. There are inherent problems with this as it relates to the objectives of governance.

Basing gate reviews on methodology can create confusion when using agile or other iterative development methodologies.  Further it leaves the door open for project managers to get too far down the path of being over budget before executing a gate review. It’s easy to burn a lot of budget in development, still be under budget but at the same time be nowhere near being able to finish the project within budget. 

I suggest considering the following alternate approach for defining the triggers for IT Governance reviews or gates. I’m assuming that no matter what, anytime scope, schedule or budget changes are needed they will be approved by governance so I’m leaving the “event” based reviews out of scope for the purposes of this post.

First, base the trigger event on budget consumption.  For example, a project manager must bring a project to IT Governance for a gate review once 20% of the budget is consumed.

 Second, create 2 or 3 models that align with various budget sizes.  For example, if a project’s budget is under $100k, then it should go through governance gate reviews at 25%, 50%, & 75% of budget consumption.  Also, the project should go through a project closure gate as well.  A second model may be for projects that are between $101k and $1M.  These projects should go through review gates at 15%, 30%, 50%, 65%, and 80% plus at project closure.  I think you get the idea.  The larger the project the smaller the % of budget consumption needed to trigger a gate review.  Therefore, more frequent reviews of larger projects.

By taking this approach you ensure that no matter what methodology is used to execute the project, governance can keep a tight control of resource consumption.

In working with IT organizations to help them improve processes related to Project or Program Offices, IT Governance and Human Resource Capacity Management I’ve found that there’s a very broad range of how a project is defined.  As such, I thought I’d post my views on this subject in hopes of helping a few out there with this issue.

First, here’s some context for my definition of a project.

The primary need for the definition I’m about to provide is to support the following IT organizational functions.

1.    IT Governance (Link to Wikipedia Definition)

a.     Determining what projects you should or should not do

2.    IT Portfolio Management (Link to Wikipedia Definition)

a.     Management of all the ideas for projects, turning some of them into business cases and presenting those business cases to IT Governance for approval

3.    IT Human Resource Capacity Management

a.     Knowing if you have the available labor force to deliver the project commitments you’re IT organization has made to your business customers

As you can see, the term “project” is a common word used in all three of these major IT functions. Therefore, how you define a project in your organization will have an impact on these three critical functions.

Within this context, here’s how I like to define a project.

1. Use a qualitative definition that targets the intent of the IT functions the definition needs to support.

For example, a Project is a work effort that …

·         Does not reoccur, or is not part of day-to-day operations

o    Meaning you don’t perform the activities/tasks every day, every week, every month or every year

§  Installing Microsoft Server Patches every month is not a project, you should plan for this activity as part of standard operations

o    There are some IT work activities that do happen once per year and if these meet the other criteria of a project I recommend treating them as a project versus operations

·         Has a start date, an end date and a defined set of objectives to accomplish

o    This gets to the intent of the work effort, you’re executing the work effort to achieve a desired goal rather than keep a server running or update an application with a minor enhancement or new report

2. Create a quantitative guideline to help make sure the qualitative definition isn’t misused to avoid IT governance

For example, a Project is a work effort that…

·         Is in excess of 40 hours of labor

·         Requires in excess of $1,000 of capital

·         Requires in excess of $1,000 of expense

This may seem low to some organizations and it’s not the numbers that I’m highlighting but rather the need for some qualitative guidelines. 

I like to use 40 hours for the labor example because it does a good job of supporting Human Resource Capacity Management.  If you make the number too large then you end up with a large number of requests that aren’t treated like projects and you lose the ability to plan how your IT human resources are used to provide services.

The key point is that the quantitative definition is a guideline and not a hard and fast rule.  For example, if you have enough servers my example of installing server patches once a month will be more than 40 hours.  However it doesn’t meet the qualitative criteria.

So in order to be called a project, a work effort must meet both the qualitative and quantitative criteria.

What you need to ask yourself when defining a project for your organization are the following?

·         How do we want to manage the usage of human resource capacity?

·         How do we want to manage the use of capital resources?

·         How do we want to manage the use of expense resources?

Answers to these questions should drive how you define a project such that work efforts meeting your project criteria are funneled to the PMO, governance and resource planning processes which in turn help you meet the objectives of these key functions.

In summary, use both qualitative and quantitative definitions to define a project that supports the objectives of the critical IT functions.

When asked this question…Are you prepared for a disaster?

….. does this sound familiar?

“Yes, we backup our data every day or week and someone takes the tape home with them.”

In providing IT advice to many small to mid sized businesses I hear this, or similar, answer often.  I felt it was time to share some thoughts regarding all of the components that should be thought about when answering this question.

The following list is a good checklist of things you should be thinking about when answering this question for your company.

1. What is the Recovery Time Objective (RTO)?

- RTO is the time it will take to bring up a disaster site/server in the event that the production site/server is destroyed by a disaster

- To answer this question, ask yourself how long you can go without your system till the impact of not having it is material

2. What is the Recovery Point Objective (RPO)?

- RPO, in laymen terms, answers the question - “How much data can I afford to lose?”

- Think of it this way, if a disaster strikes and your production servers and data are gone, how old can your backups be without the lose of data causing a material impact to your company

3. Do I backup frequently enough to meet my RPO objective?

4. Are my backups taken offsite – reliably?

5. Is my backup solution reliable – can the data be recovered when needed?

Now if you’re not thinking there’s more to Disaster Recovery (DR) than just backups then this post is for you. Backups are only one component to DR. Remember RTO (#1 in the list)? Here are some things to think about regarding RTO.

6. What server will I use to host my software if I don’t have my production server?

7. How will I access the server/software if my network is gone?

8. Where will my server be? How long will it take before I can get one?

9. What will I use to read my backup tape?

10. Will I need to install my software before I can load the data from the backup?

11. Where will the install files come from?

12. How will my software get configured like it was just before the disaster?

13. Who will do all of this work? How will they know my configuration?

Questions 6-13 highlight why backups aren’t enough.  Even if you do full backups every day and they’re automatically stored off site, you still have #’s 6-13 to worry about.  Without those questions addressed, you’ll be in a situation where you’ll have to address them at the time of disaster versus planning for them ahead of time. If that’s the case you’re looking at a best case scenario of a couple days for a simple system restore (assuming you have immediate access to servers and networks) to a couple weeks for more complex systems and perhaps longer.  This also assumes you have all the software you need to install.

My intent with this post was to highlight that backups aren’t enough to address your disaster recovery needs. You need to think about everything you need to run your software and your business. Then plan for how to address each component if the building burns down, floods, is wiped out by a tornado or some other disaster strikes.

Want to understand the risks?  Here’s a great paper that provides you with a solid appreciation for the risk you take by not planning for a disaster.

http://www.score.org/pdf/HP_Download_ImpactofDisaster.pdf.

I recently spent some time reviewing a client’s approach to software selection with the objective of looking for ways to improve the selection process.  In my experience I’ve seen the same process used by most of corporate American and I’ve used the same approach myself.

I’m sure many have the same experience of creating and categorizing a list of requirements, weighting them in terms of priority or importance and submitting them out for software vendors to respond.  Many times a short list of vendors come in to do a demo and you score how well the meet requirements.

What I’ve found is that this process will all too often fail and the results will be either the wrong selection or an incorrect setting of expectations with regard to what you are and aren’t getting from the software package.

There’s a fundamental flaw in the typical approach that needs to be solved.  The flaw is the fact that software users aren’t just interested in if the software does or does not have the functionality listed in the selection requirements list but rather explicitly how the software performs the specific function or workflow that’s required.

Selection requirements are too often summarized to levels too high to make any actual determination regarding specifically how a package meets the requirements.

My recommendation, adjust the process and the traditional order of how you implement software.  At a high level, I suggest documenting the key processes/workflows for the software package at a very detailed level.  This process should articulate exactly what needs to happen (in business process language – not what screen should be displayed) and these requirements should be used to select software.

Many times people don’t want to spend this much time up front but the reality is that you need to spend that time before you implement so you really aren’t wasting or adding time or cost to the overall process.  All you are doing is making a better decision regarding the software you select.

With these processes defined, you can have your software vendors use them to create very detailed scenario based demonstrations.  I recommend you have vendors be very clear regarding what comes out-of-the-box and what they’ve customized to meet the scenario requirements.  In fact, requiring an out-of-the-box “only” demo may be the first place to start.

Essentially, get down into the details of how a package meets key process requirements before you select the software.  Further, know if customization is needed to meet key requirements so you can set expectations regarding scope, budget and schedule accordingly. This isn’t wasted time as you’ll use the detailed documentation to configure the selected package and you have to do it at some point to get the package running.  Therefore, do it before you select software and use it in your selection criteria.

I’ve recently run into a situation that I’ve experienced many times in the past and if this post prevents it from happening only once it will be worth it for someone.  I won’t bore you with the details but the scenario I’m talking about is the lack of project planning and the impact it has on all involved.

To be a little more specific I frequently see IT initiatives get started, albeit with good intentions, with little planning and understanding of what’s actually needed to achieve the intended objective. I see this lack of planning take place both on the side of IT leaders and business leaders. Next time you see an initiative get started that seems to have lacked planning, try to get the owner to address the following issues before too much gets done.

1.    Clearly state the objective of the initiative – in simple (no consulting speak) terms

a.     I remind you that I’m a consultant, so I feel I can complain about consulting speak, it usually lacks clarity

2.    Identify who will do the work for the initiative and how they will have time to do it based on their current workload – who will take over for what they won’t be able to do?

a.     This simple concept of managing workloads is often underestimated and as a result many projects go over budget and schedule as a result

3.    Plan the work that needs to get done – before it starts happening

a.     Inputs, outputs, which tasks in which order, dependencies, etc…

                                          i.    All too often I see work planned at high levels without thinking about details. The result: Very poor utilization of time which as we all know translates into money.

4.    Plan for and provide the tools the people need to get the job done

a.     Don’t send carpenters to a job without a hammer (or nail gun these days) and don’t forget your IT resources need tools to be effective and productive

                                          i.    I often see overly optimistic estimates coupled with manual processes for requirements management, test cases, etc….

                                        ii.    If you want things done fast, create the environment needed to make it happen

5.    Plan for what happens afterwards – up front

a.     How will the new system be maintained?

b.    What are the backup and DR requirements?

c.     Does the existing operational staff have the skill, time and resources to support the new system?

I hope this helps a small number of you get your projects set up for success - at least one or two of you.

If you’ve never created a Service Catalog or are just starting to do so but aren’t sure exactly what benefit a Service Catalog provides, I hope this post will help you.

I’ve had the opportunity to create multiple service catalogs for clients and for my own IT organization.  The best way I can describe the benefits of the service catalog is that it’s a foundational component that supports many other IT management functions or processes.

First and foremost, the Service Catalog creates a standard language, that’s not tech speak, that can be used to communicate with IT customers.  Think of it as the list of services provided to customers that’s documented in the customer’s language.  I advise my clients to take the viewpoint that they now have to provide their services to the general public (not their internal departments).  With that perspective, how would you document your services on your web site for the general public to purchase?  This perspective helps create a customer view of how to document your services.

Once you have a customer centric list of services, you can now start to organize your IT functions and processes around the list of services.  For example, the service catalog can be used as a way to organize data to support the following.

1.     Knowing who (what departments/users) use what service

2.     Setting up cost allocations or charge backs based on what services are consumed

3.     Creating IT budgets organized by service versus servers, databases and other technical categories

4.     Setting up capacity (human and technical) requirements per service and the subsequent measurement of actual skill and capability to see where you have gaps

5.     Reporting performance metrics back to your customers based on SLAs organized by service

While it’s not exactly the same thing, there are some similarities between a Chart of Accounts in accounting and a Service Catalog in IT. For example, you categorize costs & revenue using a Chart of Accounts in accounting.  In IT you categorize costs, SLAs, customer usage, OLAs, Standard Service Requests, etc… using a Service Catalog. The Chart of Accounts becomes the language used by business executives to discuss details related to revenue and cost, the Service Catalog becomes the language used to discuss IT issues with customers and internal IT departments.

When thinking of the reasons why you create a service catalog, think in terms of how it supports customer communication and other IT functions.  If you’re not going to use it to support those functions then there’s little benefit to creating a Service Catalog.

Have you ever had to deal with the following issues?

·         Projects delivered late and over budget

·         Support/maintenance work conflicting with project work

o   Developers sitting idol due to the fact that the server engineer couldn’t build the dev server because he/she was handling production issues

·         Over/under utilization of human resources

·         Knowing what skills you have and what skills you need

·         Knowing if you really need to hire someone to support the new software you’re installing or not

If any of these (or similar) issues are familiar then you may have an issue with human resource capacity management.

The first step in improving these issues is to implement standard processes, procedures and structure that will accomplish the following.

1.       Capture all demand for human resources

2.       Triage the demand and route the demand to defined processes for processing

3.       Have the ability to re-route demand if the initial triage was incorrect

4.       Track demand to completion and track the time spent processing the demand

By “demand” I’m talking about the following.

·         Incidents

·         Service Requests (both standard and ad hoc)

·         Requests for Change

·         Problems

·         Events (alerts from monitoring systems)

·         System Maintenance

·         Defects/Bugs

·         Enhancements

·         New Projects or System

To solve the capacity management issues IT organizations need to implement specific processes that capture all demand.  For example, using a Service Desk or Ticket System to capture all Incidents, Service Requests, RFC’s and Problems provides you with the ability to capture most of you day-to-day operational service demand. You can also have your monitoring system automated to create tickets in your service desk system for critical events.

Leveraging a scheduling system can help manage system maintenance.  By defining standard maintenance plans for a system and entering those plans in a scheduling system you can get a handle on just how much time it will take to perform most of your maintenance work.

Using portfolio and application lifecycle systems can be used to capture demand related to bugs, enhancements and projects.

Lastly, aggregating all of this data such that you can see an enterprise view of all demand compared to all capacity provides you with the information you need to effectively manage human resource capacity.

The following provides an example what managing capacity can look like at an individual level.

1.       Work the Help Desk from M-F from 8-12

2.       Perform System Maintenance every Friday from 1-5

3.       Allocate 8 hours per week to meetings and administrative duties

4.       This leaves approximately 8 hours per week of available time to work on projects

a.       This 8 hours is only available from M-Thu because Fridays are booked solid with Help Desk and Maintenance work

b.      Now you can look at the resource estimates for your projects and assign this resource to projects that need 8 hours of work per week. Further, you instruct your PM’s to only schedule tasks for this resource on the available days (M-Thu).

If you’re managing an IT organization and haven’t looked at the ITIL framework as a baseline for how to run your organization, I highly recommend that you take the time to do so.

ITIL (Information Technology Infrastructure Library) is a set of best practices that go a long way in providing a logical/organized foundation from which you can base both the design of an IT organization as well as the processes needed to run the organization on a daily basis. You can find more info about ITIL here.

If you’ve ever been in a meeting and tried to get a hand full of IT people to agree on what a ticket, case, incident, problem or event is then you know what a struggle it can be to agree on terms and definitions.  Not to mention trying to get agreement on the detailed processes and responsibilities needed for an IT organization to run efficiently.

ITIL helps to move those activities forward much more quickly than starting from scratch because it provides a well organized structure combined with defined terms that allow you to spend time on the more important details versus arguing over what to call something or determine what list of processes you need. 

While ITIL does address much of what you need, don’t think it’s an “out-of-the-box” solution.  There are some things missing in terms of the level of detail needed to actually implement and a full coverage of all the processes you need.

We use ITIL as our baseline but find that we have to add to it in the following areas to make it effective.

1. We expand on and customize the operational processes to meet our specific business needs.
2. We add some structure & process that provide an effective mechanism to connect operational processes to application lifecycle processes.  For example, we define specific processes to manage things like enhancements, defects and new projects for application lifecycle management.  We also define processes for Incident Management and Service Requests that (when applicable) can route requests to the lifecycle management processes.  For example, sometimes a user will enter a support ticket (an Incident in ITIL terms) that’s really a functional enhancement request.  We’ve defined how the Service Desk should handle this scenario so that the request doesn’t get lost and the business customer doesn’t get the feeling that IT doesn’t listen to them.
3. We also define some processes around how to interact with the business customer to address more strategic thinking that provide us the ability to get in front of the business needs versus just reacting to them. Think of this creating a structure to what tyically takes place with ad hoc meetings or discussions. This creates a structure to the processes that feed portfolio management and IT governance.

These three areas are where I feel ITIL falls short of meeting all the needs of an IT organizations.  However, it’s much easier to address these needs when you start with ITIL and enhance it to meet your specific situation than to start from scratch.  As a consultant I’ve seen many organizations start from scratch and never achieve the improvements they’re working towards because the task of organizing what an IT organization does & how it should function can be overwhelming.

Using ITIL as the baseline puts you several steps ahead of the game and can significantly improve your chances of success.  One tip I can offer is that before you have a team of IT resources start implementing ITIL or some version of it, have them all go through ITIL training first.  This ensures your entire team is starting from the same baseline and without this you’ll burn many hours bringing those not familiar with ITIL up to speed.