Making A request to IBM

[Robin Wilson]

In article <1991Mar22.181533.21699 at ux1.cso.uiuc.edu> resnick at cogsci.uiuc.edu (Pete Resnick) writes:
>I first want to thank Robin for his last two postings. It did clarify
>some stuff. I have dealt with Robin for a couple of defects I have
>reported, and he knows what he is talking about. I think, however,

thanks Pete, but I did have to reply to this article... 

>this method of defect support for Unix workstations is just bad policy
>on the part of IBM. This is exactly the kind of support you want for
>VM machines, where the customer is usually totally dependent on IBM
>for it's system software and is usually running a limited number of
>different system level programs. But in a Unix workstation environment
>when you have 15 programs off the network and system administrators
>like me who need to get software working without having to call IBM
>at the drop of a hat, this is really unacceptable.

What you really want is bug-less code... right?   If the code always 
worked, you would never have to call IBM for anything...  then we
wouldn't be having this discussion.

>Officially, I have to deal with my SE first. I don't. Chances are
>my SE doesn't know any better what is going wrong than I do, and
>can not just drop by in the next hour all the time to help. No doubt,
>once when I was really hung using IBM's updatep program (more to say
>about this later), my SE was great; but for 99 percent of the problems
>he does not have the resources to be effective.

OK, I forfiet this one.  SEs aren't trained well enough.  If you; as the
customer, can figure out the difference between a defect and a "how-to"
then by all means, call defect support (if it is a defect).  But if they
disagree with you; remember that you will have to get your SE out to do
the PD/PSI (Problem Determination/Problem Source Identification).

>So I call defect support when I have a problem that is a defect. I
>have had both good and bad experiences here. The problems that I have
>had stem basically from the fact that there is a policy to have the
>customer talk to level 2, and have level 2 talk to level 3. Half,

I think you probably thought you got Level 3 when in fact you only got 
the local Level 2 expert... (As an example; I worked in Level "2", and
later in this article, you refer to me as a Level 3 person...)

When I worked at level 2, they had people that they called the "Response
Team".  This was a select group of "experts" in most areas of AIX.  I
was a member of the response team the last 3 months I was in Level 2.
(I am told that this system was disbanded after I left.)  Basically the
response team was used to provide a point-of-contact for the less experienced
Level 2 personnel to go to for help working problems.  Now they just have to 
go to their manager, who will appoint an expert to assist.  There are a 
limited number of "experts" at level 2, so their time is very valuable.  I
often spent 9 hrs (of my normal 10-11 hour day) on the phone with customers
helping out on other Level 2 people's problems.  The other 'experts' had
similar schedules.  Just try to imagine how hard it is to hold a phone to
your ear for several hours at a time... 

Granted, there are more "experts" in Level 3, and for most problems there
would be a significant speed increase if Level 2 just moved you straight 
to the Level 2 expert, or the Level 3 expert... But this just isn't
feasible.  Would you rather have level 3 talking on the phone, or coding
the fixes?  Basically the problem is a lack of resources.  If IBM could
hire 100 more "experts" to man the phones, then there would be no 
problem.  But IBM can't afford to hire 100 MORE people, much less experts
to man the phones.  The more people they hire, the more the box costs...
The people they have will take time to become 'experts'.  Then when they
do, they will eventually move on; meaning there is a natural attrition 
from phone support; so they will be replaced with NEW people, who will 
have to learn all over again.

Part of the problem is the level of acceptance of the product.  IBM never
expected the RS/6000 to take off like it did.  They expected a slow 
methodical build up of sales.  Instead, sales skyrocketed right out of the
chute, causing a huge overhead expense for staffing support; and that
expense was not expected, so staffing took a little long.  Now, I believe
they are nearly adequately staffed for the number of problems that they 
have, but it will take several months before the existing staff is 
adequately trained to handle the most complex problems.  Give them a 
little more time, and see if things don't improve.

>Sometimes, level 2's diagnosing techniques are annoyingly rote:
>if you have a TCP/IP problem, someone is going to ask your for
>your /etc/hosts file. Not that it has anything to do with the
>problem, but they ask for it anyway. I have heard the same questions
>20 times for a lot of defects because some, and only some, level 2
>people are not asking logical questions; it sounds like they read out
>of a manual. I have had good experiences with people at level 2,
>but the bad ones stick out more.

Well, I would expect the bad ones to stick out more.  And they should too.
As for "reading out of a manual"; funny you should mention that... For 
network problems of ALL KINDS, it is virtually impossible to find experts
who are willing to talk on the phone 8 hrs a day.   They make too much 
money working elsewhere, doing other more meaningful jobs.  That means 
that most of the Level 2 network support (with 3 or 4 exceptions only) are
home grown.  This area will take a long time to bring up to speed.  
You know that networks are one of the hardest things to understand within
the 'computing arena'.  So learning the nuances of networking will take
time.

>Then, once a problem is addressed, I get an update. I get a disk
>that says "Use updatep command", no clue of what is ever being
>changed, no bug fix list except the APAR list, nothing. Not only
>that, I only get updates when I ask for them, and the production
>to install them borders on the unbelievable. Again, this method
>is great for a VM machine, but not for a Unix machine that I am
>taking care of. I want to know what's happening, and be able
>to back off if I need to. (*No, applying updates without comitting
>them does not always work, and you can not do more than one
>change, test them together and then reject.*)

I can't really address this other than to say; thats the way it works.
That's basically why IBM suggests (strongly) that you make a backup 
before doing the update.  You can restore whatever you want.  Also,
the System Xtra offering provides a "bug report" for each interested 
customer (I think...).

>IBM's defect support is not oriented for Unix workstations. This
>is a problem of philosophy. It takes alot of change to get me
>to talk live to a "level 3" type person quickly, but that is what
>is necessary in most cases. I think the savings of time and money
>would be incredible.

On the contrary, the costs would be enormous.  It would save time
until IBM tried to actually start fixing the problems that were reported.
Then it would be a nightmare to find someone who wasn't on the phone to
a customer.  Some people have this image of RS/6000 people in the 
thousands down here in Austin... But in reality there are less than 300
people in Level 2 and Level 3 combined for the RS/6000.  Take the number
of calls (around 300 per day), extend the problems that take several 
conversations (callbacks) and you have about 10 people left to actually
code fixes.

On the RT, the numbers are even more meager.  There are 5 Level 2 people
and 7 Level 3 people.  The RT calls come in much slower (thank god) but
there just aren't that many people available.

Its not a perfect system.. no doubt.  But they are still working on it.
The live call support is New for the RS/6000.  The organization is based 
on the lessons they learned from RT support (they have corrected many
mistakes) and will take time to completely iron out.  After all the product
is not even officially a year old.

>End of pontification. Thanks again to Robin, one of the level 3
>people I have actually talked to, and the bunch of level 2's that
>have helped me in the past. This is certainly not a flame of the
>people in defect support; most of them are rather smart people.

Again... I was Level 2.  

>This is a flame of a system that is just poorly operated for a
>product that it was never designed for.

The system was designed completely from scratch for the RS/6000.  They
still have a lot of tweaking to do, but I think in the end they will
have a 'good' (if not great) system.


+-----------------------------------------------------------------------------+
|The views expressed herein, are the sole responsibility of the typist at hand|
+-----------------------------------------------------------------------------+
|UUCP:     pensoft!robin                                                      |
|USNail:   701 Canyon Bend Dr.                                                |
|          Pflugerville, TX  78660                                            |
|          Home: (512)251-6889      Work: (512)343-1111                       |
+-----------------------------------------------------------------------------+