Recently we upgraded to a new version of HP SIM (systems insight manager), of course not testing it or letting the DBA's know.  Suddenly some things crash.  The new version of HP Sim provides a "richer discovery model"; oh it's rich!

Seems the new version performs some type of scan on the scsi bus, which causes our multi-path software (EMC Powerpath) to loose connectivity to the SAN, this causes the file system to "Panic", and all filesystems unmount.  Nice.

SIM is a hardware monitoring solution from HP for HP servers.  The server administration team loves it, the SQL DBA's don't mind it.  It of course uses SQL Server for a back-end database, so it helps keep us employed as well !  Basically SIM provides hardware inventory and monitoring of the servers.

Specifically: 

Version:  Systems Insight Manager 5.3 with SP1 - Windows
Build version:  C.05.03.01.00 

Using the Manual Discovery Task that ships with the product.

The issue was most notable with our clustered servers, especially Polyserve.  Below were the error messages:

Work closely with your administrators and be careful of how these monitoring solutions will affect your production servers.

Kansas City Natural Wheat

• 6 lbs.  Weyermann Pale Wheat
• 5 lbs.  German Pilsen
• 1 Oz. Perle leaf hops, 7.9 AAU, bittering, 1 hour
• 1 Oz. Saaz leaf hops, 6.7 AAU, 30 minutes
• 1 Oz. Styrian pellets, 1.9 AAU, 15 minutes
• Wyeast Weihenstephan Weizen 3068

Mash schedule

• 153 degrees F for 60 minutes
• 170 degress F for 10 minutes

Ferment, Rack onto secondary with 1 pound belgian white candi sugar, or 1 pound corn sugar, dextrose, complete fermenation, prime and bottle.

SQL Server SAN Migration

I think this is my 3rd or 4th SAN Migration casued by:
 - completely moving data centers
 - changing the storage backends to different vendors
 - consolidating SANS
 - growing to a bigger san to consolidate more

There are several different scenarios to consider
 1.  Stand alone SQL Servers Instances on Internal Disk to SAN
 2.  Stand alone SQL Servers on SAN Disk (changing sans).
 3.  VM Ware SQL Servers (required to be on SAN)
 4.  Microsoft Clustered SQL Servers
 5.  Polyserve Clustered SQL Servers
 6.  BCV's / SAN Mirroring / replication technologies
---------------------------
The most important thing to remember is to backup.  The next most important thing, no matter what the SAN Engineers, Windows Engineers or Vendors tell you about SAN Migration, YOU AS THE DBA ARE RESPONSIBLE.  Understand the migration plan for each scenario, regardless of what anyone says, the DBA is always left holding the bag.  If you don't understand the migration plan / scenario, make them explain it, learn it, try and practice it.  I'll explain with an example. 

One of the important items in most scenarios is updating to latest drivers and versions of software, even this step can be dangerous.  In a recent effort to patch servers to the latest version of drivers the SAN disk just "disappeared", when it came back, there was NO DATA.  No amount of research could yield what happenned to the data.  Refer back to the most important thing!
----------------------------

All of the different scenarios are simple, with careful preperation and a good windows and SAN team. 

1.  Stand alone SQL Servers instance on Internal Disk to SAN.

Usually the most difficult thing here is that you are taking an existing stable server and adding a lot of new complexity to it.  Drivers and hardware for the SAN, sometimes this does not always go well, if possible I try and get new hardware and completely swap the machines, configuring the new machine in advance for the SAN and installing sql; taking several dry runs to ensure it's stable.  The next issue is the down time in copying the data from the internal disk to the san.  Than switch the drive letters and start SQL.  Don't forget the most important thing.

2.  Stand alone SQL Servers on SAN Disk (changing sans).

Here is where having a good windows and san team can help you.  Most of the migrations I've been involved with the Windows and SAN team will setup a mirror between the SANS.  Than on migration day we stop SQL Server, the windows and san team ensure the mirror is up to date, than split the mirror and hook the server up to the new storage and ensure the drive letters and mount points come up.  SQL DBA restarts sql, boom, your done.

Sometimes if your switching san vendors, you can't setup a mirror.  Now things get dicey.  Can you get "hooked" up to both sans simultaneously, of so than your ok, stop sql, copy the data to the new san, reset drive letters and mount points, restart SQL. 

If you can not get "hooked" up to both sans simultaneously than you need to default to some type of backup and restore mechanism, tapes or copying sql files to local disk (if you have room).  I'm usually not a fan of this, as I find that different vendors use different drivers, switching vendors means now you have both vendors drivers on the machine, the machine decays and becomes unstable, NOT FUN.  Don't forget the most important thing.

3.  VM Ware SQL Servers (required to be on SAN).

These have been my easiest san migrations.  That's because we have awesome dedicated VM Ware administrators.  They do it all, just schedule the outage.  But trust me, Don't forget the most important thing, check your backups went to tape, double check your Disaster Recovery plan.

4.  Microsoft Clustered SQL Servers

I've only done Microsoft Cluster migrations with SAN Mirrors, and it has been uneventful.  This is because the "mirrors" end result is the preservation of clustered resources (quorum), drive letters etc.  But don't forget the most important thing.  Again a great windows and san team makes this easier.  I'd fret to switch vendors on the Clusters, but if I ever do I'll update this post.

5.  Polyserve Clustered SQL Servers

This is the SAN Migration worst case scenario, "The clustered file system".  Below are the steps we followed for migrating sans with Polyserve.  Don't forget the most important thing.

-Dump vsql and vsqlinstance information from cluster
 -mx vsql dump >> vsql.txt
 -mx vsqlinstance dump >> vsqlinstance.txt
-Get a listing of all storage by copying the grid on storage summary to excel
-stop and disable all SQL instances and VSQLs
-copy the virtual root for each sql server instance to another server (outside the cluster)
-deport ALL dynamic volumes (paths are automatically unassigned)
-stop cluster services on all cluster nodes
-copy the entire c:\polyserve directory to another server (outside the cluster) for each machine (CYA)
-manage the storage to unpresent all LUNs from the old array
-break the mirror relationships and then present all of the mirrored LUNs
-create three new 1GB LUNs on the new array and present them for new mem parts
-put partitions on the three new LUNs
-go into the config utility on node 1 and delete old membership partitions and add the three new membership partitions
-start cluster services on this node
-export the config to other nodes and start the service on the rest of the nodes
-import all importable dynamic volumes
-assign paths
-enable instances and vsqls
-done
Polyserve SAN Switch.doc (29.5 KB)

 

6.  BCV's / SAN Mirroring / replication technologies

Administring the advanced SAN technologies is different for each vendor and quite proprietary.  You definetly want to test and work with each one individually and ensure it all works.  The details of this are far outside the scope of a simple blog post, but having great SAN engineers will make this easy, as they generally setup the mirrors, clones and movement of them to different machines or remote locations.

Don't forget the most important thing.

I'm in the middle of an all-nighter for a SAN Migration.  Lots of down time, here's what I found worth reading:

The unspoken truth about managing geeks.

Understanding Service Level Agreements for Database Development.

Is SQL Server's latest security hole a real threat?

Very interesting syntax of the in-clause I didn't know existed:

SELECT * FROM person.contact

WHERE 'thomas' in(Firstname,LastName,LEFT(emailaddress,7))

From the I did not know this would work department

Big competitors spark Netezza’s partner-friendly strategy

Keeley Hazell looks hot while she's shilling for something.

Don't ask me how I'd go from Keeley to Jeremiah Johnson, but all nighters due that to you, than and i've never been normal, it's boring.

"where ya headed?"
"Same place you are Jeremiah, Hell, in the end."

Link to download sound track to the movie, Crow Corner., a long time ago I had it on LP, doubt I'd ever find it on cd or mp3, but all night'ers have a way of helping you find things.

There are many emphases for the SQL DBA, one of the major items is security.  Where I work security and patching have taken on new importance over the past 12 months.  Patching which used to be a yearly event, is now monthly or even weekly.  Combine that with SOX Controls and other internal controls, there is a focus on security.  Part of that is seen in SQL Server from Microsoft, kudos! 

The product gained a new level of security with SQL Server 2005.  Service Pack 2 gave us Login Triggers / Service Broker Events.  SQL 2008 saw the Builtin\Administrators account gone (separation of duties) and even more controls.

Than today it was distressing to see a news report about Microsoft ignoring a vulnerability with SQL Server, basically dismissing it as anyone with administrator privileges already has control [What happenned to separation of duties?].  The whole point of where security with SQL Server was going was to ensure that administrators could be properly segmented.  Where I work they'd like to get to the point where a SQL DBA can't even view or query the data that they administer.

If you find the below security vulnerability to be an issue than complain to Microsoft.  Also if you didn't know about this vulnerability, than you need to ensure to plug-in to some other industry news sources to get your information on vulnerability and patches, as you can't always count on Microsoft to disclose issues.

-----Original Message-----
From: Security Wire Daily [mailto:SearchSecurity@lists.techtarget.com]
Sent: Wednesday, September 02, 2009 10:38 AM
Subject: New SQL Server password flaw surfaces

::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
SearchSecurity.com: Security Wire Daily
Breaking security news, the latest industry developments and trends
September 02, 2009
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

UNPATCHED VULNERABILITY DISCOVERED IN MICROSOFT SQL SERVER
Michael S. Mimoso, Editor, Information Security magazine

Microsoft SQL Server administrators are being warned today about an
unpatched vulnerability in the popular database software that exposes
user passwords in the clear, as well as credentials delivered by
applications trying to access the database server.

Researchers at San Mateo, Calif.-based Sentrigo Inc., announced the
flaw this morning, and also revealed that Microsoft has no immediate
plans to release a patch for the vulnerability. Sentrigo, meanwhile,
said it has developed a free utility that will erase these passwords
from memory.
Read more:
http://go.techtarget.com/r/9124859/8704472