New Tool: VNXArray Python Module

After being inspired by Clint’s work, I decided to built something very similar – a Python module.  It connects with naviseccli to any given VNX array, collects 90% of the relevant data and compiles it into a set of true Python objects.  Why Python?  Because its better than PoSH, because its cross platform .  I plan to extend this further with information about tiers, etc, but this is a start.

myArray = VNXArray("1.1.1.1","sysadmin","sysadmin,0,True) print myArray.controllerPair.model ------ VNX5500

or, a little more complex:

for disk in myArray.disks: print disk;

Disk
 {'ID': u'Bus 0 Enclosure 0  Disk 0',
 'capacityMB': u'274845',
 'currentSpeed': u'6Gbps',
 'driveType': u'SAS',
 'lun': u'77 81 ',
 'maximumSpeed': u'6Gbps',
 'numLuns': u'2',
 'numReads': u'7535989',
 'numWrites': u'4122963',
 'private': True,
 'productId': u'ST930060 CLAR300',
 'productRevision': u'FF03',
 'raidGroupId': u'0',
 'serial': u'3SE2DTK5    ',
 'state': u'Enabled',
 'type': u'77: RAID1/0 81: RAID1/0 ',
 'userCapacity': u'5',
 'vendorId': u'SEAGATE'}
 Disk
 {'ID': u'Bus 0 Enclosure 0  Disk 1',
 'capacityMB': u'274845',
 'currentSpeed': u'6Gbps',
 'driveType': u'SAS',
 'lun': u'77 81 ',
 'maximumSpeed': u'6Gbps',
 'numLuns': u'2',
 'numReads': u'7413955',
 'numWrites': u'4138719',
 'private': True,
 'productId': u'ST930060 CLAR300',
 'productRevision': u'FF03',
 'raidGroupId': u'0',
 'serial': u'3SE2DR0Y    ',
 'state': u'Enabled',
 'type': u'77: RAID1/0 81: RAID1/0 ',
 'userCapacity': u'5',
 'vendorId': u'SEAGATE'}

for lun in myArray.luns: print lun;

LUN
 {'Name': u'multipiece datastore lun_2',
 'RAIDType': u'N/A',
 'capacityMB': 10240,
 'currentOwner': u'SP A',
 'defaultOwner': u'SP A',
 'number': u'56',
 'prctBound': 100,
 'prctRebuilt': 100,
 'private': True,
 'raidGroupID': u'N/A',
 'state': u'Bound',
 'uid': u'60:06:01:60:34:50:29:00:16:BC:FD:54:6A:71:E1:11'}

Get it here:

VNXArray.py.zip

vCenter Ops (vCops) Data Extraction Tool 1.0

This morning I’m pleased to release the vCops Data Extraction Tool 1.0

Have you ever wanted to extract raw data from your vCops installation.  By raw I mean really raw – each and every timestamped value for each metric and resource?  I had such a need recently for an upcoming secret project, so I decided to write such a tool  Because I wanted this to be cross platform, my previous work with Clint Kitson based on Powershell was out.  So, my backup language is always Python, and I chose that.

I’m posting the code here under a Creative Commons license for everyone to use as needed.  As an example of how to run it (it does include help if you need it):

python vcopsextract.py –oraclehome /u01/app/oracle/product/11.2.0/xe/ –sqlplus /u01/app/oracle/product/11.2.0/xe/bin/ –user vcops11 –password usr4vcops –oracleSID XE –oraclehost localhost –outputfile data.out –debugfile debug.out –vcopsurl http://localhost/HttpPostAdapter/OpenAPIServlet –days 1

Most of this just specifies the Oracle database (which I need to connect to and query for the resource metric names, etc), the output files, the OpenHTTPAdapter location and the number of days of data to retrieve.  I highly recommend restricting the number of days of data to retrieve to a reasonable number like 7, as the data generated by this tool is quite significant.  As an example, in my small vCenter + VNX demo environment with about 566 resources, 7 days of data was well over 8GB of text files.

When it executes it looks something like this:

 

 

 

Note the pretty progress bars

When its complete, you’ll see in the datafile:

The first like is a token indicating when this file was created (milliseconds since Epoch).  The second line is the vCops-style information about the resource, including its name, Adapter Type, Identifiers, etc.

The following lines are the metrics for each individual metric with:

metricname,alarmlevel,<blank>,timestamp,value,<blank>

This format is intended to be easily reimported to vCops later.

TODOs:

• Enable gzip’d output
• Enable selection only of certain resources
• Enable selection only of certain metrics
• Other?

Download Here

Getting the AdapterKindKey & ResourceKindKey for every vCops Resource

Just a quick post today as I’m working on a secret project.  I had a need to identify the AdapterKindKey & ResourceKindKey for every resource in vCops.  Thats not as easy as it sounds, because the OpenHTTPAdapterAPI doesn’t allow you to just list resource – you have to know what resource you want, along with its KindKeys before you can get its metric data.  As a result, you have to get the information from the SQL database directly.

Because these are in a database schema thats pretty solidly normalized, so we have to do a couple of SQL joins to get all the information back in one query.  In my case, I wanted the resource Name, ResourceKindKey and AdapterKindKey all together.  

The query ended up looking like this:

 

SELECT Name,RESKND_KEY,ADAPTER_KEY FROM AliveResource,ResourceKind,AdapterKind WHERE (AliveResource.RESKND_ID = ResourceKind.RESKND_ID AND AdapterKind.ADAPTER_KIND_ID = ResourceKind.ADAPTER_KIND_ID);

 

So how did I get to there.  First, I simply took a look at the AliveResource table to see what I got:

SELECT * FROM AliveResource

 

RESOURCE_ID RESKND_ID NAME INFORMATION CRED_ID ATTRPKG_ID COLLECTOR_ID MONITORING_INTERVAL LAST_MAINTAIN_TIME MAINTAIN_UNTIL maintenanceScheduleId TYPE Latitude Longitude CURRENT_HEALTH BUSSVC_ID TIER_ID SMPackageId CreationTime SUBTYPE EarlyWarningAlertEnabled DT_ENABLED FLAG

1 1 vCenter Operations Adapter (vCenter Operations Server) null null 2 1 5 null null null 7 null null 0 null null null 1324564710237 2 null 1 null

2 3 vCenter Operations Collector (vCenter Operations Server) null null 6 null 5 null null null null null null 0 null null null 1324564710294 2 0 1 null

 3 2 vCenter Operations Analytics null null 4 null 5 null null null null null null 0 null null null 1324564710343 2 0 1 null

Wow, thats a lot, but I see that I have alot of what I want, including a Name column and a RESKND_ID column, which would give me the ResourceKindKey (a string) from another table.  So lets limit the query a bit more:

 SELECT Name,RESKND_ID FROM AliveResource;

Name RESKND_ID

vCenter Operations Adapter (vCenter Operations Server) 1

vCenter Operations Analytics 2

vCenter Operations Collector (vCenter Operations Server) 3

 vCenter Operations Message Queue 4

 

OK, thats a little better and cleaner.  But now what I want to do is use the RESKND_ID (which is just a number) to get the ResourceKindKey for that ID from the ResourceKind table.  This is called a ‘join’, more specifically its an INNER JOIN.  For the math nerds out there like me, its a Cartesian product of the two tables.  So what I need to do is specify to the SQL engine which parts of the table are the same thing.  Specifically, that the AliveResource.RESKND_ID values and the ResourceKind.RESKND_ID are the same thing, and would should combine based on that.  The query looks like this:

SELECT * FROM AliveResource,ResourceKind WHERE AliveResource.RESKND_ID = ResourceKind.RESKND_ID

 

Name RESOURCE_ID RESKND_ID NAME INFORMATION CRED_ID ATTRPKG_ID COLLECTOR_ID MONITORING_INTERVAL LAST_MAINTAIN_TIME MAINTAIN_UNTIL maintenanceScheduleId TYPE Latitude Longitude CURRENT_HEALTH BUSSVC_ID TIER_ID SMPackageId CreationTime SUBTYPE EarlyWarningAlertEnabled DT_ENABLED FLAG RESKND_ID RESKND_KEY ADAPTER_KIND_ID NAME_ID TYPE SUBTYPE ICON SHOW_TAG ATTRPKG_ID SMPackageId DYNAMIC DT_ENABLED EarlyWarningAlertEnabled

vCenter Operations Adapter (vCenter Operations Server) 1 1 vCenter Operations Adapter (vCenter Operations Server) null null 2 1 5 null null null 7 null null 0 null null null 1324564710237 2 null 1 null 1 vCenter Operations Adapter Instance 1 10 7 null 1.png 0 null null null null null

vCenter Operations Collector (vCenter Operations Server) 2 3 vCenter Operations Collector (vCenter Operations Server) null null 6 null 5 null null null null null null 0 null null null 1324564710294 2 0 1 null 3 Collector 1 3 null null named_Collector.png 0 null null null null null

vCenter Operations Analytics 3 2 vCenter Operations Analytics null null 4 null 5 null null null null null null 0 null null null 1324564710343 2 0 1 null 2 Analytics 1 2 null null named_Analytics.png 0 null null null null null

vCenter Operations Web Server 4 5 vCenter Operations Web Server null null 10 null 5 null null null null null null 0 null null null 1324564710371 2 0 1 null 5 Web 1 5 null null named_Web.png 0 null null null null null

Whoa, thats a lot of data, so lets narrow it down by just selecting the columns I want:

SELECT Name,RESKND_KEY FROM AliveResource,ResourceKind WHERE AliveResource.RESKND_ID = ResourceKind.RESKND_ID

 

Name RESKND_KEY

vCenter Operations Analytics Analytics

DISK ARRAY_SUMMARY

LUN_EFDCACHEADMIN ARRAY_SUMMARY

LUN_FLARE ARRAY_SUMMARY

LUN_MLUADMIN ARRAY_SUMMARY

SP ARRAY_SUMMARY

VMware vCenter Operations BusinessService

Aha, thats a little better.  But I also wanted the AdapterKindKey too, and thats in a third table.  Well, another JOIN is certainly allowed (albeit computationally expensive, as we are talking about a 3D Cartesian product here).  However, I have fast computers and this is a pretty small database, so lets give it a show, this timeing joining ResourceKind.ADAPTER_KIND_ID and AdapterKind.ADAPTER_KIND_ID along with my previous join.  Also, we’ll only select the columns we want to keep it clean:

SELECT Name,RESKND_KEY,ADAPTER_KEY FROM AliveResource,ResourceKind,AdapterKind WHERE (AliveResource.RESKND_ID = ResourceKind.RESKND_ID AND AdapterKind.ADAPTER_KIND_ID = ResourceKind.ADAPTER_KIND_ID);

 

Name RESKND_KEY ADAPTER_KEY

vCenter Operations Adapter (vCenter Operations Server) vCenter Operations Adapter Instance vCenter Operations Adapter

vCenter Operations Analytics Analytics vCenter Operations Adapter

vCenter Operations Collector (vCenter Operations Server) Collector vCenter Operations Adapter

vCenter Operations Message Queue MQ vCenter Operations Adapter

vCenter Operations Web Server Web vCenter Operations Adapter

Http Post (vCenter Operations Server) Http Post Adapter Instance Http Post

VMware vCenter Operations BusinessService Container

Et voila!  We have what we wanted!  Neato!

 

 

 

Installing the vCops Symmetrix Collector Plugin

My colleague, Tommy Trogden, recently asked me if I could demonstrate the native Symmetrix plugin for vCenter operations. This is my favorite kind of request, because it means I get to try something new .

One thing I like about how this collector was designed is that it uses the existing TTP (Text Transform) performance files that the Symmetrix Solutions Enabler application already outputs, making it easy for existing Symmetrix admins. As a result, I won’t go into installing SMC/Solutions Enabler, because most shops will have that setup. If anyone wants, in the future I could do a post on it.

Without further ado – the installation process for the Symmetrix adapter for vCenter Operations.

1) Download the plugin itself from your VMware representative (its not currently available on the website). Also download the installation tech note at the same time.

2) Because my vCops server runs on Windows, and my SE system is Linux, I need a way to access the Linux filesystem from Windows. I chose to use SMB (Samba) to do this, simply because its easy to setup.

3) First, we need to configure an Samba export on the Linux side. Specifically, we need to export the directory with the TTP files, which are in (by default) /var/symapi/stp. So lets share that out. First, lets make sure the Samba service is running:

[root@smc samba]# /etc/init.d/smb start

Starting SMB services:                                     [  OK  ]

Starting NMB services:                                     [  OK  ]

Then lets configure it by adding the following to /etc/samba/smb.conf

[global]

    security = share

    ;passdb backend = tdbsam

    guest account = root

    ;yes, this is horrible, but this could be fixed in a production environment with some clever umask settings on the linux side

[symapi]

    comment = TTP Files

    path = /var/symapi/stp/

    public = yes

    writable = yes

    printable = no

    guest ok = yes

    browseable = yes

We then restart SMB:

[root@smc samba]# /etc/init.d/smb restart

Shutting down SMB services:                                [  OK  ]

Shutting down NMB services:                                [  OK  ]

Starting SMB services:                                     [  OK  ]

Starting NMB services:                                     [  OK  ]

4) Now we need to configure the STP daemon in solutions enabler to capture what we need/want. Specifically, the following options in the in /var/symapi/config/daemon_options need to be changed:

storstpd:dmn_run_ttp = enable

storstpd:ttp_collection_interval = 1
storstpd:ttp_rdflnk_metrics = enable
storstpd:ttp_se_tcp_metrics = enable
storstpd:ttp_se_nw_metrics = enable

storstpd:ttp_dev_metrics = disable
storstpd:ttp_dgdev_metrics = enable
storstpd:ttp_se_tcp_metrics = enable
storstpd:ttp_se_nw_metrics = enable
storstpd:ttp_se_nwi_metrics = enable
storstpd:ttp_re_sg_metrics = enable
storstpd:ttp_re_nwc_metrics = enable
storstpd:ttp_rdflnk_metrics = enable
storstpd:ttp_se_tcp_metrics = enable
storstpd:ttp_se_nw_metrics = enable

Save the file, then we restart storstpd to make it reread the config:

[root@smc config]# stordaemon start storstpd

  Waiting for daemon to start.  This may take several seconds.

[root@smc config]# stordaemon list

Available Daemons  ('[*]': Currently Running):

[*] storapid              EMC Solutions Enabler Base Daemon

    storgnsd              EMC Solutions Enabler GNS Daemon

    storrdfd              EMC Solutions Enabler RDF Daemon

[*] storevntd             EMC Solutions Enabler Event Daemon

[*] storwatchd            EMC Solutions Enabler Watchdog Daemon

    storsrmd              EMC Solutions Enabler SRM Daemon

[*] storstpd              EMC Solutions Enabler STP Daemon

[*] storsrvd              EMC Solutions Enabler SYMAPI Server Daemon

Once its running, we can check that sep directory from before and we see some new contents:

[root@smc ttp]# ls -la /var/symapi/stp/

total 16

drwxr-xr-x  4 root root 4096 Dec 21 14:55 .

drwxr-xr-x 14 root root 4096 Jun  8  2011 ..

drwxr-xr-x  3 root root 4096 Jan  5  2011 spa

drwxr-xr-x  3 root root 4096 Dec 21 14:55 ttp

We have a new ftp directory – win!

Now, the last part is we need to force the STP daemon to archive the files off every 5 minutes, by adding the following cron job:

[root@smc 000194900728]# crontab -l

2-57/5 * * * * /opt/emc/SYMCLI/bin/stordaemon action storstpd -cmd archive  > /dev/null

5) Now, install the adapter into your vCops server:

Run the downloaded executable:

And follow all the defaults.

6) Now, restart the vCops service using standard methods.

7) Now, lets setup the collector.

Go into the adapter instances part of vCops:

Then select the adapter type and hit the ‘+’ button:

Fill in the required information:

And test to make sure it works:

NOTE: You CANNOT use a mapped driver letter, because services cannot access them in Window (pshaw! Use Linux!)

8) Now, we wait for the files to get processed. Give it 15 minutes or so to really gather some reasonable data and import it.

9) Take a look at the environment detail, and you should see your arrays:

And the detail for it:

Seems like there is plenty of data being collected, including throughput of the FEs, BEs, individual ports, cache stats and the RDF link performance. Cool! On my (pretty small) Symmetrix VMAX-SE, I have almost 1000 metrics being collected.

As for what the dashboards from this data can look like, I’ve created a few samples. First, its important to monitor the health of the cache in a Symmetrix – specifically to monitor the number of write pending tracks compared to the max write pending limit. Here’s a dashboard showing that. Clearly, this is pretty healthy.

Next, we look at the total IOPs and Read / Write throughput on the array, just to get a sense of how hard its working overall:

Last, we look at the FA ports supporting my VMware environment, to make sure they aren’t getting killed:

Using VMware IO Analyzer

As a general storage nerd, I’m always interested in new benchmarking tools, especially if they can make it easy to help get the numbers people need.

VMware’s new fling (e.g. unsupported experimental software) called IO Analyzer is pretty cool. Quickly, its a preconfigured virtual machine that gives you a web interface to run performance benchmarks using IOmeter (excellent software) and collect the relevant performance statistics, all in 1 clean interface. I like it.

While it does come with instructions, I thought I’d put up a post on how to install it and run through a first benchmarking session.

• First, download the IO analyzer OVA from VMware’s Flings site.
• Next, deploy the OVA just like any other Virtual Appliance:

As the installation instructions state, select thick provisioned disks for the best performance evaluation:

• Once the virtual machine appliance has been uploaded and powered on, give it about 2 minutes to properly boot for the first time. Then, open the virtual machine’s console:

If you are like me, you don’t have a DHCP server on your network, so you will need to setup manual networking. If you do have a DHCP server and the VM picked up a reasonable IP address that can access your ESXi hosts, then skip ahead . If not, follow along:

After the networking is configure, you MUST login to the console of the VM, or nothing will work. Do so now using the default password and username (root / vmware). You will end up with a grey screen – this is normal:

Now, open up your web browser and shoot over to the IP address of your VM (note, don’t use the URL that the text GUI spits out, it doesn’t work):

Click on the Setup/Run section:

Add the hostname or IP address of the host that your appliance is running on, along with credentials to access. IO Analyzer needs to used this in order to run resxtop against that host during the test to collect data. Then, add the VM’s information:

The key here is to select a Workload spec that matches your needs. I tend to like the ones listed at the bottom, specifically the 8k and 4k OLTP workloads – they are reasonable realistic for server type workloads. If you have a different workload (e.g. VDI), you’ll need to choose something else. Developing a workload spec is something we’ll tackle in a future article).

Enter a reasonable run duration (at least 2 minutes – preferably more like 30 minutes) and hit Run



While the test is running, if you are super curious, you can take a look at the VM’s console and watch IOmeter running:

I’d advise against playing with it while its running

When it completes, the appliance will let you know:

Do as it says and click the results tab:

And click View Summary for your test:

Excellent! As you can see, my LUN was able to support 25,000 IOPs at about 100MBit on a Database workload. Great to know! Also, we can see that it basically only used vmhba3, which is a good hint for me that I need to check my patching and ensure I’ve got proper balancing setup (PowerPath/VE to the rescue!).

Hope you enjoy!

Manually Aligning a VM the Hard Way

Some of you may know that the EMC VNX VSA was the virtualized storage used on most of the VMware Hands On Labs.  Its was a great chance to really help out our VMware brethren.

However, there was a problem, one that we didn’t catch.  Vendors are made up of humans, and none of us is perfect, including EMC.  Our friends over at NetApp noted (via Nick Howell’s blog) that they had discovered that the VNX VSA was not a properly aligned VM (e.g. it wasn’t aligned on a sector boundary).  Nick mentioned that he was going to clarify how they determined it (Nick?  Maybe you can write that post? Hint! Hint!), but in the meantime, I’d like to thank them (NetApp) for helping us out and letting us know.  While we may be competitors, when it comes to the VMworld HoL, we can (and do!) put aside our differences and just help out.

So I went about fixing that oversight, because VM alignment is important and should be fixed, especially when running hundreds of them like at the Labs.  I would have used Nick Weaver‘s awesome UBERAlign tool, but it doesn’t support LVM volumes quite yet, which the VSA uses.  So I had to go after it manually.   I also couldn’t do live (for obvious reasons – you can’t realign while powered on).  Here was the process I undertook:

Examine the File System – The first order of business was to look at the configuration.  The easiest way to do this was to attach the VM’s disk to an existing Linux VM as a secondary disk:

1. 
   This way, I can make changes to the disk without having to do it live.  In this case, I noted that the single disk on the VM was broken into 2 partitions using the ‘fdisk’ command:

   There was a single partition marked as bootable (/dev/sdc1 in this case) and a second marked as lvm (/dev/sdc2).   Note that neither starts on a 512-byte sector boundary – they are unaligned.  This is a pretty common Linux structure, and usually means that the boot partition contains basically grub and the kernel (usually ext2 or ext3), and the second partition contains one or more LVM logical volumes.  As a first step, I mounted /dev/sdc1 (the boot partition) at /tmp/boot so that I could back up its contents using the mount command:

   mount /dev/sdc1 /tmp/boot

   Then I investigated the LVM volume.  I ran ‘pvs‘, which looks for LVM Physical Volumes (its short for pvscan).  I saw a single Physical Volume (as expected):

   Then, I ran vgscan to search for volume groups. Because this is a pretty simple VM, there is only one volume group (VolGroup00):

   

   Further, ontop of that volume group, there are 2 Logical Volume on top of that Logical Volume:

   

   Lets figure out what those are (just try mounting them):

   OK, now we know that one of them is the root filesystem (ext3) and the other is the swapspace. Pretty Cool!

2. Copy Off Data – While it can be possible to do this in-situ (UBERAlign can do this!), I find it easier and clearer to do this against a new disk.  By doing it this way, we also reduce the fragmentation in the filesystem and increase the free/zero space, making the VM as small as it can be for best performance.  So, now that we have both the boot and root filesystems mounted, we are just going to copy off the data.  We are going to use rsync for this, because it gives nice status updates, is fast and supports sparse files efficiently (which are used in the VSA).  We copy the data like this:

   rsync -avS /tmp/boot/* /backup/boot/
   rsync -avS /tmp/lv0/* /backup/root/

   In my case, this took about 10 minutes, but it depends on how fast your storage is, and how much data you have.

3. Recreate Partition Table – We now need to recreate the partition table in such a fashion that its aligned.  In short, we need to make sure that partitions start on a 512 byte boundary, which fdisk doesn’t do by default (ugh!).  So lets use GNU parted to modify this partition table:You’ll get a number of errors about informing the kernel, but this is normal.  At this point, its best to reboot your helper VM, just so we can clear out any of the LVM state (its possible to do this live, but its a pain).How did I choose the starting values of 4 and 120?  Well, I chose them because I used parted’s ability to set the units:units MiB

   This ensures that all future numerical values are in Mibibytes, and will therefore be aligned on a 512 byte boundary.  It also makes it easier to think in Mibibytes.  This is the most important step, so get this right!

4. Create LVM Setup and Filesystems- Now we need to recreate the setup we had before.  First, we create the physical volume, volume group, and logical volumes as they existed before:The ‘-l 100%FREE’ technique tells LVM to create the Logical Volume with whatever space remains in the volume group.  In other words ‘Use the rest’.
   
5. Copy Data Back - Now we copy all the data back, using the same method as above (rsync):

   rsync -avS  /backup/boot/* /tmp/boot/
   rsync -avS  /backup/root/* /tmp/root/

6. Fix the Bootloader- Now that we have all the data copied, you’d think we were done, but we have one more step.  Because we moved everything around, the bootloader, called grub, will be confused about where its boot files are (because it uses disk offsets).  We need to reinstall the boot loader.  Fortunately, this is a simple process.  We start the grub bootloader utility, then tell it where to consider its grub root (NOT the same as the root filesystem).  After that we have it setup grub on the drive (in this case hd2, because its the third drive in this system):
7. Disconnect – From here, just shutdown your helper VM, disconnect the VM’s HD from the helper, and boot your old VM.  It should boot up just fine!

Phew!  That was a lot of work, and now you know how to do it!   Just use UBERAlign instead, if you can!

VNX OE for Block / vCenter Ops Adapter Beta 0.2

Here’s what I’ve been able to add in the last few days:

1) I’m capturing all metrics that we can output, even non-numeric ones, just in case they are useful someday
2) I’m capturing metrics for the following types of resources: Regular Cache, Flash Cache, Individual Disks, RAID Groups, LUNs & Service Processors.  Its over 4800 metrics on my little VNX:

3) All metrics are now encapsulated properly under groups

4) I’m using a guaranteed method for uniquely naming a resource – using the array set VNX name and the Base UUID for its WWNN concatenated together.  The WWNN would be enough, but I prepended the name for easier interpretation by humans
5) Why did I bother with #4?  So that I could get multi-array support!  Yay!

6) And the last one – in my mind the coolest and was certainly the hardest to get working – Delta Calculations!  As you may know, vCops doesn’t do internal delta calculations (e.g., if I had X writes last time, and X this time, thats Z writes/second.  The array also generally doesn’t do this.   So, I developed a method for the script to keep state between runs (using a per-array temp file) that doesn’t rely on daemon being running, so that it can intelligently survive reboots and not have weird spikes if it goes 20 minutes without an update.  For each metric in the recorded metric (there are about 3300 metrics it collects), it determines if the metric is a numeric value, and, if it is, compares it with the last value and calculates a ‘perSecond’ version of that metric (averaged over the interval, of course – you aren’t going to see 5 second spikes like this).  Now, a per second counter like this doesn’t make a lot of sense for some metrics (like % Busy), but those can be ignored and only the relevant ones (like Writes_perSecond) used:

Neato!

Next up:  Internal device relationships and possibly external (to datastore) relationships

Also:  I looked at capturing storage pool statistics, but NaviCLI outputs almost no performance info about them (just config info, mostly), so I cant get that data yet.  Will bug the Navisphere developers about it

Questions, comments?