The 2012 Des Moines Farmers’ Market kicked off with a very large crowd this season, despite the humidity hanging in the air. As always, please feel free to skip to the Photo Set on Flickr, or the Slideshow, or click on any image to go directly to it on Flickr. Also keep in mind that these are all licensed under the Creative Commons Attribution, Noncommercial, Share Alike license. Enjoy!
A client called in with an interesting problem. They had recently performed a planned outage due to a power issue at their premises, but upon powering everything back up one of their hosts was unable to access the iSCSI SAN volumes. Fortunately, they were able to bring up all of the VMs on the remaining hosts, but capacity was down enough that they had to disable HA and DRS.
Upon interviewing the client, it became clear that they had already checked all the usual suspects. Nothing in the configuration prior to the power cycling had changed. Switch configurations, cables, ports, port groups, iSCSI settings, etc. were all exactly as before. In desperation, they had even wiped and re-installed the host using ESXi, carefully re-configuring everything to match the working hosts in the hopes that this would resolve the issue. It did not.
We know that VMware and many other experts recommend separating your vSphere network into separate Storage, Management, vMotion, and Production VM networks, typically using VLANs. This client, however, had opted to stay with a flat network configuration with all port groups and vmkernel interfaces configured on the same IP subnet. While this isn’t a Best Practice, there really isn’t anything wrong with doing things like this. As long as the vmknics can talk to the SAN, all should work fine, right?
At some point in the past, however, the decision was made to place an air gap between the storage interfaces and the rest of the network. All of the storage-related physical interfaces were connected to a switch which was not uplinked to the rest of the network in an effort to isolate that traffic so it wouldn’t overload the Production VM and Management traffic. Again, this isn’t a Best Practice, but it should work (and it did for quite some time) configured this way.
Where to start? I first plugged my laptop into the storage switch and attempted to ping the IP addresses assigned to the iSCSI vmkernel ports on the troubled host. No pings were returned, yet I was able to successfully ping all other storage interfaces present on the switch. Also, as expected, I was unable to ping all the interfaces connected to the Production/Management switches — a quick check to make sure there wasn’t an uplink between them. This pretty much established what we already knew, but more importantly, laid the ground work for my next test.
Next, I used PuTTY to ssh in to the troubled host and perform vmkpings. Here, I noticed a pattern that led me to my conclusion. I was not able to ping the iSCSI interfaces of the SAN, but when I tried to ping IPs that I knew were only on the Production/Management switches, the pings were returned. This made it clear that the Storage network traffic for the troubled host was exiting the host via the physical interfaces connected to the Management/Production switches and not via the interfaces on the Storage switch.
So what was happening? Upon booting up, that host was binding its iSCSI software initiator to the Management vmkernel port and not the vmkernel ports uplinked to the Storage switch. Since all vmkernel ports are automatically enabled for iSCSI traffic and there is no way to disable iSCSI traffic on a vmkernel port, there was no way to force the iSCSI software initiator to bind to a particular vmkernel port — except to do things right and set them up on separate IP subnets/VLANs.
So, of course, the real solution is to re-work the storage network so that it uses a different IP subnet than the production network. This, however, requires planned down time to re-IP all the storage interfaces on all hosts and the SAN. Until that can be planned, they were still down a host and running without HA/DRS. On a hunch, I came up with the following work-around, which got the host back up and running until such time as the reconfiguration could take place:
This worked because the only vmkernel interfaces available while the server was booting were the ones connected to active uplinks — the ones connected to the Storage switch. Once that binding took place, it would not change so it was then safe to plug the Management vmkernel uplinks back in. Obviously not an ideal situation, but it did get the host back in service until a outage window can be scheduled to properly configure the Storage network interfaces.
I just got through changing almost all of my 1200+ photos on Flickr from an all rights reserved license (the default) to the CC BY-NC-SA license. That is, Attribution (BY), Non-Commercial (NC), and Share Alike (SA). Breaking that down, this quite simply means:
Why on earth would I simply give my work away like this? My primary motivation is exposure. I feel that releasing my works like this will bring more attention to my work. Plain and simple. I’ve been inspired by artists like Nina Paley and musicians who share their music on sites like Jamendo — all under Creative Commons licensing.
I chose this particular license because I feel it is the most fair to all parties involved. For example, I do not collect model releases from any of my subjects. This would be quite difficult in situations such as the Des Moines Farmers’ Market or shooting on the streets of places like San Francisco. Therefore, the Non-Commercial clause is pretty much required.
Attribution to me is the very least anyone can do to repay me for my work, and it is all I ask from these photos I present on Flickr and this blog.
Share Alike, while a restriction on the freedom to use these works, is also a small price to pay for remixing my works and creating your own from them. Think of it as paying it forward. I gave freely of my work, so you must do the same.
Finally, my intention of seeking attention is without ambition. I’m simply sharing my work and hoping for a greater audience.
Agenda:
My Notes:
I recently purchased three albums from iTunes. After downloading them, syncing them to my iPod, and listening through them I was happy. While driving the next day, I thought I’d shuffle through the playlist I had created with the new tracks. I quickly discovered that all the tracks on one of the albums had a flaw that was only evident when shuffling them. Tracks 2 through 17 were missing the first 2 seconds, and tracks 1 through 16 had the first 2 seconds of the following track tacked on to the end. This is easy enough to fix in Audacity, but I felt it important to report to iTunes, if only to call their attention to the issue so they could fix it before too many others reported it.
After surfing through the apple.com site for a bit, I finally landed on their Express Lane Support page. I had to dig around a bit more before I found “Quality of purchased content” under “Purchases, Billing & Redemption.” I filled in the information required and opened a case describing the issue. An automated message stating my support request would be responded to within 24 hours came back very quickly.
The “real live person” response came back within 4 hours of posting the complaint. Impressive, since I posted my complaint at about 12:20 AM. The responding service representative apologized and gave detailed instructions on how to delete and re-download the content. I as pretty sure this would not resolve the issue (it wasn’t a corrupt download, as they seemed to think), but I went through the motions anyway. Upon listening to the re-downloaded content, I confirmed that the issue was not resolved and replied to the service representative, advising them that I believed the source files on their servers were not correct.
Hello Kenneth,
X here again from the iTunes Store Support. I am very sorry about my delay in responding to you. I have been away from the office for the last 2 days. I understand the album is still incorrect. When it comes to your money, I can certainly appreciate how important it is to feel that you are treated fairly, and I would be more than happy to help you out with this today.
I’m sorry to learn that this item did not meet the standard of quality you have come to expect from the iTunes Store. I have submitted this item for investigation. Apple takes the quality of the items offered on the iTunes Store seriously and will investigate the issue with this item, but I can’t say when or if the issue will be resolved.
In five to seven business days, a credit of $9.99 should be posted to your card that appears on the receipt for that purchase.
Kenneth, I want to thank you for choosing the iTunes Store and for being such a big part of the iTunes family.
Thank you for contacting iTunes Store Customer Support. Have a great day.
Given that response, I had no choice but to spend some time in Audacity repairing the tracks. The general procedure was as follows:
After all the tracks were repaired (I worked on a copy from out of iTunes), I deleted the originals from iTunes and re-imported the repaired versions. I then had to go through and repair the tagging, as it was a bit messed up. For some reason, the tagging didn’t import consistently from the Apple versions (either that or it was inconsistent to begin with).
I wonder how many free copies of this album they’ll give out before they correct the files on their server? I wonder how many people will complain and get a refund vs. the number who will just put up with the issue? I wonder how many other albums are messed up on this way?
Took a couple of hours to walk around downtown Des Moines, IA this weekend along with some other fellow photographers. Unfortunately, I showed up late and had to leave a little early so I did not get to socialize very much. Also, I must have taken off in a different directly than the bulk of the group, because I didn’t end up running in to too many other photographers. From Java Joe’s, I headed West towards West End Salvage. I went to the top of a few parking garages to try to get a different perspective of the Des Moines sky line.
Here are my photos from that session. As always, please click on a photo to show it on Flickr, visit the whole set on Flickr, or just fire up the slideshow.
Iometer started life as a utility built by Intel to generate and measure i/o loads. It was released by them under the Intel Open Source License. The date this happened isn’t clear from their Web site, but the project was first registered on SourceForge in November 2001.
You can grab the latest stable release from the downloads page. Although the latest stable build is from 2006, I recommend using it rather than the newer, unstable versions available from the SourceForge project page (unless you like crashing your VMs, that is).
There are downloads for Linux, Netware, and Windows. All are 32-bit (i386) builds, but the source code is available.
I’ve not used the Linux version yet, so here is a walk-through of the installation (pretty much next, next, finish) on Windows 7:
The test file (in our example 1 GB in size) is created either under the root of the drive selected, or under the user’s folder: C:Users%username%AppDataLocalVirtualStore. The name of the file is iobw.tst.
This file is only generated the first time you launch Iometer and is not generated again — even if you close, re-launch Iometer, and select a different Maximum Disk Size. Therefore, if you need to use a different size, you must do the following:
I couldn’t find a way to reset the size of this file or remove it from within the Iometer GUI.
If you want to throw more IOPs at your storage, you can add multiple worker processes under the main manager process. These workers can be clones of the first one you set up, or they can be new ones set up with different Access Specifications. All of them will run at the same time when you start the test.
A good write-up about Iometer and simulating various server workloads is available on the VMware Communities Forum. That post gives some example settings for simulating Exchange and SQL Server workloads with Iometer.
Iometer is a great utility to use in your Test/Dev environment to simulate workloads. You could also use it to stress test a pre-production environment to make sure you haven’t mis-configured anything, or accidentally created any bottlenecks in your design.
Got up early this morning and took some photos during “golden hour” light at Des Moines’ Western Gateway Park. As you can tell, Nomade is my favorite subject.
As always, please click the photos or visit the photo set on Flickr to see more from this set.
A selection of my photos from both Des Moines sessions of Scott Kelby’s Worldwide Photowalk 2011. As always, please visit either the First Set or the Second Set for more!
I’ll lead off with my two submissions. Neither of them won, but that’s not surprising because all of the submissions were excellent!
Now for a selection of other photos from the walks:
