Select from database ID where name = RICHARD;

Returns and ID of 55 for example.

Then go and do a query like this:

Select * from some_other_table where ID = 55;

Returns, “Sorry does not exist, time to die…..”

Well apparently indexes when corrupt – which is NOT SUPPOSED TO HAPPEN – can cause PostgreSQL to go all stupid and not do a table lookup for real. This happened to me. So I found this:

PlanetPostgresql

Turns out that a reindex and a full vacuum can do wonders – even though a full vacuum is not needed with autovacuum and indexes can’t get corrupted…..or so they say.

I have now added a system wide reindex maintenance plan for PostgreSQL every night. I know that MS-SQL server has an option for this with their maintenance jobs inside enterprise manager. Maybe someone should make an enterprise manager for PostgreSQL too?

In any case, the short of it is, that we were juggling PERC4 cards around servers (PCI-X here, PCIe there..) and also complete raid 1 and raid 10 arrays too. The cards are supposed to “detect” the correct array type from the drives if the firmware was missing. Anyway, through a comedy of errors, it worked exactly 1/3 times. The other times we had to remember the exact settings of our arrays (stripe, etc) and how it was structured. So we could clear PERC cards and then recreate the arrays – taking special care to not initalise the new arrays.

So in the end, you can move arrays and channels about. And with LVM, even designations like /sda /sdb reording is also not an issue. However you should rely on good old fashioned hand held way of doing things. Before you start write down all the salient details of your arrays first.

CPU

System load was DOWN. This is due to much better disk IO (reduced wait times) and as such, user processes get to breath and send stuff back to clients in a speedy fashion.

LOAD

Load being a matrix balanced scorecard measurement over various metrics is also down considerably. The trend to take note of here is the 15min average more than the 1 min average. While spikes and load increases over short periods may upset some users experience. If the delays compound and extend over a longer time frame, you will be tying up more people in the slowdown. Again, this is leading a bit to the parallel vs serial arguments. And in the case of a database, sometimes fewer ‘in and out’ transactions give a better user experience over time because more gets done in a given time period, relative to many threads fighting at once. Think orderly lines at the bank in a developed country compared to the Beijing Subway at Jianguomen or Fuxingmen!

TEMPERATURE

A good 5 degrees lower in temp too. That is not insignificant. It all adds up to longer system life, better availability and lower costs.

POWER

Well out APC metered units aren’t that precise! While you can’t see any change from last week to this one – probably due to the load balancing effect of our redundant power supplies and redundant PDU’s AND UPS’ in between PDU’s and Servers (in addition to the IDC’s UPS’s and 4,000 Car Batts). You CAN see the increase in Saturday and Sunday while the massive disk arrays churn flat chat to reinitialise….that will need more power and also make more heat, needing more fan speed.

I used this excellent resource before.

Even the very rudimentary tweaks I did, and with a RAID array that is in a 90% rebuild rate, background initialisation, this 8.1 version is FAST! I have no idea why people use MySQL, it really is such a piece of crud.

Let me expand on this;

This type of backup file is advertised as “more convenient” and offers more options for restore time selective data restores, data re-ording, index tricks and the like. However no matter WHAT I did, it reported and sent a copy of the current pg_restore process and all the data being restored to standard output too!! This means that basically, I was going to have the same full text of 20GB worth of database data shoved down my SSH session!

Yes – this makes the whole affair much slower!

Luckily, being the lateral thinking kind of dude that I am, I never put all my eggs in one basket. That is just a recipe for data omelette.

I also had some plain text file dumps made from pg_dump. So I went into the the PostgreSQL template1 sessions and then used the “\i” command to import my big .SQL file. Perfect! Only important updates sent to std output and not the whole damn enchilada.

Disk restore is still going on. At a max through put of 20MByte a second with a 1000BaseT network, that is at best 1.2G a minute, 72GB an hour, so for 250GB approximately 3.5 hours. However I was still doing a raid array background initialisation, so even after setting the rebuild rate to 10%, the system still needs a lot of time to move the files back – I didn’t bother to do the maths, because 3.5 hours or 10 hours it would all breach my maintenance window.

Because my advertised downtime window to clients was rapidly approaching, I had no choice but to continue to allow the copy to proceed, but redirect the main cluster to access the data store via NFS over the network and bring services back online. I will then be able to do a RSYNC later in less than 1 hour to bring the haphazardly copied set on the main cluster in line with the now used and modified data store on my hot standby server.