Coming changes on filer (RAID)

June 23rd, 2008

As you may have noticed on Friday, everything was down from 7am to 11am (CEST, GMT+2), our main NAS crashed again which required a hard reboot by our kind hoster: we think the motherboard and the RAID card are a bit incompatible and we have planned a replacement (of the RAID card) ASAP.

We’ll keep you informed of coming changes.

BTW, there has been one hour outage on the Completel network on Saturday from 7am to 8am without any known reason, this night there has been a four hours outage from 1am to 5am to work on the fiber which were broken earlier, based on our record the quick repair were not very clean and it needed some further work on it ;)

Network outage

June 11th, 2008

Hi,

Well, quite a bad news, the completel’s network around Marseille, France is completely fscked up… the time to repair window is very large… but the weather is fine here, so this is a good time to do outdoor activities !

Secondary services like mail and DNS are up, no mails are going to be lost ;)

Please bear with us.

Edit: We are back on the scene !

MySQL InnoDB crash

May 28th, 2008

Well, we managed to corrupt the InnoDB MySQL storage engine files while cleaning user databases.
We set up in production the backup of the last night.

By the way, this is normally not our job to clean the spams in your databases, but we have to do that because we cannot afford the load of billions of spams, so if you don’t want that we need to handle the files used by MySQL or PostgreSQL, clean them. In our opinion, this is the time to think to switch to something that don’t break, switch to PostgreSQL.

Edit: Well, we had to restore from databases dumps this night in order to fix the corrupted InnoDB database. The records of yesterday are of course lost, but the coherency is assured, which is the most important thing. Everything is good for now.

Upgrade of ssh and ssl keys

May 15th, 2008

Hi,

Due to a security issue with openssl, we have changed all SSL keys.

If you are using a ssh key to log in to our servers, please renew your key and destroy the old key ASAP.

Here is a summary of the new keys fingerprints’:

Type of service Concerned hosts New fingerprint
SSH / SFTP ssh.tuxfamily.org
sftp.tuxfamily.org
8d:b8:fe:cf:3a:2a:58:4b:fb:ad:0a:40:e6:5c:40:76
HTTPS tuxfamily.org
tuxfamily.net
tuxfamily.info
SHA1: FF:53:58:26:12:2B:F8:2F:80:F1:26:67:08:5B:30:00:30:25:C1:2F
MD5: C3:F1:C2:25:9E:0B:9D:B5:22:04:83:AE:24:C0:9F:E9
HTTPS All hosted websites SHA1: 9F:A8:3E:2F:9A:4E:B0:74:8D:A0:C4:16:22:CB:9D:10:40:A0:B2:E8
MD5: BE:38:75:48:65:63:B2:DF:EC:8A:4C:2F:2B:AD:DE:C3
HTTPS panel.tuxfamily.org
webmail.tuxfamily.org
SHA1:C2:20:FB:31:94:EA:62:4F:56:51:AA:CA:AB:FB:CF:7E:CA:BC:B6:A4
MD5: 42:E3:8E:EA:33:E4:F6:3B:C4:96:24:F1:B5:63:8B:DB
POPS pops.tuxfamily.org SHA1: 5A:21:0D:A5:7F:EA:BE:E4:8C:C9:7D:EB:BA:AD:64:E1:AE:CA:47:82
MD5: 87:18:6B:6F:40:0E:38:A2:9C:E7:D0:6A:37:D6:2E:48
IMAPS imaps.tuxfamily.org SHA1: 7E:3E:9A:98:E7:17:36:2C:50:E7:74:79:A3:01:E5:30:D4:93:6C:DD
MD5: B1:DA:79:EE:55:14:39:8F:DC:C7:67:C4:54:65:CC:4B
FTPS ftps.tuxfamily.org SHA1: 66:A1:74:8A:02:E0:F9:01:6A:EE:8E:87:AF:91:F0:84:55:02:0B:29
MD5: 6B:5D:E9:AF:B1:98:C3:29:F7:6E:79:AF:BF:62:0A:02
CVS / SVN / GIT cvs.tuxfamily.org
svn.tuxfamily.org
git.tuxfamily.org
d3:f9:06:91:36:83:42:c5:85:d3:e2:55:b9:e3:88:c0
JABBER jabber.tuxfamily.org SHA1: 91:E1:8B:95:C3:30:41:DD:0D:07:60:D3:F9:BC:64:6A:C2:1D:FF:D2
MD5: D1:2B:FD:BA:69:69:4C:D3:87:A0:70:37:99:CD:F3:43

Optical fibers cut

May 14th, 2008

Well, the optical fibers connecting the datacenter where we are hosted were cut by some work on the public road…

A cut of an optical fiber shouldn’t be that bad, but it was on a short length where all fibers are, so all the fibers were cut in once.

The fibers are currently being soldered, but this will probably take several hours.

But, the secondary services (ns2 and mx2) are up and running ;)

Please bear with us.

Edit: Soldering done, everything is back online.

Tux is moving back to Iceland \o/

April 1st, 2008

We are pleased to inform you that TuxFamily’s servers are going to move to Iceland on Tuesday, June 19th

Our main hoster have just finished his new datacenter and asked us if we wanted to put our servers there. Of course, we are delighted to be the first organization to use the new datacenter : The naturally cold weather and the environment friendly electricity production of Iceland allow to reduce a datacenter’s operating costs from 10 to 20%.

The only difficulty is that we have to move the servers by ourselves because our hoster don’t want to handle the costs of carrying our stuff to Iceland. We noticed that we can save on shipping costs by sending the servers by sea through New York in order to pay in US Dollars. As the Euro is strong, we can save 10 % which is quite nice regarding to our banking account situation.

All the services will be down during the trip that should last between 27 and 34 days depending on the weather.

Secondary servers are not going anywhere so your mails won’t be lost during this period ;)

Your admins

Edit: Of course it was our annual April fool ;-)

Mailing-lists, forwards, and the great mail service of free.fr

March 24th, 2008

Hi folks,

Instead of finding a good way to handle the load of their millions users due to their attractive low-cost service, free.fr decided to blacklist almost everyone who is sending spams based on the IP source of the MTA which connect to their servers. So all of us who are providing forwards or mailing-lists services are blacklisted because we are forwarding spams through those ways (what a pity !).

We used to be whitelisted by free.fr when they were not that lobotomized, it seems that the whitelist have been wiped out… The “solution” as said by free.fr is “to set up a spam filter which should stop spam that you send, relay or forward“, in other words “please do on your side what we don’t want to do on our side anymore” and they call that “solutions to provide to your email solution in order to fight spam with us“. Dear free.fr staff : If you think this is good for your users then, considering the amount of complains we are receiving about that… well, think again.

Of course, we are almost permanently blacklisted, so we recommend you to stop using @free.fr addresses.

Anyway, the free patronage is hosting one of our servers and we are really grateful about that, but this does not mean that we can’t disagree with their spam policy ;-)

Let’s hope they will change their mind ;)

Corrective maintenance planned

March 15th, 2008

Hi,

To fix one of our file servers related issue (here, here and the emergency fix here), we planned a corrective maintenance, it will be this week-end.

Be prepared for a long service outage of all the services (except secondary services) at anytime between Saturday and Sunday.

SSL certificate was renewed on web service

February 25th, 2008

The SSL certificate for the web service had expired, so we renewed it.

The new fingerprints are:
MD5: 34:EF:D1:EA:A6:9A:BA:84:B6:79:1A:A3:9E:6E:3D:80
SHA1: ED:DD:BD:69:91:A1:2D:F6:BF:AF:1A:11:20:FE:C9:F8:E3:C6:5E:1D

It is still self signed, we will see soon to use a CAcert certificate instead.

Beginning of the survival period

February 22nd, 2008

Well, our main file server is going to die pretty soon, luckily it is still working enough to keep the service up most of the time. We have good reasons to suspect the hard RAID controller to hang up the server in an interrupt request which never ends up.

We noticed that it always crashed during very heavy I/O operations like stats generation, database dumps, mailing lists archives updates or backups. So to avoid further crash of this file server we have temporarily disabled all of those features.

Unfortunately we are in a holiday period here in France and the server is about 1000 kilometers away from those who are ready and able to do the hardware maintenance. As we cannot afford a very expensive train ticket during holidays, we had to buy those tickets for the middle of March and we hope that the server will not definitively die before.

Stay tuned…

Your admins