A Place of Safety
I expect simple behaviours here. Friendship, and love.
Any advice should be from the perspective of the person asking, not the person giving!
We have had to make new membership moderated to combat the huge number of spammers who register
















You are here: Home > Forum > A Place of Safety > General Talk > Server migration - Outages
Server migration - Outages  [message #71748] Thu, 07 July 2016 07:08 Go to next message
timmy

Has no life at all
Location: UK, in Devon
Registered: February 2003
Messages: 13739



Over the next couple of weeks there is very likely to be total disruption to our server. Our service provider is having to migrate all hosted servers to another data centre. The current data centre is ceasing trading.

Experience of moves like this says it will not go smoothly, however well planned and executed. This is a service interruption we could all do without, but can't be helped

What you can expect:

* Random period of outage, could be up to several days
* Withdrawal symptoms from reading our wonderful authors and stories
* Emails to us are likely to bounce. Email systems are designed to be robust and retry behind the scenes, but the outage may exceed the retry period.
* We wil have a new IP address for our email server. We will thus inherit that IP address's blocklist status. It may be clean, or we may need to fight to clean it. Outbound emails are likely to be disrupted

We will use Twitter to keep you posted if we have the problems we expect. See https://twitter.com/iomfats  You have no need to join twitter nor to follow us on twitter to see the tweets.



Author of Queer Me! Halfway Between Flying and Crying - the true story of life for a gay boy in the Swinging Sixties in a British all male Public School
Re: Server migration - Outages  [message #71749 is a reply to message #71748] Thu, 07 July 2016 22:27 Go to previous messageGo to next message
timmy

Has no life at all
Location: UK, in Devon
Registered: February 2003
Messages: 13739



We now have the new server in preparation. It has a new IP address as does our email server. The new IP address is spam blacklisted, something we are working with our host to address. This is often the case when moving to new IP space.

When we cut over the forum will go offline. This will seek to ensure that no posts are lost. Running a dynamic site is hard when server migration takes place.

Since we now have more control than we expected we are able to control the cutover, at least to an extent. This weekend may be too early. We have testing to perform. Then we have to synchronise the old and new servers one final time. After that we move across. We then freeze the site for story updates.

At that point we are governed by global DNS servers not under our control. At present they point to our current server. It is not as simple as pointing them to our new server, because this has to propagate around the entire global set of DNS servers. This can take some time, often 48 hours.

Once we can see the new server via our local DNS servers we can return to updates of stories. The global iomfats readers will ONLY see the updates when the DNS system synchronises in their "internet area" even though they will receive the emails (If we have solved the email blocklists) and see the tweets.



Author of Queer Me! Halfway Between Flying and Crying - the true story of life for a gay boy in the Swinging Sixties in a British all male Public School
Re: Server migration - Outages  [message #71755 is a reply to message #71749] Mon, 11 July 2016 14:11 Go to previous messageGo to next message
timmy

Has no life at all
Location: UK, in Devon
Registered: February 2003
Messages: 13739



We had what I hope to be the only major outage just now as we switched DNS to the new server's IP address. We are running the final checks now



Author of Queer Me! Halfway Between Flying and Crying - the true story of life for a gay boy in the Swinging Sixties in a British all male Public School
Bugger  [message #71759 is a reply to message #71748] Wed, 13 July 2016 10:10 Go to previous messageGo to next message
timmy

Has no life at all
Location: UK, in Devon
Registered: February 2003
Messages: 13739



The migration itself has gone well. What has not gone well is a technical email issue, rDNS, an arcane but essential topic for which Google is your friend, and something our service provider has not yet handled. We "could" handle it ourselves, but:
http://forum.iomfats.org/?t=getfile&id=2861&private=0
Support says that this aspect is currently faulty. Well, duhhhhh!

What this means is that we are suspending story updates until this is sorted out. We do not dare send out announcement emails because we expect to be auto-blocked by the anti-spam community after our third set of bounces. The email announcement scheme is an integral part of our story announcement software

There is only a certain amount of pressure we can put on our host, despite our knowing that they were actually unready for their customers to make this migration



Author of Queer Me! Halfway Between Flying and Crying - the true story of life for a gay boy in the Swinging Sixties in a British all male Public School
Still stuck  [message #71767 is a reply to message #71759] Tue, 19 July 2016 08:44 Go to previous messageGo to next message
timmy

Has no life at all
Location: UK, in Devon
Registered: February 2003
Messages: 13739



We are in a cleft stick. Technically it is easily possible to update the stories, but that has no point without being able to announce them.

Our announcement mechanism is a small following in Twitter (broadly irrelevant) and a 1,000 email database. Sending to that email database has generated over 20% of bounces that we know about, plus silent bounces that we do not.

Every bounce generates a risk of our email server being reported to a spam block database. That prevents our emails from being received anyway. Multiple bounces form the same commercial server increase that risk substantially.

So we need to be patient and wait for our host to finish its job. We are with them because they are price competitive, and their terms of service allow adult material. Since this is a "set once and forget" thing we believe it to be worth waiting. Finding another host is not a route we are going down right now. This incident apart they have proved reliable and responsive for several years.



Author of Queer Me! Halfway Between Flying and Crying - the true story of life for a gay boy in the Swinging Sixties in a British all male Public School
Militant patience  [message #71768 is a reply to message #71767] Wed, 20 July 2016 08:58 Go to previous messageGo to next message
timmy

Has no life at all
Location: UK, in Devon
Registered: February 2003
Messages: 13739



We are adopting the plan of militant patience. This is on the basis that other customers will be pestering the bejasus out of the hosting company. At the end we will be seen as the good guys because we identified the problem and then were not ridiculously pushy.

It's not great that they didn't think this through, but telling them so just rubs salt into the wound. Everything else about our host has been timely and wonderful. You will not remember a single serious outage before this, and, to be fair, the site is up and healthy. It is just the thing we are known for, the publishing of stories, that is delayed.

What I want to do is to tell them with clinical precision where to go. That would be cutting of my nose to spite my face.



Author of Queer Me! Halfway Between Flying and Crying - the true story of life for a gay boy in the Swinging Sixties in a British all male Public School
Re: Militant patience  [message #71769 is a reply to message #71768] Fri, 22 July 2016 17:40 Go to previous messageGo to next message
timmy

Has no life at all
Location: UK, in Devon
Registered: February 2003
Messages: 13739



We are not sitting idly by. Militant patience has given way to businesslike complaint. The problem is that we cannot actually do the task ourselves. If we could, then we would.



Author of Queer Me! Halfway Between Flying and Crying - the true story of life for a gay boy in the Swinging Sixties in a British all male Public School
Progress, of a sort  [message #71770 is a reply to message #71769] Sat, 23 July 2016 21:35 Go to previous messageGo to next message
timmy

Has no life at all
Location: UK, in Devon
Registered: February 2003
Messages: 13739



We have now been able to set rDNS, at least locally. Since rDNS does not propagate, if this has gone right, we simply have to await the nameserver cache timeout before our reverse lookup is queryable by external mail servers.

So, one step forward. Not sure when the final step is made. Seems you can cross a chasm in more than one step! 



Author of Queer Me! Halfway Between Flying and Crying - the true story of life for a gay boy in the Swinging Sixties in a British all male Public School
Cautiously optimistic  [message #71772 is a reply to message #71748] Mon, 25 July 2016 17:01 Go to previous messageGo to next message
timmy

Has no life at all
Location: UK, in Devon
Registered: February 2003
Messages: 13739



Today we had rDNS set by our host. So far so good.

New stories were published. Also good

39 emails bounced, but not for rDNS reasons. [rDNS is a validation that the email server is associated with the sending domain and helps prevent spam]. I am now addressing those 39 bounces. They are with at least two providers. So, at present, any AT&T email including all their companies, and any iCloud or mac.com emails are blocked. This ought to be a matter of process and procedure to change.

[Updated on: Mon, 25 July 2016 22:22]




Author of Queer Me! Halfway Between Flying and Crying - the true story of life for a gay boy in the Swinging Sixties in a British all male Public School
Re: Cautiously optimistic  [message #71774 is a reply to message #71772] Tue, 26 July 2016 07:59 Go to previous messageGo to next message
timmy

Has no life at all
Location: UK, in Devon
Registered: February 2003
Messages: 13739



We are now down to 29 bounces. We await AT&T's pleasure for 28 of them. The other we await someone else's pleasure for.

It appears that the migration is now complete.

Now I can start to play catch-up



Author of Queer Me! Halfway Between Flying and Crying - the true story of life for a gay boy in the Swinging Sixties in a British all male Public School
Re: Server migration - Outages  [message #71775 is a reply to message #71748] Thu, 28 July 2016 00:53 Go to previous message
Geron Kees is currently offline  Geron Kees

Likes it here
Location: USA
Registered: February 2016
Messages: 147



Now that you have the rDNS done, you need to do something about the furious echo in here.

What...is everybody on holiday?
Previous Topic: A Little Visual Art :)
Next Topic: UNK Author for "My Beach Mate"
Goto Forum: