Auteur DenBolle | Datum: 23-1-2011 |
Ondanks de voorbereidingen liep het allemaal net wat anders als gepland. Zo waren er eigenlijk maar drie problemen die impact hadden.
We startten wat later dan gepland door een uitgelopen backup. Beter safe then sorry. Tijdens het omprikken van de kabels bleek dat de kast waarin de apparatuur stond precies met een poot op de vloertegel stond die ik er uit moest werken. Om dat goed voor elkaar te krijgen moest er om de kast heen extra tegels worden verwijderd zodat de eigenlijke tegel die ik er uit moest halen vrij kwam te liggen. Op zich niet erg maar ..
Door dat gebouw en geheister zat een kabel me hinderlijk in de weg. Dat ding hing niet weggewerkt vervelend te wezen. Omdat nu toch alles plat lag mooi een moment om de kabel door een ander te vervangen en weg te werken. Kabel opgesnort, ingeprikt, mooi weggewerkt en verder gaan.
En toen. tja.. toen bleek ineens dat dat rare roze kleurtje van die kabel niet voor niets was. Het duurde even voor ik me realiseerde dat die kabel een crosslink was. De oude kabel weer op de plek en alles werkte weer. (va. 11:51 uur)
Althans.. Op de een of ander manier ging het kopiëren van de data niet zo snel als we dachten. Waar een beetje snelheid nodig was bleek de copy slag traag te zijn, en waar we op een paar uur data pompen hadden gerekend werd het al snel een dikke 40 uur (is dus ergens maandag ochtend). Daarom op een noodplan overgestapt, en maken we de laatste stappen volgende week af.
En op het laatst (ik ben bijna aan bewaker 3 toe, ze werden telkens afgewisseld) bleek dat het hele printer systeem plat lag. De printerserver had een schop gekregen en aangezien onze printers hun configuraties uit de printerserver trekken ging dat fout. Na een kort telefoon gesprek met een van de locale jongens die dit soort dingen vaker bij de veter had gehad besluiten we alsnog om de printeromgeving een keer plat te gooien en dan op te starten. Langzaam aan (zeg een 5-10 min/ printer, we hebben er bijna 20 staan) kregen we ze stuk voor stuk weer aan de praat en aan het printen.
Dus de ingeschatte eindtijd van 1800 hrs. werd wat overtrokken…
Voor de geïnteresseerde, hieronder staat de hele listing (geanonimiseerd en ontdaan van specifieke tech info) van een hele dag waanzinnig lekker samenwerken op een site, met 4 landen (Nederland, Polen, België, Italië), serieus en minder serieus gegein.
een lange dag, maar met plezier gedaan. En… als het goed is merken de gebruikers er niets van.
De spelers hieronder: Italian (in de gesprekken Ig-1) : een van de project managers werkend vanuit Itailië, Polish-1 , Polish-2 en Polish-3 (respectievelijk Pg-1, Pg-2 en Pg-3 in de gesprekken): de poolse jongens in het poolse datacenter, werken met de VM ware machines, unix storage en Backup omgeving. C, K, A zijn testers/adviseurs die op afroep voor ons klaar stonden.
8:26 [ Italian ] Morning Den
8:26 [ Polish-1 ] Hi Den!
8:26 [ Polish-2 ] Hi Den
8:27 [ DenBolle ] Goodmorning everyone!
[Ig-1] is generous : he treats us all in
coffee!
8:27 [ Italian ] yes this for [Pg-2]
for [Pg-1]
8:28 [ Polish-1 ] thx
8:28 [ Italian ] for Den that has to wake up
8:28 [ DenBolle ] Gracie!
8:28 [ Italian ] yw
8:29 [ Polish-2 ] Thank you!
9:00 [ Italian ] Hi [Pg-3]
9:00 [ Polish-3 ] Hello
9:00 [ Italian ] please advise us when done
9:00 [ Polish-3 ] around 10 minutes left I think
9:01 [ DenBolle ] ah ok
9:01 [ Polish-3 ] blowing from home will not speed it up,
sorry
9:02 [ Polish-2 ] ;-)
9:02 [ Italian ] 9:02
9:02 [ DenBolle ] I'm in the office, I could spin up the
tape?
9:02 [ Italian ] good idea
9:02 [ DenBolle ] use a microwave to compress the data on
the tape?
9:03 [ Polish-3 ] Den, you are the biggest of us /except
[Pg-1]/ could you tell few words to the
Adic Lib to write a little bit faster ?
9:04 [ DenBolle ] sure..
[walks into server room with a stick and
blowtorch]
9:04 [ Polish-3 ] ;-)
9:04 [ Italian ] roftl
better ..... grin grin
9:06 [ DenBolle ] ok, I've discussed it with the lib, he's
appologizing and will work harder.
btw I never realized that they actually
can scream if you put them on fire.
9:06 [ Polish-3 ] buahahahaha
9:08 [ Polish-3 ] I though Adic is a woman..strange world..
9:10 [ DenBolle ] It screamed like a woman though..
9:14 [ Polish-3 ] btw started to write system state...
she is coming or he...
9:15 [ DenBolle ] she is coming.. this story get's a nice twist.
Go on [Pg-3]
9:16 [ Polish-3 ] heheh who will make some pictures - ->Den?
It is done gents!
9:16 [ DenBolle ] and put them on a site " www.hornytapelibs.com" ?
ok, let go on then
9:16 [ Polish-3 ] yep
9:17 [ Polish-2 ] it looks like I've missed some nice stories here
ok, I'm proceeding with stopping the BU1SMLXV0001
9:18 [ Polish-3 ] well..it will not be published on our webside
9:18 [ Polish-1 ] Can I start stopping VMs now?
9:18 [ Polish-3 ] logged off
9:19 [ DenBolle ] ok, I'm over to wireless to be safe.. just a se
9:19 [ DenBolle ] and back
9:20 [ Polish-3 ] [Pg-1] could you call me when the servers will be
back - I will start backups - I mean afternoon
9:20 [ Polish-1 ] Sure.
9:20 [ Polish-3 ] 504 110 321
cool,
9:31 [ DenBolle ] is this working
9:31 [ Polish-2 ] yep, now it's working
9:31 [ DenBolle ] mhm. glitch..
what did I miss?
9:31 [ Italian ] ;-)
9:31 [ Polish-1 ] I know. Some functional accounts logged
on the console. I need to power them down
manually
9:32 [ DenBolle ] last remark i've seen is froim [Pg-3] :
9:32 [ Polish-3 ] 504 110 321
cool,
9:35 [ Polish-2 ] I can't paste the OCS chat, because it
says that it is too large..
So to summarize it quickly: [Pg-3] logged off,
because his wife was chasing him. I've stopped
the BU1SMLXV0001 and [Pg-1] is stopping now
other VMs and ESX hosts.
9:36 [ Italian ] All Vms stopped ...great job [Pg-1]
9:36 [ DenBolle ] no problem.. So no strange things yet and
all goes as planned. Besides the huntdown of
[Pg-3]'s wife.
9:36 [ Italian ] ;-)
9:36 [ Polish-2 ] Maybe she heard the library's screams...
9:37 [ DenBolle ] tsk.. women united... If you hurt one ...
9:37 [ Polish-2 ] ;-))
9:38 [ Italian ] when finished I'll copy this chat and send
to Manager1, Manager2 and Manager3
with title "drunk men at work"
9:38 [ DenBolle ] grin grin.. and we never are allowed to
work on saterday..
9:44 [ Polish-1 ] @all: All VMs are powered down now. All
ESX hosts are shutted down and powered off now.
9:44 [ DenBolle ] ok, thx
9:44 [ Italian ] Den ..... go there and do your best
9:45 [ Polish-2 ] no no
now it's my part
please hold on
9:45 [ DenBolle ] [stopping]
9:45 [ Italian ] sorrrrrrryyyyyy
9:45 [ Polish-2 ] Den, I'll let you know
It will take approximately 15 mins
9:45 [ DenBolle ] I'll be waiting for you
[not to self : this chat's getting stranger
by the sentence]
10:01 [ DenBolle ] I've got screaming equipment
10:01 [ Polish-2 ] OK, all FC switches and FAS270`s are going down
or are already down.
This is the reason of those screams.
10:02 [ DenBolle ] ok, let me check in the room when they actually
die..
10:02 [ Polish-2 ] Let's wait now 2-3 min, just to be sure that all
went down.
10:03 [ DenBolle ] sure
10:03 [ Polish-2 ] After this 2-3 mins, I'd like to ask you, Den,
to power off the following devices:
BU2DSBS001
BU2DSBS002
BU1DSBS001
BU1DSBS002
BU2DSNTP001
BU2DSNTP002
BU1DSNTP001
BU1DSNTP002
so - all 4 FAS270's and all 4 FC switches.
10:04 [ DenBolle ] I'll do. I'll start at 10.05
10:05 [ Polish-2 ] ok
10:08 [ DenBolle ] ok, all shelfs are down (no more hd activity) :
I'll go shutdown the devices.
Do you want me to reconnect the fibers straight
away?
10:09 [ Polish-2 ] Did you power off the additional shelves as well?
10:09 [ DenBolle ] no, thats going to mbe done now
10:10 [ Polish-2 ] To be honest - it's probably not needed.
10:10 [ DenBolle ] if you feel safer when I do: no prblem.
10:10 [ Polish-2 ] OK - If you want to do so, we can do it. Just to
remember: later, when we will be powering it on,
we need to start them in reverse order.
So first of all: the additional disk shelves,
after that 2-3 min break to let the disks spin up
and afterwards the FAS270 controllerrs
10:12 [ DenBolle ] ok, I'm in, back in 10 -15 mins
10:12 [ Polish-2 ] one additional thing
10:12 [ DenBolle ] yes
10:12 [ Polish-2 ] I've set new IPs for BU2DSNTP001 & BU2DSNTP002.
So please reconnect also their ethernet cable to
the new subnet.
..and you have to reconnect the FC cable as well
as we discussed on the diagram.
10:13 [ DenBolle ] I'll do, stay tuned
10:13 [ Polish-2 ] th
thx
10:40 [ DenBolle ] ok, almost done, just one issue.
the 4 utp cables for the 270's : the core switch
is full, give me 20 mins to isolate cables we can
disconnect
10:41 [ Polish-2 ] 4 utp cables?
10:42 [ DenBolle ] yes the ethernet cables
10:42 [ Polish-2 ] to the BU2DSNTP001 & BU2DSNTP002 4 cables are
connected ?
10:42 [ DenBolle ] yes
10:42 [ Polish-2 ] they have dual ethernet ports, however only one port
(e0a) is used
10:43 [ DenBolle ] ok, in that case I'm still one port short : got one
emtpy one..
10:43 [ Polish-2 ] ok, I see
ok, we will wait
when you finish
you can power on following devices:
BU1DSBS001
BU1DSBS002
BU2DSNTP001
BU2DSNTP002
BU1DSNTP001
BU1DSNTP002
and please do not power on ESX servers
10:44 [ DenBolle ] No, I;ll do. ok, [Ig-1], in that case could you
serve all once more a cup of coffe in the mean
time?
10:46 [ Italian ] here they are ......
I'm boiling water
.........
and ..... that's it
11:03 [ DenBolle ] ok, done with the cables, powered up shelfs
(not FAS270's), and BU1DSB001 and BU1DSB002 are
starting. When these are finshed I'll boot
BU2NSNTP001/2 and BU1DSNTP001/2
11:03 [ Polish-2 ] great news
11:03 [ Italian ] [happy]
11:03 [ DenBolle ] at .05 I'll start
BU2NSNTP001/2 and BU1DSNTP001/2
11:04 [ Polish-2 ] FC switches haven't started yet.
11:04 [ DenBolle ] ok, when they have, ping me, then I'll boot the
FAS270's
11:04 [ Polish-2 ] ok
Still no progress.. usually, they are starting
quite fast.
11:06 [ DenBolle ] mhm.. what Can I check now?
11:07 [ Polish-2 ] Did you do the power-cycle of those switches?
I mean - were they powered off?
11:07 [ DenBolle ] they were
and disconnected power cables / reconnected
when I was done.
11:07 [ Polish-2 ] ok let's wait additional 1-2 mins
11:07 [ DenBolle ] ok
and if they aren't up.. that's were the terminal
connection kicks in?
11:09 [ Polish-2 ] I think that before that, we will do do the power
cycle once again.
11:09 [ DenBolle ] ok, I'll wait for your instrcutions
11:09 [ Polish-2 ] terminal connection was meant for BU2DSNTP001 & 002
- if they would be problems after changing IPs
11:10 [ DenBolle ] ah.. I understand
11:11 [ Polish-2 ] this is really wird
weird*
11:11 [ DenBolle ] eh....
don't go there..
11:11 [ Polish-2 ] Den, could you check ping to any other device from
affected subnet (192.46.157.*)?
11:11 [ DenBolle ] sure, let me reconnect to wired network, I'll ping
you when I'm in
11:12 [ Polish-2 ] ok
11:14 [ DenBolle ] ah.. not the cleverest move..
no dhcp server
11:14 [ DenBolle ] let me fetch it in a other way'
11:14 [ Polish-2 ] ok
11:16 [ Italian ] even without dhcp
192.46.157.* switched should ping,
or am I wrong?
switches
11:17 [ Polish-2 ] yes, you're correct
I'm trying to investigate the problem starting from
their internal network.
11:17 [ DenBolle ] ok, isolated the problem..
pinging internally works..
11:18 [ Polish-2 ] to FC switches as well?
11:18 [ DenBolle ] pinging outside (eg. my own site) no go...
ips for those?
11:18 [ Polish-2 ] 192.46.157.133
f.e.
11:18 [ DenBolle ] jep, no problem...
11:19 [ Polish-2 ] ok, so we have some network problem..
11:19 [ DenBolle ] and default gateway also replies : .213
so let me rerun my cabling and I'll get
back in say 10 mins
11:19 [ Polish-2 ] ok, thx
11:30 [ DenBolle ] cables seems in order, the riverbed device seems to
have an issue. So I rebooted it, but it doesn't show
links... what IP's did we use? perhaps a msitake
and used a ip from the riverbed?
11:31 [ Polish-2 ] I've used following IPs for BU2DSNTP001 & 002:
192.46.157.166
192.46.157.167
However, they aren't even up.. So this is probably
not the issue.
11:31 [ DenBolle ] agree.
I can ping internally, no issues.
outside : doesn't work
how can we verify the correct function on the
riverbed?
11:33 [ Polish-2 ] Unfortunately, I am not a network specialist and do
not have any knowledge about that...
[Pg-1], [Ig-1] - any ideas?
11:33 [ Polish-1 ] No idea at all.
11:34 [ Italian ] I'm looking for someone from network team ......
11:34 [ DenBolle ] ok
11:34 [ Polish-2 ] from Poznan, they are all offline.
11:35 [ DenBolle ] wait just a second
to get the cable truogh I had to reconnect one cable
on the back of the riverbed, let me fetch the cabling
diagram
11:37 [ Polish-2 ] ok
11:50 [ Italian ] I don't understand
we don't touch any netw appliance
only FC switches ....
11:51 [ Polish-2 ] Den mentioned that he was doing reorganization
in LAN switches (and/or Riverbed device) -
this probably is causing problem
YES!!!
FC switch pingable!
11:51 [ Italian ] it works
Great job Den
!!!!
11:52 [ DenBolle ] pfoe..
my mistake!
11:52 [ Polish-2 ] :-))
11:52 [ DenBolle ] ok, back to business.. : I'll power up the FAs270?
11:53 [ Polish-2 ] yes, please
11:53 [ DenBolle ] ok, just a sec..
[notes to self to stay the hell away of those
cables.. ]
done, bootup in progress
11:54 [ Polish-2 ] great, I'm monitoring the.. ping to them.
let's hope that here we won't have problems...
11:54 [ DenBolle ] don't worry.. I know what went wrong..
network 101
11:54 [ Polish-2 ] hell yeah!
BU2dsntp001 and BU1dsntp001
up
11:55 [ Italian ] 2 good news in 2 mins .... my heart could stop
11:55 [ DenBolle ] har har..
we're a good team, it's dun to work like thois
this
dun = fun..
12:00 [ Polish-2 ] ok, the whole storage booted just fine..
now I'm going to make mess with it
12:00 [ Italian ] ok
12:02 [ DenBolle ] so at this point, we about 2,5 hrs behind on
schedule..
12:02 [ Italian ] yes
12:02 [ DenBolle ] sorry.
12:02 [ Italian ] np
How long [Pg-2]?
12:02 [ Polish-2 ] i think max. 30 min
12:03 [ Italian ] great
12:03 [ DenBolle ] So you're making up for my mistake, and then
we're back on schedule..
cool
12:03 [ Italian ] don't worry [Pg-2] even 1 hour
12:03 [ DenBolle ] no, I'll stay onsite as long as is needed.
12:29 [ Polish-2 ] OK - seems to be done.
LUNs from BU2 storage are remapped to the new ESX
hosts with IDs starting from 10.
Zoning on the FC switches is also reconfigured.
12:31 [ DenBolle ] ok, so next steps are powering up ESX hosts?
12:31 [ Polish-2 ] [Pg-1] - I think that you can start ESX hosts
now. Please check the connection to the storage
before starting up the VMs.
12:31 [ Polish-1 ] OK. Please let me celebrate for a while this moment
when world's eyes are on me.
12:31 [ Polish-2 ] haha
12:32 [ DenBolle ] hahaha. Big aplause for you !
12:34 [ Polish-2 ] I've made it in 30 mins - I can afford to eat
breakfast now.
12:34 [ Polish-1 ] I'll power them up one by one.
BU1sxve001 goes first
12:34 [ Polish-2 ] OK, I will be watching.. on the FC switches.
As per FC point of view - it's up. And each Netapp
can see it.
12:59 [ Polish-1 ] all 3 ESX host are up but they can't see new LUNs.
I performed rescan onstorage but it not helped.
I need to check this.
13:01 [ Polish-2 ] I can only say that from my side all seems to be
correct. Netapps see each ESX and those ESX are logged
in on Netapp adapters.
FYI: There should be 4 new LUNs visible,
with IDs 10,11,12,13.
13:03 [ DenBolle ] ok, np. take your time to find out whats going on.
14:05 [ Polish-1 ] OK. Now I can see all LUNs on all 3 ESX hosts.
@[Pg-2]: We need to verify paths now.
Please provide me paths for LUN S1V1 (108,5GB)
14:07 [ Polish-2 ] Sure thing.
You have to to tell me this LUN id.
Because we have now double pairs of S1V1,
S1V2, and so on...
14:08 [ Polish-1 ] LUNs from BU2echt are described as:
S1V1_BU2
S1V2_BU2 and so on
14:09 [ Polish-2 ] ok, so first the LUNs from BU1ST Netapps:
14:09 [ Polish-1 ] S1V1
14:10 [ Polish-2 ] S1V1 (880.1g, LUN id: 0)
S1V2 (805g, LUN id: 1)
S1V3 (563.1g, LUN id: 4)
Correct path for them is: 500a098195065a78
14:10 [ Polish-1 ] This should be set as prefferd right?
and active
14:11 [ Polish-2 ] S2V1 (955.1g, LUN id: 2)
S2V2 (730.1g, LUN id: 3)
S2V3 (563.1g, LUN id: 5)
Correct path for these is: 500a098185065a78
Correct. It should be considered as preferred
(primary) path.
BU2 LUNs:
S1V1 (840g, LUN id: 10)
S1V2 (840g, LUN id: 11)
Correct path: 500a098195364ca3
S2V1 (840g, LUN id: 12)
S2V2 (840g, LUN id: 13)
Correct path: 500a098185364ca3
Uff.. that's all.
14:14 [ Polish-1 ] ok please give me a few minutes to check all hosts
14:29 [ Polish-1 ] Ok. I verified everything and corrected paths.
Now I suggest to power on domain controller.
VM in BU1.
14:29 [ Italian ] ok for me
14:30 [ Polish-1 ] then I'll add to inventorythose machines which were
migrated.
14:31 [ DenBolle ] ok, well, so far all goes well.
14:32 [ Polish-1 ] Den, an all BU1st machines the second DNS entry is
set to the BU2 DC. I suggest to change it to
STVSXDMC018 147.184.54.199 and the primary DNS leave as
is (BU1SXDMC002).
14:33 [ DenBolle ] ok, do so, do you want me to submit a ticet for it?
14:33 [ Polish-1 ] no.
14:36 [ Italian ] [Pg-1], about BU2SDWN001 (FV) how long does it take
to add to inventory ?
14:36 [ Polish-1 ] it should be one click.
I'll do it now.
14:37 [ Italian ] thks
14:40 [ Italian ] I see it
14:43 [ Italian ] please remember to change IP
14:50 [ Polish-1 ] BU2SDWN001 is working under new IP now.
Now I will add BU2SXFILE001 to the inventory
and change the IP.
Then I'll call DNS team to perform DNS changes. OK?
14:51 [ DenBolle ] ok
14:51 [ Italian ] moment
DNS is already updated ?
14:51 [ Polish-1 ] no
14:51 [ Italian ] ok
go ahead
and I told C. to wait
for tests
14:51 [ Polish-1 ] ok
15:23 [ Polish-1 ] I powered on BU2SXFILE001 and changed IP. Then I added
a new disk to BU1SXFILE001 and powered it up.
Now I catch up with network team for DNS changes.
15:23 [ Italian ] ok
15:25 [ Italian ] ANy idea about new ip propagation time ?
15:29 [ Polish-1 ] [Pg-2]osz Ostrowski is workin on it now. He does not
know how log it may take as he's not doing it usually.
Only today was caought by me.
cought*
15:31 [ Italian ] because I suppose you already started DNS change
activity, from my experience it could take also
24 hours to propagate
I don't know if is possible to force it
15:31 [ Polish-1 ] I provided him numbers of those 2 requests.
15:33 [ DenBolle ] and a forced propagation isn't a option?
15:33 [ Italian ] Den What's your idea about FV tests, in the worst case
can we done them tomorrow ?
15:35 [ DenBolle ] I think we can, let me verifyt with C. on this one.
Its's possible to do it from home for him
15:39 [ Polish-1 ] So far from Poznan wi have this:
C:\>ping BU2sdwn001
Pinging BU2sdwn001.lan01.com [192.46.157.162] with
32 bytes of data:
Reply from 192.46.157.162: bytes=32 time=63ms TTL=119
Reply from 192.46.157.162: bytes=32 time=63ms TTL=119
Reply from 192.46.157.162: bytes=32 time=64ms TTL=119
Reply from 192.46.157.162: bytes=32 time=63ms TTL=119
Ping statistics for 192.46.157.162:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 63ms, Maximum = 64ms, Average = 63ms
15:41 [ Italian ] the same from Italy ....
15:42 [ DenBolle ] I've talked with C., tomorrow testing is possible,
but before 1000 hrs, he has other apointments
15:42 [ Italian ] I think he caould start test now
DNS looks updated
15:43 [ DenBolle ] ok, I'll inform him.
15:43 [ Italian ] thks
15:43 [ Polish-1 ] I'm gonna start next VMS
ok?
15:43 [ Italian ] yes please
15:44 [ DenBolle ] C. wil start testing right away.
16:00 [ Polish-2 ] [Pg-1], are you starting the other VMs as well?
I need to check the BU1SMLXV0001 server.
16:01 [ DenBolle ] C. reports an issue : mq service isn't running,
I'm working on that.
16:02 [ Italian ] ok in case we can open a ticket to GSA-WW-MIDDLEWARE
16:07 [ DenBolle ] ok, problem killed.. C. tests further.
16:07 [ Italian ] good job "terminator"
16:16 [ Polish-1 ] I'm starting VMs one by one.
16:16 [ DenBolle ] ok, al seems in good working order ?
16:17 [ Polish-1 ] so far so good
16:17 [ Italian ] [happy]
16:17 [ DenBolle ] you optimistic guy
16:32 [ Polish-1 ] BU1SMLXV0001 is up now
16:32 [ Polish-2 ] thanks, I've just finished checking it
all seems to be fine, all filesystems mouted,
oracle processes started
By the way, after the action, we need to raise a high
priority ticket to DBA team with request to check
databases on affected servers. I discussed this
with [naam] from DBA, and he asked me to do so.
16:34 [ Italian ] good idea
16:35 [ Polish-2 ] How about other VMs? Did they started properly?
No issues yet?
16:35 [ DenBolle ] C. reports no issues on FV and MQ series, [Ig-1],
he'll sent you a test report.
16:35 [ Italian ] fantastic
16:36 [ Polish-1 ] I'm checking them one by one.
We'll need to ask [Pg-3] to check if backup works
fine now.
16:36 [ Italian ] [Pg-1] did you start data copy from BU2SXFILE001 ?
16:37 [ DenBolle ] should his wife be done with hunting now?
16:37 [ Polish-2 ] We'll see.. ;
16:37 [ Polish-1 ] not yet. I need to prepare jobs for SecureCopy.
Fortunatelly I installed it yesterday.
My wife?
16:38 [ Italian ] [Pg-3]'s wife
16:38 [ DenBolle ] No, [Pg-3]'s wife
16:38 [ Polish-1 ] Also Lotus team could be informed to check Domino
server.
16:38 [ Italian ] Den
Can I promote you leader?
16:39 [ DenBolle ] Sure, does it pay more?
16:39 [ Italian ] 65% more
16:39 [ DenBolle ] Give it to me!
16:39 [ Italian ] or 67% don't remember
I really have to go ........
but I'm available on call (if necessary , but i
don't think so, your are in good hands )
16:41 [ DenBolle ] No problem. I have your mobile.
[realises I'm the guy for the coffee now.. ]
16:42 [ Italian ] ehehehe
I'm happy that FV tests were ok
16:43 [ DenBolle ] me to, specially beacuse that's something automated
outside us, and BU2 depend on it..
16:43 [ Italian ] yes
Thanks a lot [Pg-1] and [Pg-2], I didn't do nothing
"today" but I wish you feel my support remotly
16:45 [ Polish-1 ] Thank you too.
16:45 [ DenBolle ] [Ig-1], thx sofar for your help and coordination.
When we're done I'll sent you a txt message to inform
you.
16:45 [ Polish-2 ] Sure, [Ig-1]. It was good to have you here. Thank you!
16:45 [ Italian ] too good guys
Thanks Den a call or a message will be really appriaciate
16:46 [ DenBolle ] I'll do, count on it.
16:46 [ Italian ] grazie
bye bye
16:46 [ Polish-2 ] bye
16:46 [ DenBolle ] bye
16:47 [ Polish-2 ] [Pg-1], are all VMs up by now?
If yes, I would raise ticket to DBA team.
16:48 [ DenBolle ] ok, just a short next-steps overvieuw : after al VMS are
up, db checks, inform notes team, and run a test backup.
Anything else?
16:49 [ Polish-2 ] from my side, nothing to add
16:49 [ DenBolle ] ok, [Pg-1]?
16:51 [ Polish-1 ] and copy BU2SXFILE001 to BU1SXFILE001
this can last few hours
16:52 [ DenBolle ] ok, and all would take about say 2-3 hrs from now?
(I've got to inform security here that it's going to
be a little later than 1800 hrs)
16:52 [ Polish-1 ] no idea. It's 500GB to copy.
16:53 [ DenBolle ] thats (500 diveded by 16M memory sticks)
say quite some later than 1800 hrs. I'll keep it
at 2000 hrs ok?
16:54 [ Italian ] [Pg-1] could you be there unil the end of the copy ?
16:54 [ Polish-1 ] ok. If it's not finished until that time you can go
home and I'll let you know via sms if all data is copied.
I'll be here until the end of the copy.
16:55 [ DenBolle ] No, I can stay as long as needed, but I would like to
inform the security guy before he's disapointed.
16:55 [ Italian ] because last step is to stop BU2SXFILE001
16:55 [ DenBolle ] /scared because [Ig-1]'s ghost is here..
16:55 [ Italian ] eheheheh
I'm waiting for my wife
16:56 [ Polish-1 ] I know but I have no info from K. if he checked if
we have access to every file. If I need correct
something manually it may last longer.
16:56 [ Italian ] she has to drive me home
16:57 [ DenBolle ] ok, for that lets see what happens when the copy
part is starting. And have a chat about that then.
First lets see that we have the DB's checked and
Domino team informed.
16:57 [ Polish-1 ] one moment. I have an urgent reboot in Canada.
16:58 [ Polish-2 ] Den, I am wondering whether you could raise a ticket
to DBA team when all VMs are up?
16:58 [ DenBolle ] sure, np, give me the RA and I'll submit one
16:58 [ Polish-1 ] in a minute or two
16:59 [ Italian ] TECH-CENTRE-DATABASE-SUPPORT
16:59 [ Polish-1 ] right
16:59 [ Polish-2 ] if this is a minute or two, I think that I will
manage to do that by myself.
17:00 [ Polish-1 ] ok
all VMs are up now
17:02 [ DenBolle ] allright, who's going to submit the ticket?
to prefent a double entry.
17:02 [ Polish-2 ] I am going to submit the ticket to DBA team.
17:02 [ DenBolle ] ok
17:05 [ Italian ] good luck and hope you finish asap
17:05 [ DenBolle ] Thx, and snjeoy your evening!
snjeoy = enjoy
17:06 [ Italian ] :-)
17:06 [ Polish-2 ] Ticket to DBA: UKIM20002972635
17:07 [ DenBolle ] ok, let see if they pcik it up quickly
17:08 [ Polish-1 ] I talked to [Pg-3]. He'll check backups about * PM
today and he'll send us an email with update.
17:08 [ DenBolle ] ok, so that's covered. What about Domino team?
17:08 [ Polish-1 ] I'm creating a ticket now
17:09 [ DenBolle ] Great, this is going well guys, thx a lot for
your effort on this one.
17:11 [ Polish-1 ] UKIM20002972653
17:11 [ DenBolle ] ok, thx
17:32 [ Polish-1 ] OK I started copying data from BU2SXFILE001
to BU1SXFILE001
1 min 200 MB
2 min 410 MB
3.73 MB/s
17:34 [ DenBolle ] mhm..
if I calculate right 200 mb / min = 1Gb/ 5 min
--> 12Gb / hrs..
17:35 [ Polish-1 ] looks like this
17:35 [ DenBolle ] ie.. whole copy will take approx 40 hrs..
and not ready monday morning..
17:36 [ Polish-1 ] 5 min and 1.16GB
now performance is about 4 MB/s
17:38 [ DenBolle ] A. (messaging team) tells that domino is working fine..
17:38 [ Polish-1 ] ok
17:39 [ Polish-2 ] No one from DBA haven't picked up the ticket yet.
I'll try to contact them now.
17:40 [ DenBolle ] ok.
[Pg-1] : lets wait a litte to see if it speeds up,
but 4mb/sec is to slow to get ready
17:41 [ Polish-1 ] ok now I have 3.61MB /s
and 2.17GB copied
we can leave BU2SXFILE001 working and correct DNS entry.
The migration can be done later if the server is
accessable for users after migration. What you think?
17:43 [ DenBolle ] like having a shadowcopy in place and then just a smal
migration?
17:44 [ Polish-1 ] migrate files during the week and during the weekend
copy only files which changed.
17:44 [ DenBolle ] yeah, that's the best approach I think,
sync the files, and switch in the next weekend.
how does that goes with the change we requested?
can we fit it in?
17:45 [ Polish-1 ] We can do an exception. it's always possible.
3.34MB/s now
17:46 [ DenBolle ] ok, so the chance that it will speed up significant
is low, let's go for the 'hidden sync' and migrate
next weekend then.
in this speed we're doomed
17:47 [ Polish-1 ] ok so I need to rise a ticket for a DNS entry change.
Then we will check if it works. I'll keep copying task
running during the weekend to have the main part
already copied.
A new change request will be required but I'll
prepare it.
17:48 [ DenBolle ] ok, and the DNS is only for the BU2SXFILE001 right?
17:48 [ Polish-1 ] right
this will point now to the new address
192.46.157.168 right/
17:49 [ DenBolle ] let me fetch that one, it...
17:56 [ DenBolle ] just a sec again.. I've got .164 ..
let me fect the documentation to [Ig-1]
17:56 [ Polish-1 ] ok
17:57 [ DenBolle ] The last IP address 192.46.157.168 has been required as
a temporary address in case of any issue on migrated VMs.
(Temporary emergency Vms component)
17:58 [ Polish-1 ] From changerequest:
3. 15 min. Modify the IP address and default
gateway of BU2SDWN001 as per
configuration items changes.
Change the IP of BU2SXFILE001 to
192.46.157.168 and apply BU1 network
settings.
To provide screen shot of new IP
config.
17:58 [ DenBolle ] yeah, it's ok. use it
17:59 [ Polish-1 ] ok so I'm rising a ticket for DNS change.
17:59 [ DenBolle ] wait..
17:59 [ Polish-1 ] ok
18:03 [ DenBolle ] mhm.. this one isn't clear. the doc from [Ig-1]
(23/12/2010) states .164 as FS ip.
and the 168 as emergency...
Go ahead: use .168: this is an emergency.
18:03 [ Polish-1 ] ok
18:04 [ DenBolle ] I'll update [Ig-1] on this per email.
18:08 [ Polish-1 ] 7002306
18:10 [ DenBolle ] ?
18:10 [ Polish-1 ] request number
to network for DNS change
18:10 [ DenBolle ] ah ok
[Pg-2], any luck on the DBA team?
18:13 [ Polish-2 ] not yet, I wrote a SMS to person on-duty,
without any luck yet
I'll wait additional 5-10 mins and I will call them
18:13 [ DenBolle ] ok, in that case I'll hand out the coffee
18:13 [ Polish-2 ] ok
18:14 [ DenBolle ] for [Pg-1], and this one for you [Pg-2]osz
18:14 [ Polish-2 ] thx a lot
18:14 [ DenBolle ] mail from [Pg-3] :
Hello All,
Just FYI. The backup service is OK after the migration.
I have just started all quarter jobs except BU1SXFILE001
that is during data copy. So far so good for the backup
side.
Regards,
[Pg-3]
18:14 [ Polish-2 ] good
18:26 [ Polish-2 ] weird.. the guy who is supposed to be on-duty from DBA
team is not answering neither the SMS and the phone calls..
18:26 [ DenBolle ] and a 'second' one isn't available?
18:27 [ Polish-2 ] I will try to call the 'second' one
18:31 [ Polish-1 ] 21,72Gb in 1 hour
18:31 [ DenBolle ] hey..
18:31 [ Polish-2 ] so it seems that speeded up?
18:32 [ DenBolle ] thats twice what we expected..
18:32 [ Polish-2 ] 25h for 500GB
18:32 [ DenBolle ] ie.. tomorrow al is done?
18:32 [ Polish-1 ] now 6.17 but we don't know how many small files are befor
us
18:32 [ DenBolle ] if no files have issues that is.
18:32 [ Polish-1 ] no it's about 26 hours
it's still risky
18:33 [ DenBolle ] lets keep the BU2SXFILE001 alive.. to be save.
18:33 [ Polish-1 ] yes
18:34 [ DenBolle ] and stick to the plan. A dns change is easy.
If all is save we can do it perhaps earlier then next
weekend.
18:36 [ Polish-1 ] sure
18:49 [ Polish-2 ] I've contacted [naam] from DBA at last..
He has muted his phone and left it in thie jacket's
pocket.. and didn't hear that it was ringing.
He will be on-line in a minute and check DBs.
18:50 [ DenBolle ] pfoe.. I assume we can recognize [naam] at his black
eyes monday?
19:00 [ Polish-1 ] C:\>ping BU2sxfile001
Pinging BU2sxfile001.lan01.com [192.46.157.168] with
32 bytes of data:
Reply from 192.46.157.168: bytes=32 time=95ms TTL=119
Reply from 192.46.157.168: bytes=32 time=92ms TTL=119
Reply from 192.46.157.168: bytes=32 time=77ms TTL=119
Reply from 192.46.157.168: bytes=32 time=75ms TTL=120
Ping statistics for 192.46.157.168:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 75ms, Maximum = 95ms, Average = 84ms
C:\>
19:01 [ DenBolle ] great thx. Solved this one. After DBA verification we're
done for today ?
19:02 [ Polish-1 ] Please check if you can map shares without problems.
If yes, we can say that's all for you today.
19:07 [ DenBolle ] BU2sxfile001 resolves to 192.46.157.160 --> BU1sxfile001
when was the dns set?
19:08 [ Polish-1 ] I talked to this guy. This will be set in a minute.
let's wait for a while. They also have a tough day.
19:08 [ DenBolle ] sure, np..
19:23 [ DenBolle ] on the BU1SXFILE001 the BU2SXFILe001 resolves to
192.46.157.168, but on the network it doesn't,
I'll wait some more.
19:24 [ Polish-1 ] [Pg-2] is working on it now.
19:25 [ DenBolle ] sure, no hurries on this one
19:25 [ Polish-1 ] He had a call to Riga
network outage till monday
19:25 [ DenBolle ] in Riga?
19:25 [ Polish-1 ] yes. Packets get lost from time to time.
my remedy queue is blue
19:26 [ DenBolle ] because you're working all day allready ?
19:26 [ Polish-1 ] yes
19:27 [ DenBolle ] grin grin..
just keep up a little.. we're almost done for today
19:27 [ Polish-1 ] you are almost done
I'm till midnight
19:28 [ DenBolle ] wow.. thats 18 hrs ?
19:28 [ Polish-1 ] 17
19:28 [ DenBolle ] what happend?
19:28 [ Polish-1 ] when we are on remedy during weekend we have to proceed
all weekend's tasks that's all.
my weekend shift is between 12PM - 24
because today we had BU2migration I had to start earlier.
There was an option
KL...
19:29 [ DenBolle ] in that case : my compliments ! I appriciate it that
you supported us outside your regular shift times.
19:30 [ Polish-1 ] It's always better to have work in one heands that
pass it on and on from person to person.
19:31 [ DenBolle ] I know what you mean.. losing that one detail and then
no one knows what happened.
19:31 [ Polish-1 ] eg.
19:39 [ DenBolle ] allright : in the network the BU2sxfile001 in the
bu1 subnet works at 192.46.157.168 and mapping is
possible. I'm now going to the bu2 network to verify a login
19:40 [ Polish-2 ] A. just sent the info about DBs: all is fine.
In that case, I think that my work for today is done.
19:42 [ Polish-1 ] ok. thank you for your assistance. Have a nice sleep.
19:44 [ Polish-2 ] Thank you for your support. I wish you [Pg-1] a calm
on-duty time. I hope that you'll be able to relax tomorrow.
19:44 [ Polish-1 ] thanks, I hope so.
20:19 [ DenBolle ] ok, the followme system has issues. I've shut down
all printers and will restart them one by one
20:19 [ Polish-1 ] ok and now?
20:20 [ DenBolle ] I wait a few mins to calm the server.
It can't handle the amount of config requests,
and then I'll boot up the printers again
20:20 [ Polish-1 ] ok
20:22 [ Polish-2 ] OK, I'm logging off now.
@Den: Thanks for cooperation. I hope that all remaining
issues will be resolved soon.
Thank you guys, have a nice Sunday!
bye
20:22 [ Polish-1 ] bye! Thank you!
20:43 [ DenBolle ] [Pg-1], I need a reboot on the BU1SPWN001 ...
the priner server isn't acting wel
20:46 [ Polish-1 ] ok I'll do it right now.
20:46 [ DenBolle ] great
20:50 [ DenBolle ] and?
20:50 [ Polish-1 ] server is booting up now
20:50 [ DenBolle ] top, thx
20:53 [ DenBolle ] ah.. it's also the DHCP server...
20:53 [ Polish-1 ] yes
20:53 [ DenBolle ] lost my connection
I'm on wifi, is it back on ?
20:55 [ Polish-1 ] yes the server is back on
20:56 [ DenBolle ] ok, let me see that I'm getting the printer back on
20:56 [ Polish-1 ] ok
21:48 [ DenBolle ] ok, I'm back.. All printers tested and are working.
21:48 [ Polish-1 ] ok good.
what next?
21:48 [ DenBolle ] beer?
21:48 [ Polish-1 ] I'm at work
21:49 [ DenBolle ] I've got to drive the motor for 1,5 hrs...
then beer
We're done I think. Db's checked, mappings checked,
printers checked, all works, only thing now is
copying the data
21:50 [ Polish-1 ] 83Gb so far and 5.52 MB/s
21:50 [ DenBolle ] so we're done in the majority of task
21:50 [ Polish-1 ] I think yes. I'm happy it worked.
21:54 [ DenBolle ] me to.
Thx a lot for all. I'll inform the users.
Next weekend the last step ?
21:55 [ Polish-1 ] I think yes. I'll contact you in the week.
Thanks you. Have a nice end of weekend.
21:55 [ DenBolle ] ok, excellent. Thanks for your help and
patience. Hear you next weekend.
Cookie policy
Daar zaten wel een paar momenten tussen waar je het zweet van in de handen krijgt 🙂
Vooral vm’s met multipathing, dan moet je de kop er wel even rustig bij houden.
Onder het mom van : als je het niet wil dan wel en als je het wel wil dan niet…
(Heuh? : welnu komtie:)
We hebben hier een 4-tal oude ml350 (proliants) staan, met allerlei raid combinaties, 78 en 36 GB verdeeld over verschillende raid sets op verschillende servers. Die disken moeten leeg, dus ik ‘schud’ ze even. (Alles eruit, door elkaar husselen en hop, random terug duwen en laten initialiseren…) Hoe het kan weet ik niet, maar toch mooi dat ik stomverbaasd 3 minuten later naar een windows 2003 server inlogscherm stond te kijken.
Wel een Blue screen natuurlijk na het inloggen, maar toch! (zie je wel,ik doe ook spannende dingen 🙂 )