A reason why I love our database administrator, is because he’s facing some really interresting problems !

For exemple last week he tries to copy a huge mysqldump from a slave in Datacenter 1 to another mysql server in our Datacenter 2… and I help him to optimize those recurent transfer between our datacenters…

We studied different solutions to improvethe speed of transfer and find some solutions to retrieve some transfer when they fail.

1- What you should not do gzip / scp / gunzip

gzip -c mysqdump.sql >mysqldump.sql.gz
scp mysqldump.sql.gz root@master_dc2:/var/database/tmp/
ssh root@master_dc2 "gunzip /var/database/tmp/mysqldump.sql.gz"

naaarf  …. not the best choice, you will have  to zip your datas, then copy (using encryption), and unzip on the remote server…

Some reasons why I don’t like this solution :

• - All steps will be serialized (we’ll see later that we can avoid at least one step)
• - Step 1 will need a lot of reads from and writes to disks, which is really slow
• - Step 1 requires a lot of CPU
• - Step 2 needs cpu to encrypt all your datas, and will read from  slave@dc1 and writes@dc2
• - Step 3 (same issue than step 1)
• - Step 3 need you to have a large hard drive (i.e. if your dump size is 50GB zipped, you will need at least 100GB on your hard drive)

This Solution failed because of the point 5, we don’t have enough available space to store both the zipped file and the unzipped data.

2 What you can do gzip / ssh / gunzip

Another solution to avoid the write to server 1 et read from server 2 of steps 1 and 2


gzip -1 -c mysqdump.sql | ssh root@master_dc2 "gunzip -c - > /var/database/tmp/mysqldump.sql"


This will read the file mysqldump.sql, zip it (-1 for fast zipping) and pipe the result directly to master@dc2.

The master will directly unzip the file to write the unzipped data on your hard drive.

The resutl is that you will never write on slave@dc1 and will write only one time on master@dc2.

Some reason why I don’t use this solution everythime :

• this requires a lot of CPU, as for encryption as for transferring data through ssh.

3 What you should do gzip / netcat / gunzip

As said before ssh is not the fastest way to transfer data because of the data encryption

When you’re using a dedicated black fiber between your Datacenter you can use netcat to send raw bits over the fiber

Open a listening socket on your client

root@master_dc2# nc -l -p 9999 | gunzip -c > /var/database/tmp/mysqldump.sql

Start sending raw packets over the network

root@slave_dc1# gzip -c mysqdump.sql | nc -q 1 master_dc2 9999

4 What you could do tar / netcat

Pay Attention to the locations of your file that must be the same

Open a listening socket on your client

root@master_dc2# nc -l -p 9999 | tar xvzf -

Start sending raw packets over the network

root@slave_dc1# tar zcvf - mysqdump.sql | nc -q 1 master_dc2 9999

4 retieve your transfer after a network failure

Of course when you’re trying to send more that 35GB over the network, it is hard to be sure that you will not have any broken connection when you have already sent 30GB….

To be sure to have a solution to this type of problem we did some tests for different type of protocols :

Sending datas via SCP

To tranfer some files on our local network, we always use scp because of the encryption and because of the possibility to use rsa/dsa key exchange to script our transfer.

Even if it’s rare, we can face sometimes some problem of failure on our network during transfer, so we find this solution :


root@slave_dc1# scp mysqldump.sql.gz  root@master_dc2:/var/database/tmp/


if the connection screwed up, with a ssh tranfer (scp does not support retrieving) your can use rsync…


root@slave_dc1# rsync --partial --progress -e ssh mysqldump.sql.gz root@master_dc2:/var/database/tmp/


Sending datas via ftp

One of our partners does not support sftp connection, so we have to use ftp with encrypted files to send some informations….

The solution when you lost the connection during the transfer…., is to use dd to split the file and send missing data.

Obviously, we need to know that ftp copies file sequentially, and this is the key to success in retrieving data

How it works :


root@slave_dc1# ls -al myfiletocopy.log
-rw-rw---- 1 mysql mysql 35704013289 2010-07-14 18:31 myfiletocopy.log
root@slave_dc1#ncftpput -u ronan -p ronan partners.org /remoteronan myfiletocopy.log
-----failed tranfer----
root@slave_dc1#ncftpls "-al" -u ronan -p ronan ftp://partners.org/remoteronan/myfiletocopy.log
-rw-rw-rw- 1 ronan ftp 25462988800 Jun 22 2008 VIRG_VALID_200806.zi


Did you see that?

• Original file size : 35704013289
• Remote file size : 25462988800
• What is missing : 10241024489 so approximatively 10241024000 (1024 divisibility)

Lets try to send what is missing :

The common blocksize on a linux is 1024, so to be as fast as possible

Local Site


root@slave_dc1# dd if=myfiletocopy.log iseek=10001000 bs=1024 | nc -l -p 2121


Remote Site


root@partner# nc mycompany.com 2121 | dd of=myfiletocopy.log seek=10001000 bs=1024


You can change the blocksize but 1024 is a common value on ext3 partition so your system will be able to cut your file really quickly when he will have to read it on your hard drive

++

One of the most useful tool for unix debug is lsof, I use it many times for many problems for day to day monitoring and debug.
It just lists all files openned by processes no more no less, but because on *NIX system, everything IS a « file », this gives you a lot of information on your system use (e.g. ttys, directories,sockets, pipes,memory mapping).

1 – List all open files used on your system .

gollum:~# lsof

2 – Unmounting a partition

You’re trying to unmount /dev/sdd1, but it is still in use by a process or a user
gollum:~#lsof /dev/sdd1
and then kill each processes or users to free the partition
=> aggressive mode :
gollum:~#kill `lsof -t /dev/sdd1`

Find some issues with user’s script

List all connection on your server openned by/for non root users
gollum:~#lsof -i -u^root
List all files on your server used by/for user_ronan
gollum:~#lsof -u user_ronan

Find all open files in a directory recursively.
gollum:~#lsof +D /usr/lib64/

- with +D argument lsof will return all files in the specified directory and subdirectories.
another way is using grep (but if your system is using ia32 or if you’re looking in symlinks +D argument will be necessary)
gollum:~#lsof | grep '/usr/lib64/'

Find some issues with programs

Find all open files by program’s name.
gollum:~#lsof -c mysql
-with -c argument lsof will return all files openned by/for processes whose name begins with ‘mysql’.

You can compile multiple option using -a (and):
gollum:~#lsof -a -c apache +D /usr/lib64/

If you don’t specify the -a option by default lsof will return an « OR » result
gollum:~#lsof -c apache +D /usr/lib64/

And you can use ‘^’ translate by except to find all files open by all processes except PID 1
gollum:~#lsof -p ^1

Find some issues with space left

df and du does not match…

You should have a problem with some files that have been deleted but one or more processes are keeping the files open.
The system will preserved the file and its blocks, even though it appears to be deleted from the filesystem.

gollum:~#lsof +L1
another way :
gollum:~#lsof | grep deleted

Too many open files issue

Find the top 10 of procsesses with the most files open.
gollum:~#lsof | awk '{printf("%s (%s)\n", $1, $2)}' | sort | uniq -c | sort -rn | head

You can check the number of script opened by a process using

gollum:~#ps -ef | grep MyScripts*.sh | grep -v grep | awk '{print $2}'
46899
gollum:~#lsof -p 46899
gollum:~#lsof -p 46899 | wc -l


This will give you idea of the files that your script opens.
(for more than one process just separate pids using comas)

gollum:~#ps -ef | grep MyScripts*.sh | grep -v grep | awk '{print $2}'
46800
46801
46802
gollum:~#lsof -p 46800,46801,46802


And what about Network?

As said before, everything on Linux is a file, so IP sockets can also be listed using lsof and help you to know more about your IP system use.
(Of you can also use netcat or netstat for some of those examples)

List all IP sockets openned

gollum:~#lsof -i
gollum:~#netstat -anp

Lsof with -i option lists all processes with open Internet sockets (TCP and UDP).
List all TCP network connections.
gollum:~#lsof -i tcp
List all UDP network connections
gollum:~#lsof -i udp

You can check if a port is open and which process binding this port
gollum:~#lsof -i :80
You can also use the service name listeed in /etc/services :
gollum:~#lsof -i :www

Problem with NFS sharing

You can list all NFS files in use :
gollum:~#lsof -N

Another way to find those informations

This tool uses the information in /proc/$PID/ (man 5 proc) to list the files openned by a process.
Of course you can debug by yourself a process using special commands like

gollum:~#cat /proc/1/maps | awk '/\//{print $6}' | uniq

1 – Introduction

A good solution to get a reliable solution for your Datacenter’s LAN without configurations replication problems is using the CISCO solution VSS.

Basically, you will configure a « Virtual Switch » instead of configuring two switches, and still have the hardware advantage of an hardware redundancy.

The solution implemented named Virtual Switch System 1440 (VSS) on the Catalyst 6500, create a single virtual switch from two 6500 switches

Some interresting features are :

• Redundancy of Hardware and links
• High availability of supervisor (active/standby chassis)
• Usefull system management

Hardware needed

• We use a 720-10GE Uplink on both 6506
• DFC3C in conjunction with th PFC3C Supervisor
• 2 Cisco 3750 Stackwise 48 Ports for Top of the Rack switching
• Ethernet bonding on servers (round-robin or 802.3ad)

With this configuration you SHOULD not have any Point Of Failure…

2 – Description

The Virtual Switching System is based on the Virtual Switch Link (VSL) which a dedicated 10GE link (or link aggregation up to 8 links) that create an extension of the backplane between 6500.

The VSL carries control and data traffic between VSS members thru an Etherchannel. Bascically you have only one configuration and mac-address, as for the 6500 VSS configuration as for 3750 configuration.

The Logical NAD gives you an idea how much the configuration becomes simple with those technologies.

3 – Configuration

3.1 – Enable the required Feature

The VSS works in « active » / « hot standby » mode. The Supervisor manage this feature using Statefull SwitchOver (SSO) and NonStop Forwarding (NSF).

First Router : RTR_TEST1


RTR_TEST1# conf t
RTR_TEST1(config)# redundancy
RTR_TEST1(config-red)# mode sso
RTR_TEST1(config-red)# exit
RTR_TEST1(config)# router ospf 10
RTR_TEST1(config-router)# nsf
RTR_TEST1(config-router)# end


Second Router : RTR_TEST2


RTR_TEST2# conf t
RTR_TEST2(config)# redundancy
RTR_TEST2(config-red)# mode sso
RTR_TEST2(config-red)# exit
RTR_TEST2(config)# router ospf 10
RTR_TEST2(config-router)# nsf
RTR_TEST2(config-router)# end


3.2 – Virtual Switch Domain configuration

The Virtual Switch Domain defines the group id (1-255) of the virtual switches

First Router : RTR_TEST1

RTR_TEST1# conf t
RTR_TEST1(config)# switch virtual domain 100
RTR_TEST1(config-vs-domain)# switch 1


Second Router : RTR_TEST2


RTR_TEST2# conf t
RTR_TEST2(config)# switch virtual domain 100
RTR_TEST2(config-vs-domain)# switch 2


3.3 – Configure Switch Priorities

If you need to know which of your switchs is active by default, you can enforce priority and then configure your preference by giving a higher priority to the active router.

For exemple, if you prefer that RTR_TEST1 becomes the active router you just have to set the following options :


RTR_TEST1(config-vs-domain)# switch 1 priority 120
RTR_TEST1(config-vs-domain)# switch 2 priority 110


3.4 – Configure the Virtual Switching Link (Port Channel)

First Router : RTR_TEST1

RTR_TEST1(config)# interface port-channel 10
RTR_TEST1(config-if)# switch virtual link 1
RTR_TEST1(config-if)# no shutdown
RTR_TEST1(config-if)# exit
RTR_TEST1(config)# interface range tenGigabitEthernet 3/4 - 5
RTR_TEST1(config-if-range)# no shutdown
RTR_TEST1(config-if-range)# channel-group 10 mode on
RTR_TEST1(config-if-range)# end


Second Router : RTR_TEST2


RTR_TEST2(config)# interface port-channel 20
RTR_TEST2(config-if)# switch virtual link 2
RTR_TEST2(config-if)# no shutdown
RTR_TEST2(config-if)# exit
RTR_TEST2(config)# interface range tenGigabitEthernet 3/4 - 5
RTR_TEST2(config-if-range)# no shutdown
RTR_TEST2(config-if-range)# channel-group 20 mode on
RTR_TEST2(config-if-range)# end


3.5 – Applying the VSS configuration

Currently your configuration is not completely setted, you’ll need to « convert » your routers to the VSS mode :

Check that the PFC mode is PFC3C (VSL)


RTR_TEST1#show platform hardware pfc mode


if not set the pfc operating mode :

RTR_TEST1#platform hardware vsl pfc mode


Then launch the convertion (you should be prompted to confirm the action [yes])


RTR_TEST1#switch convert mode virtual
RTR_TEST2#switch convert mode virtual


Executing this mode will require a reload to merge both switches configurations, renumber all ports and to negotiation NSF/SSO etc between chassis and supervisors.

This step creates a VSS system. This will convert all interface names in to a three mode notation, chassis/slot/port.

4 – Status Command


RTR_TEST1#show switch virtual
RTR_TEST1#show switch virtual role
RTR_TEST1#show switch virtual link


You can now configure your Ethernchannel to your stacked 3750.

A good way to improve security in your local network is using ACLs and VLAN on your LAN.

But configuring your network with a lot of vlans becomes quickly boring, and hard to maintain.

A good solution is using 802.1x solution to set the vlan during the user authentication.

This will allow you to get the same configuration for all ports of your switchs, and setting a segmentation with vlans during the authentication of the users of your network.

I will describe in this article how to set up you’re Active Directory and Radius servers to manage the security, and the configuration of catalyst 2960 for 802.1x solution. This article will not describe the ACL’s configuration it will only explain how to split your services in different Subnets.

1- Introduction

You will find below the NAD of the network. We will split 3 services, for exemple Admin, Support and Project :

• Vlan 10 : 10_VLAN_Admin
• Vlan 11 : 11_VLAN_Support
• Vlan 12 : 12_VLAN_Project

We use 3 computers connected through ip-phone to the 2960. The vlan network for IP-Phone is vlan 220.

2- Update the switch to the latest version :

To get the 802.1x solution working you MUST have you’re switch in Version 12.2(44r)SE1 (at least).

2.1- Download the latest version on your switch using TFTP

TFTP server on 192.168.1.2

You can use tftp32 on windows to share the IOS binary.

DIST_04#copy tftp: flash:

Address or name of remote host [192.168.1.2]?

Source filename [c2960-lanbasek9-mz.122-50.SE3.bin]?

Destination filename [c2960-lanbasek9-mz.122-50.SE3.bin]?

Accessing tftp://192.168.1.2/c2960-lanbasek9-mz.122-50.SE3.bin…

Loading c2960-lanbasek9-mz.122-50.SE3.bin from 192.168.1.2 (via Vlan1): !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

2.2- Set your new version to be used after next reload

DIST_04#conf t

Enter configuration commands, one per line.

End with CNTL/Z.

DIST_04(config)#boot system flash:/c2960-lanbasek9-mz.122-50.SE3.bin

2.3- Configuration your switch for ssh authentication

enable secret 5 $1$EEte$azzzzzzABAIrVDF0

username Administrator secret 5 $1$R1.T$oizajedfpazdXqNYXVbWf4.

aaa authentication login default local

ip domain-name virgin.fr

crypto key generate rsa

line con 0

line vty 0 4

privilege level 15

length 0

transport input ssh

line vty 5 15

2.4- Save and Load your configuration before configure 802.1X for rollback

write

reload

3- Configure vlan on the inter-vlan router

3.1- Configure vtp on you inter-vlan router

Wikipedia : VLAN Trunking Protocol (VTP) is a Cisco proprietary Layer 2 messaging protocol that manages the addition, deletion, and renaming of Virtual Local Area Networks (VLAN) on a network-wide basis. This will share the vlan database of your inter-vlan router to simplify the configuration of your network.

conf t

vtp domain Test_VTP

vtp password MyVTPPASSWD

vtp mode server

3.2- Configure your vlan

We will create 3 vlans for each services of the company :

• Vlan 10 : 10_VLAN_Admin
• Vlan 11 : 11_VLAN_Support
• Vlan 12 : 12_VLAN_Project

3.3- Set your new vlan in the database

vlan database

vlan 10 name 10_VLAN_Admin

vlan 11 name 11_VLAN_Support

vlan 12 name 12_VLAN_Project

3.4- Configure vlan interface of your router

The helper address will be used to handle dhcp request to the DHCP server (192.168.1.2) on another vlan (vlan1)

interface Vlan10

description 10_VLAN_Admin

ip address 192.168.10.254 255.255.255.0

ip helper-address 192.168.1.2

!

interface Vlan11

description 11_VLAN_Support

ip address 192.168.11.254 255.255.255.0

ip helper-address 192.168.1.2

!

interface Vlan12

description 12_VLAN_Project

ip address 192.168.12.254 255.255.255.0

ip helper-address 192.168.1.2

4- Configure your Active Directory for 802.1x authentication

We will use en new Security Group to split users in different services. And affect user (Ronan FOUCHER) to the security group (vlan_admins): Now you’re Active Directory is configured for 802.1x, you can configure your Authentication server (IAS for exemple)

4.1- Configure the Active Directory Users

We use a new Security Group instead of global group to split Users because this allow you to control change in the organization.

Create a security group for each service and affect the user to the new Security Group.

4.2- Configure The Radius Server

4.2.1- Create Remote Access policy for each vlan

Start->Administrative Tools->Internet Authentication

Service Remote Access Policies->Right Click->New Remote Access Policy Wizard

4.2.2- Configure the IAS to accept request from your switch

4.2.3- Configure your DHCP server

We use option 43 and 150 for IP-Phone configuration.

There’s no special configuration for the DHCP server.

5- Configure your CISCO Catalist 2960 for 802.1x

5.1- Configure the authentication server on your switch

aaa new-model

aaa group server radius Testradius

server-private 192.168.1.2 auth-port 1812 acct-port 1813 key myownsecretkey

aaa authentication dot1x default group Testradius

aaa authorization network default group Testradius

aaa session-id common

radius-server vsa send authentication

clock timezone fr 2

ntp clock-period 36029068

ntp server 192.168.10.1

5.2- Configure your Switch as a VTP client

conf t

vtp domain Test_VTP

vtp password MyVTPPASSWD

vtp mode client

5.3- Test the radius authentication

TEST_SW#test aaa group Testradius ronanfoucher mytestingpwd legacy

Attempting authentication test to server-group Testradius using radius

User was successfully authenticated.

(If the authentication failed, you should find some information in the Event Viewer or your IAS)

6- Configure The VoiP on your catalyst 2960

mls qos map policed-dscp 24 26 46 to 0

7- Configure the ethernet ports for 802.1x authentication

We use the vlan 400 for guest vlan and 220 for VoIP.

interface range FastEthernet0/1-24

8- Configure you computers for 802.1x

8.1- Using GPO

http://technet.microsoft.com/en-us/library/dd348501%28WS.10%29.aspx

8.2- On each Computer

http://technet.microsoft.com/en-us/library/dd348470%28WS.10%29.aspx

Configure Alteon using for a Web Application

Below you will find some steps to configure a Web Application with a mysql server (master/Slave-slave) and Website – webservices, with an Alteon Applcation Switch

First Time Installation

Reset the configuration

Connect the Alteon using a Serial Cable and your Laptop 9600/8/n/1

To reset the password you can use : admin/ForgetMe!

(or forgetMe! depending on your hardware version)

And Then erase complitely the configuration


/boot/conf
factory(y)


After the reboot you should not configure using the wizard (it’s like the CISCO wizard, it helps only when you really don’t know what you have to do

Configure the Management Interface

First of all you will need to configure the management interface to configure remotely you’re Application Switch :


/c/sys/mmgmt
addr 192.168.30.10
mask 255.255.255.0
broad 192.168.30.255
gw 192.168.30.254
dns mgmt
ena
/c/sys/sshd/ena
/c/sys/sshd/on


This will configure the ip address for ssh management, you should now be able to configure your switch (switchport mode access//swictport access vlan 30)

Configure The Network Vlan Interface

As you can see in the Network Diagram to get the best flexibility on the Alteon Platform, we are using the vlan trunking to limit the use of ports on the Alteon and limit the physical intervention on the Alteon.

Enable Vlan tagging

We are using the port 1 for inbound/outbound traffic


/c/port 1
tag ena


Vlan Layer 2 Configuration

Let’s configure all vlans


/c/l2/vlan 20
ena
name "VLAN 20 FRONT NET"
learn ena
def 1 2
/c/l2/vlan 21
ena
name "VLAN 21 WEB SERVICES"
learn ena
def 1 2
/c/l2/vlan 22
ena
name "VLAN 22 VIPs NET"
learn ena
def 1 2
/c/l2/vlan 30
ena
name "VLAN 30 Management"
learn ena
def 0
/c/l2/stg 1/add 1 20 21 22 30


USAGE

• def : the VLAN is defined to be spread over the ports listed
• stg : the vlan will be setted with the STP protocol enable (not necessary) you should disable this feature in Production context

  /cfg/l2/stg 1/port 1/off

Configure the Network Interfaces

You will configure the ip address on each subnet configured before

You will need to configure your servers to use those IPs as Gateways, in this simple network we won’t use the DSR features to avoid configuring the Alteon as a gateway

(I will write an article to implement this feature later)

Configure Ip adresses


/c/l3/if 1
ena
ipver v4
addr 192.168.20.252
mask 255.255.255.0
broad 192.168.20.255
vlan 20
/c/l3/if 2
ena
ipver v4
addr 192.168.21.252
mask 255.255.255.0
broad 192.168.21.255
vlan 21
/c/l3/if 3
ena
ipver v4
addr 192.168.22.252
mask 255.255.255.0
broad 192.168.22.255
vlan 22


Configure the routing table


/c/l3/gw 1
ena
ipver v4
addr 192.168.20.254
/c/l3/gw 2
ena
ipver v4
addr 192.168.21.254
/c/l3/gw 3
ena
ipver v4
addr 192.168.22.254


Configure the Load Balancing Part

Enable Layer 4 (Load balancing)


/c/slb
on


Configure your real server


/c/slb/real 1
ena
rip 192.168.20.1
inter 15
retry 2
restr 2
name "FRONT1"
/c/slb/real 2
ena
rip 192.168.20.2
inter 15
retry 2
restr 2
name "FRONT2"
/c/slb/real 3
ena
rip 192.168.20.3
inter 15
retry 2
restr 2
name "FRONT3"
/c/slb/real 4
ena
rip 192.168.21.1
inter 15
retry 2
restr 2
name "WEBSERVICE1"
/c/slb/real 5
ena
rip 192.168.21.2
inter 15
retry 2
restr 2
name "WEBSERVICE2"
/c/slb/real 6
ena
rip 192.168.21.3
inter 15
retry 2
restr 2
name "WEBSERVICE3"


USAGE :

• ena (enable)
• rip (real IP address)
• inter (Health check interleave)
• retry (number of try before server considered ad dead)
• restr (number of try before server considered as alive)

Configure your loadbalanced group

We’ll now create 2 group for each web cluster, you can test different parameters to check the ealth of servers in the group.

I use a webpage named index.html for all servers, the Application Check will launch every « inter » seconds, a connection to each servers to check the connectivity and the service.


/c/slb/group 1
health http
viphl dis
content "/index.html"
add 1
add 2
add 3
/c/slb/group 2
health http
viphl dis
content "/index.html"
add 4
add 5
add 6


the add option describe the real server id

Configure the Virtual Ips for each services

Configure client / server mode

The port 1 is used for inbound traffic to VIPs and OUtbound traffic to real server, so you need to enable both features on port 1


/c/slb/port 1
client ena
server ena


Declare each virtual IP for HTTP services


/c/slb/virt 1
ena
ipver v4
vip 192.168.22.1
vname "FrontWebsiteVIP"
avail 43
remrule 1
/c/slb/virt 1/service http
group 1
hname "HTTP_FrontWebsiteVIP"
epip ena
/c/slb/virt 2
ena
ipver v4
vip 192.168.22.2
vname "WebServicesVIP"
avail 43
remrule 1
/c/slb/virt 2/service http
group 2
hname "HTTP_WebServicesVIP"
epip ena


Apply and Save Alton’s configuration


apply
save


Configure your webserver

Configure the network


vim /etc/network/interfaces
# The loopback network interface
auto lo
iface lo inet loopback
# The primary network interface
allow-hotplug eth0
iface eth0 inet static
address 192.168.20.1
netmask 255.255.255.0
network 192.168.20.0
broadcast 192.168.20.255
gateway 192.168.20.252 #IP address of the LB


Configure a webpage for test

Hello my name is FRONT 1
Hello my name is FRONT 2
Hello my name is WEBSERVICE 1
Hello my name is WEBSERVICE 2

Now you should be able to test from Client PC 192.168.20.1 a connection to VIP FRONT and VIP WEB SERVICES 192.168.22.1/2 the connectivity.
And from FRONT your web services should be able to call soap services on WEB SERVICES.

Configure Mysql Slave Cluster

The Alteons are not able to check, as for http, the health of a server, so you will need to configure a new service to check the port 3306 on mysql Slave servers

IP configuration


/c/l2/vlan 40
ena
name "VLAN 40 MYSQL SLV NET"
learn ena
def 1 2
/c/l3/if 4
ena
ipver v4
addr 192.168.40.252
mask 255.255.255.0
broad 192.168.40.255
vlan 40
/c/l3/gw 4
ena
ipver v4
addr 192.168.40.254


Configure you real mysql slave servers


/c/slb/real 7
ena
rip 192.168.40.2
name "Slave 1"
/c/slb/real 8
ena
rip 192.168.40.3
name "Slave 2"
/c/slb/real 9
ena
rip 192.168.40.4
name "Slave 3"


Configure the new group


/c/slb/group 3
add 7
add 8
add 9
name "MySQL Slave Cluster"


Configure the new service for mysql


/c/slb/virt 3
ena
ipver v4
vip 192.168.22.3
/c/slb/virt 3/service 3306
group 3
hname "MYSQL_SlaveVIP"
dbind ena
tmout 32768


Configure your mysql servers

You must be aware of the replication Network which splitted from the load balancing network (the amount of data can be critical for the Alteon)

Configure the network


vim /etc/network/interfaces
# The loopback network interface
auto lo
iface lo inet loopback
# The primary network interface
allow-hotplug eth0
iface eth0 inet static
address 192.168.40.1
netmask 255.255.255.0
network 192.168.40.0
broadcast 192.168.40.255
gateway 192.168.40.252 #IP address of the LB


And Now you should be able to connect the the slave server using the Mysql Slave VIP

root@192.168.20.1:~#mysql -h192.168.22.3 -uuser_test -p

A good way to share folders, CDs, DVDs etc between 2 systems (need fuse) is using SSHFS.
Indeed, ssh can be used to copy remotely folders/files, unfortunately, if you want to synchronize your modification, access a file quickly on a dvd, the solution is a bit useless…

Of course you can use SAMBA or NFS, unfortunately, this type of solution needs root access for file sharing, and is not quite simple to implement for a remote access (WAN Sharing)…

1 – Sample Configuration

A simple example, is to use SSHFS to access your $HOME on the system solarisX at work and mount it on your local $HOME/mnt_skylwalker at home.
(you wan do exactly the same thing with your dc drive, a USB Storage, …)

Install needed package


gollum:~$ apt-get install sshfs

Dependencies fuse-utils libfuse2 libglib2.0-0 sshfs

Network diagram

Configure your system

To use fuse, you will need to add you user to the groupe fuse :

usermod -a -G fuse bobze_


Mount your new sharepoint

It’s simple as that :

bobze_:~#whoami
bobze_
bobze_:~#mkdir mnt_skylwalker
bobze_:~#sshfs skywalker@working_site.com:/home/skywalker/ /home/bobze_/mnt_skylwalker



bobze_:~#mount
/dev/mapper/gollum.dc02.virginmobile.fr-root on / type ext3 (rw,errors=remount-ro)
tmpfs on /lib/init/rw type tmpfs (rw,nosuid,mode=0755)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
procbususb on /proc/bus/usb type usbfs (rw)
udev on /dev type tmpfs (rw,mode=0755)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=620)
/dev/sda1 on /boot type ext2 (rw)
fusectl on /sys/fs/fuse/connections type fusectl (rw)
skywalker@working_site.com:/home/skywalker/ on /home/bobze_/mnt_skylwalker type fuse.sshfs (rw,nosuid,nodev,max_read=65536,user=bobze_)


Here we are you can list your remote files using

bobze_:~# ls -al /home/bobze_/mnt_skylwalker


Automatical mounting using /etc/fstab

You will need to share your rsa/dsa keys between your servers (I will write an article about that later)

vim /etc/fstab


and add the line below :

sshfs#skywalker@working_site.com:/home/skywalker/ /home/bobze_/mnt_skylwalker fuse user,noauto 0 0


Description of the NAT implementation

When you interconnect your network and a partner’s network you can face some overlapping problem.
Basically, you often have to hide partner’s network to simplify subnet distribution for your partners. Unfortunately, sometimes, you will have to hide your proper network for only one partner.

For exemple, last week I had to interconnect our DC with a partner’s L2 network, using 2 Routers configured with GLBP… In theorie, it should not be difficult using both inside and outside NAT, but I face a strange problem using my configuration…

Network diagram :

My local network is 172.20.5.0/24 and the real partners network (172.19.5.0) is already used for another partner.

This part is simply resolve using the ip nat inside 172.19.5.0 <-> 172.20.13.0 /24

My partner gave me a full /24 subnet for all my L3 equipments and the NAT subnets to hide my subnet on his network.

NAT restriction

There are 2 problems in this type of configuration :

• I must split my subnet to manage the « outsite NAT » because SNAT does not allow you to nat from outside with the same subnet than the inside Network
• exemple : I want to NAT 172.21.5.0<->172.17.11.0 /24First I need to add in the new static route :

  Router1_1(config)#ip route 172.17.11.0 255.255.255.0 10.10.10.1

  And the static NAT rules :

  
Router1_1(config)#ip nat outside source static network 172.21.5.0 172.17.5.0 /27 add-route
Router1_1(config)#ip nat outside source static network 172.21.5.0 172.17.5.0 /27 add-route


  From 172.21.5.10 ping 172.17.5.1

  
Router2_1#sh ip nat translations
Pro Inside global Inside local Outside local Outside global
--- --- --- 172.17.5.0 172.21.5.0
icmp--- --- 172.17.5.10 172.21.5.10
--- --- --- 172.17.5.0 172.21.5.0
--- 172.19.5.0 172.20.13.0 --- ---


  If you launch a trace on you FW you will see the echo from 172.17.5.10 and reply to.

  But when router R2_1 or R2_2 receive the reply packet from 172.17.5.1 to 172.17.5.10 it won’t reach the outside SNAT layer, the packet is dropped by the L3 because it is a part of the directly connected subnet on Fa0/0.

• I must to split my subnet between my two routers because glbp does not share nat translation use

The solution is to feign routing part between nat pools and interconnection network by splitting the subnet in 3 subnets

So lets configure our 3 routers :

So you will need a piece of paper, a pen, and use VLSM to create 3 subnets.
My solution (not the only one) :
=> 172.17.11.0/25 [Network Interconnection]
=> 172.17.11.128/26 [NAT Subnet Router 2_1]
=> 172.17.11.192/26 [NAT Subnet Router 2_1]

Using CLI you will get something like this :

For Router2_1


Router2_1(config)#ip nat inside source static network 172.19.5.0 172.20.13.0 /24
Router2_1(config)#ip nat outside source static network 172.21.5.0 172.17.11.128 /26 add-route
Router2_1(config)#ip route 172.19.5.0 255.255.255.0 172.17.5.1


For Router2_2


Router2_2(config)#ip nat inside source static network 172.19.5.0 172.20.13.0 /24
Router2_2(config)#ip nat outside source static network 172.21.5.0 172.17.11.192 /26 add-route
Router2_2(config)#ip route 172.19.5.0 255.255.255.0 172.17.5.1


Configuring the NetApp Filer can be sum-up in 4 steps :
1. Create the flexible volumes that will store the LUNs for iscsi Servers.
2. Create the LUNs that Servers will use as a new partition.
3. Create the initiator group(s) (igroups) that will be used to log server to the NetApp Filer.
4. Map the LUNs to the appropriate igroup.

Let’s go for the configuration :
FYI : In this exemple we’ll create a volume for a server named gollum and use 95% of this volume vol_gollum for the iscsi partition.

1. First Launch of Netapp

You can refer to the article for basic configuration of you fas250

2. Configure the FAS 250 volume

2.1. Check the space left on aggregate


fas250> aggr show_space -h


Aggregate 'aggr1'
Total space WAFL reserve Snap reserve Usable space BSR NVLOG A-SIS
1593GB 159GB 71GB 1362GB 0KB 0KB
Space allocated to volumes in the aggregate
Volume Allocated Used Guarantee
vol_1 402GB 283GB volume
vol_2 402GB 205GB volume
vol_3 201GB 83GB volume
Aggregate Allocated Used Avail
Total space 1005GB 573GB 356GB
Snap reserve 71GB 22GB 49GB
WAFL reserve 159GB 10GB 148GB


Aggregate 'aggr0'
Total space WAFL reserve Snap reserve Usable space BSR NVLOG A-SIS
1593GB 159GB 71GB 1362GB 0KB 0KB
Space allocated to volumes in the aggregate
Volume Allocated Used Guarantee
vol0 11GB 725MB volume
vol_4 402GB 305GB volume
vol_5 402GB 288GB volume
vol_6 402GB 273GB volume
Aggregate Allocated Used Avail
Total space 1217GB 868GB 144GB
Snap reserve 71GB 24GB 47GB
WAFL reserve 159GB 17GB 142GB


2.2. Create a new volume in one of your aggregate

There’s a lot of way to configure you filer for iscsi, we

vol create vol1 -l fr aggr0 80g
Thu Feb 12 20:31:07 GMT [vol.language.changed:info]: Language on volume vol1 changed to fr
The new language mappings will be available after reboot
Thu Feb 12 20:31:07 GMT [vv_config_worker:notice]: XL - Language of Volume vol1 has been changed to fr.
Creation of volume 'vol1' with size 80g on containing aggregate
'aggr0' has completed.


2.3. [Option] Rename the new volume


fas250> vol rename vol1 gollum
'vol1' renamed to 'gollum'


2.4. Force the snapshot reservation to n%


fas250> snap reserve gollum 5


3. Create the luns


fas250> lun create -s 79g -t linux /vol/gollum/vol_gollum


4. Create the igroup

In order to make the LUNs visible to connected hosts, we must create an initiator group (igroup). In the iSCSI world, the igroup will contain the iSCSI node names of the initiators that will be connecting to the storage system.
On debian systems (refer to article debian iscsi) you will find the initiator name of your systems using this command :

gollum:~# cat /etc/iscsi/initiatorname.iscsi | grep -v '#'
InitiatorName=iqn.1993-08.org.debian:01:d6000000912


To create an iSCSI igroup, use the following commands:

fas250> igroup create -i -t linux gollum iqn.1993-08.org.debian:01:d6000000912


5. Map the igroup to the lun


fas250> lun map /vol/gollum/vol_gollum gollum
Thu Feb 12 22:31:22 GMT [lun.map:info]: LUN /vol/gollum/vol_gollum was mapped to initiator group gollum=0


6. Check the configuration


fas250> lun show -v /vol/gollum/vol_gollum
/vol/gollum/vol_gollum 79.0g (84828749824) (r/w, online, mapped)
Serial#: hp00000cs3
Share: none
Space Reservation: enabled
Multiprotocol Type: linux
Maps: gollum=0


Now the rest of the configuration will be done on the server

1. Open-iCSCI Installation :


apt-get install open-iscsi


Use the good folder for iscsi configuration :

ln -s /etc/{iscsid.conf,initiatorname.iscsi} /etc/iscsi/


2. Configure the iscsi client :

2.1 Enable automatic startup

vim /etc/iscsi/iscsid.conf
And Set the value :

node.startup = automatic

3. Configure the filer (Netapp FAS 250)

Refer to the following article

4. Configure the Target :

Discover the available target on the iscsi server

iscsiadm -m discovery -t sendtargets -p ISCSI-SERVER-IP-ADDRESS

You should get womething like this :

root@debian:~# iscsiadm --mode discovery --type sendtargets --portal 192.168.1.100
[fe80::a0:ffff:ffff:ffff]:3260,2000 iqn.1992-08.com.netapp:sn.84000037
192.168.1.100:3260,2000 iqn.1992-08.com.netapp:sn.84000037


Login to the target


root@debian:~# iscsiadm --mode node --targetname iqn.1992-08.com.netapp:sn.84200007 --login
Logging in to [iface: default, target: iqn.1992-08.com.netapp:sn.84200007, portal: 192.168.1.100,3260]
Login to [iface: default, target: iqn.1992-08.com.netapp:sn.84200007, portal: 192.168.1.100,3260]: successful


You can also login by restarting the initiator

root@debian:~# /etc/init.d/open-iscsi restart

Check the new partition and prepare it to be used

Check the new parition


gollum:~# fdisk -l
Disk /dev/sda: 80.0 GB, 80000000000 bytes
255 heads, 63 sectors/track, 9726 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00000080

Device Boot Start End Blocks Id System
/dev/sda1 * 1 31 248976 83 Linux
/dev/sda2 32 9726 77875087+ 8e Linux LVM

Disk /dev/sdb: 84.8 GB, 84828749824 bytes
255 heads, 63 sectors/track, 10313 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00000000

Disk /dev/sdb doesn't contain a valid partition table


Use fisk to create the new parition table

fdisk /dev/sdb
If you need some help, '?' is your friend

gollum:~# fdisk /dev/sdb
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0x4d5e25da.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won't be recoverable.
The number of cylinders for this disk is set to 10313.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)


Create a new primary partition numbered 1 begun at 1 finish at default end : 'n' -> 'p' -> '1' ->'enter'

Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-10313, default 1): 1
Last cylinder or +size or +sizeM or +sizeK (1-10313, default 10313):
Using default value 10313


Then write the new partition table using 'w'

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.


Use the new partition


gollum:~# fdisk -l

Disk /dev/sda: 80.0 GB, 80000000000 bytes
255 heads, 63 sectors/track, 9726 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00000080

Device Boot Start End Blocks Id System
/dev/sda1 * 1 31 248976 83 Linux
/dev/sda2 32 9726 77875087+ 8e Linux LVM

Disk /dev/sdb: 84.8 GB, 84828749824 bytes
255 heads, 63 sectors/track, 10313 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x4d5e25da

Device Boot Start End Blocks Id System
/dev/sdb1 1 10313 82839141 83 Linux


Now you can use you new parition and format it :

gollum:~# mkfs.ext3 /dev/sdb1
mke2fs 1.41.3 (12-Oct-2008)
warning: 409 blocks unused.

Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
5187456 inodes, 20709376 blocks
1035489 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=0
632 block groups
32768 blocks per group, 32768 fragments per group
8208 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000


Mount the new parition


mkdir /mnt/netapp
mount /dev/sdb1 /mnt/netapp


To mount automticaly you partition during startup

Edit your fstab

vi /etc/fstab


And adapt the sample configuration below to your system

/dev/sdb1 /mnt/netapp ext3 defaults,auto,_netdev 0 0