Disabling ip_conntrack to save memory, restore happiness

By Mark Farver on January 4, 2012 5:00 PM | No Comments | No TrackBacks
I found a new trick in iptables I wasn't aware of.  Turns out iptables tracks all connections, even ones that are not going to be NAT'd.  On a busy webserver or router this means iptables is wasting resources tracking connections when it should be just routing.

The raw table in iptables can be used to set a NOTRACK flag on a packet, and that packet will not traverse the ip_conntrack stuff.

So for a busy webserver, you could disable tracking for web connection states:
iptables -t raw -A PREROUTING -d <server IP> -p tcp --dport 80 -j NOTRACK
iptables -t raw -A PREROUTING -s <server IP> -p tcp --sport 80 -j NOTRACK

Or on a machine that is routing between several internal networks and also NATing external connections you could disable tracking for the internal to internal stuff.
iptables -t raw -A PREROUTING -s 172.16.0.0/16 -d 172.16.0.0/16 -j NOTRACK

On one of our core Linux routers the number of ip_conntrack entries dropped from an average of 65-70k down to about 2k. 

Upgrading Juniper EX series switches from USB.

By Mark Farver on December 6, 2011 12:35 PM | No Comments | No TrackBacks

This is for my notes. Generally I use this to upgrade the firmware of an EX4200 switch to the same JunOS version as the rest of a virtual chassis, BEFORE connecting the VC cables.  I'm doing this from memory, so if there are any mistakes let me know.

You will need to have access to a shell. The "root" user logs into shell automatically, other users should run "start shell"

From the shell run these commands:

 # mkdir /mnt/usb

Insert the USB stick (FAT32 formatted) into the port at the back of the switch. It should have the firmware on it, generally in a file named something like "jinstall-ex-10.3R3.4-domestic-signed.tgz"

 # mount -t msdosfs /dev/da1s1 /mnt/usb"

Note that tab completion works on Junos, to save some typing just enter the frist few characters of the filename and hit "tab"

 # cp /mnt/usb/jinstall-ex-10.3R3.4-domestic-signed.tgz /var/tmp

If the switch is running firmware older than 10.3 you should upgrade the loader too:

 # cp /mnt/usb/jloader-ex-3242-11.3I20110326_0802_hmerge-signed.tgz /var/tmp
 # umount /mnt/usb
 # cli
At the ">" cli prompt: If the firmware on the switch is older than 10.3 you will need to install the updated jloader 
 > request system software add /var/tmp/jloader-ex-3242-11.3I20110326_0802_hmerge-signed.tgz
then: 
 > request system software add /var/tmp/jinstall-ex-10.3R3.4-domestic-signed.tgz reboot

Dual IP stack for twice the love

By Mark Farver on July 24, 2011 10:59 AM | No Comments | No TrackBacks
Several weeks too late for world IPV6 day: My first directly connected dual stack router (no tunnel). In this case a Juniper MX-80 connected to TW Telecom.
mfarver@RTR2> ping 2001:470:1f0e:d23::1
16 bytes from 2001:470:1f0e:d23::1, icmp_seq=0 hlim=62 time=46.032 ms
16 bytes from 2001:470:1f0e:d23::1, icmp_seq=1 hlim=62 time=45.363 ms
16 bytes from 2001:470:1f0e:d23::1, icmp_seq=2 hlim=62 time=44.284 ms

LLDP: Your guide to the land of lost servers

By Mark Farver on July 21, 2011 8:29 PM | No Comments | No TrackBacks

We often use LLDP (Link Layer Discovery Protocol) to find out which switches are uplinked to other switches. Each switch sends identification packets to the opposing switch identifying its name and the power number it is broadcasting from.

Juniper EX Example

user@core1> show lldp neighbors    
Local Interface    Parent Interface    Chassis Id          Port info     System Name
ge-2/0/7.0         -                   00:26:f2:50:85:c0   47           data_sw1 
ge-2/0/14.0        ae15.0              00:26:f5:b1:3d:40   1            data_sw2 
ge-2/0/2.0         ae9.0               b4:39:d7:89:48:40   1            pub_sw1
ge-1/0/12.0        ae31.0              b4:39:d3:1a:2c:c0   1            pub_sw2 
ge-2/0/12.0        ae31.0              b4:39:d2:1a:2c:c0   2            pub_sw2

Makes it a little simpler to audit the wire maps. I wondered if it would be possible to have servers broadcast as well.

Turns out they can. RHEL6 ships with lldpd already ready to go. Just install the package and start the daemon. We are running CentOS 5 still, and I wasn’t able to find a recent package for that. Eventually I found a src RPM for lldpd 0.4.2 and just updated it. You can find the updated version here.

HP Procurve Example

SW2# show lldp info remote

 LLDP Remote Devices Information
  LocalPort | ChassisId                 PortId PortDescr SysName               
  --------- + ------------------------- ------ --------- ----------------------
  2         | 50 c5 8f b7 33 c0         36     vme.0     pub-sw1              
  19        | 78 2b cf 16 14 f0         78 ... eth2      cache-1.example.com
  20        | 78 2b cf 15 a0 2f         78 ... eth2      web-3.example.com
  21        | 78 2b cf 16 7c 24         78 ... eth2      web-2.example.com
  22        | 78 2b cf 15 bf 9e         78 ... eth2      web-1.example.com

You can query LLDP info manually as noted above, or you can get it via SNMP MIB: lldpRemoteSystemsData 1.0.8802.1.1.2.1.4. Interestingly it looks like you might be able to get remote IP information this way. The lldpd daemon also supports CDP for Cisco environments and a few other features that we don’t use. Windows hosts can broadcast the same data using the haneWIN LLDP Agent. Adding LLDP support is a simple add to your Puppet or Chef recipes, and makes a handy backup to manual wiremaps that your network team will appreciate.

With rainbows, black is the most worrisome color

By Mark Farver on June 27, 2011 12:36 PM | No Comments | No TrackBacks
This has been at the back of my mind for years now.  Every company I've worked with stored MD5 hashed credit card numbers in the DB so "search by CC number" works.  As far as I know no one has released a rainbow table of all of the common credit card number prefixes hashed, but its getting way too easy to do so.  33.1 billions hashes per second.... yikes.

http://blog.zorinaq.com/?e=42

Back of the envelope, this system could calculate hashes for all of the Mastercard or Visa prefixes in 20 days or so.. faster if it concentrates on the most common BIN numbers.  Discover cards would take less than a week.

So reconsider if you really need a "search by CC number" function.  If you do switch to a more computationally expensive hash.   Consider adding a site specific salt so an attacker needs to generate a unique set of tables for your site.

iftop: its tops...

By Mark Farver on May 28, 2011 7:10 PM | No Comments | No TrackBacks
New tool I just found today and am kinda disappointed I never noticed before: iftop.. it works exactly how you would expect.

We're watching you, little PDU

By Mark Farver on May 14, 2011 1:22 PM | No Comments | No TrackBacks

SNMP OID values for APC8941 PDUs we found useful when setting up Zenoss and cacti monitoring.

Total Volt Amps:
.1.3.6.1.4.1.318.1.1.12.1.18.0

Temperature (with optional AP9335T Temperature Sensor)
.1.3.6.1.4.1.318.1.1.26.10.2.2.1.8 in C

.1.3.6.1.4.1.318.1.1.26.10.2.2.1.7 in F

Relative Humidity (with optional AP9335TH Temp and Humidity Sensor):
.1.3.6.1.4.1.318.1.1.26.10.2.2.1.10

There are a bunch more things you can check, Temp status, breaker and power outlet status as well.  Make sure you get the latest PowerNet MIB form APC.
Continue reading We're watching you, little PDU.

Earth, Wind, Water and a Fire Alarm

By Mark Farver on April 29, 2011 12:58 AM | No Comments | No TrackBacks
Odd experience at our older colo facility:

We were onsite in the cage when the building fire alarms sounded. I did what most reasonable persons would: figured it was a false alarm but put down tools and walked out the front door.

The datacenter's technical staff had a little trouble.  They told us immediately that it was a test, and not to worry.  But usually tests involve technicians from the alarm company and there was no service trunks outside.  The techs continued to try to reach the facilities manager on the phone.  A gentleman turned up claiming to be the building owner, and said the fire department was on the way.  The fire department shows up, and the first question they ask is "What is this place?"  When we explained it was a datacenter they asked for further clarification.  In one of the most crowded datacenter corridors in the world, the fire department had no special training on dealing with datacenter fires, and didn't even know the buildings within their area.  The fire department leaves after a few minutes.  The alarm continues sounding for another 15 minutes before resetting on its own.

We later find out the issues was a false detect from an under floor smoke detector.  Seems that some additional training in reading the fire panel and when to evacuate the datacenter might be in order for the DC operator.  I was also a little surprised the fire department didn't inspect under the floor, or even walk around. 




Giving the Finger to Biometrics

By Mark Farver on January 29, 2011 6:06 PM | No Comments | No TrackBacks
I've been touring datacenters for my employer, trying to decide if any of the options in Austin are well suited to our needs.  Until recently Austin has been overshadowed by far greater connectivity in Dallas.  Austin has been popular for Disaster recovery purposes, having a very low incidence of disasters.  (Minimal tornadoes, inland enough to avoid hurricanes and seismically stable geology.)  Connectivity is improving, and the MetCenter project near the airport boasts 11 major providers in the campus.  DataFoundry is building their huge Texas One facility just south of the MetCenter, with 100MW of design capacity.  

Most datacenters today boast biometric security as a major feature.  Biometrics have quite a few issues, and have been defeated quite readily.  Still, when combined with another authentication method they can effective.  

Both datacenters I toured featured Biometric authentication combined with proximity cards and listed such on their marketing brochures.  I was disappointed to notice one of them only requiring biometric authentication for the customers.  Employees, whose credentials generally allow them wider access to the facility, only needed to be in possession of their proxcard to access the data floor.  The facilities manager argued that their security system, which displays the employees photo next to the relevant security camera image every time a door is opened, adequately compensated for the lack of two factor authentication.  I'm not sure if this is the case, since both the security guard and the NOC engineers did not seem to be paying much attention to the feeds.

The second facility, I was pleased to note, required two factor authentication of both employees and customers before entering the datacenter floor, and for access to other critical areas.

Changing your mind every 3000 miles

By Mark Farver on December 19, 2010 7:58 PM | No Comments | No TrackBacks
I was trying to explain the Monty Hall problem to Alyssa the other night and failing miserably.

Imagine that the set of Monty Hall's game show Let's Make a Deal has three closed doors. Behind one of these doors is a car; behind the other two are goats. The contestant does not know where the car is, but Monty Hall does.

The contestant picks a door and Monty opens one of the remaining doors, one he knows doesn't hide the car. If the contestant has already chosen the correct door, Monty is equally likely to open either of the two remaining doors.

After Monty has shown a goat behind the door that he opens, the contestant is always given the option to switch doors. What is the probability of winning the car if she stays with her first choice? What if she decides to switch?

The correct answer is the contestant should always change their mind, by doing so they increase the odds of winning from 1 in 3 to 2 in 3.  This is pretty counterintuitive, and even experts get it wrong.  But it makes more sense if you draw it out:



Keep your choice:

C G G Initial Config
X     Your choice
  H   Host's Choice (he could choose either goat w/same results)
X     Keep your choice (WIN)

C G G Initial Config
  X   Your choice
    H Host's Choice
  X   Keep your choice (LOSE)

C G G Initial Config
    X Your choice
  H   Hosts Choice
    X Keep your choice (LOSE)

Change your choice:
C G G Initial Config
X     Your choice
  H   Hosts Choice (he could choose either goat w/same results)
    X Change your choice (LOSE)

C G G Initial Config
  X    Your choice
    H  Hosts Choice
X     Change your choice (WIN)

C G G Initial Config
    X Your choice
  H   Hosts Choice
X     Change your choice (WIN)
More information on the Dr Math FAQ
Archives