Recently in Systems Category

I found a new trick in iptables I wasn't aware of.  Turns out iptables tracks all connections, even ones that are not going to be NAT'd.  On a busy webserver or router this means iptables is wasting resources tracking connections when it should be just routing.

The raw table in iptables can be used to set a NOTRACK flag on a packet, and that packet will not traverse the ip_conntrack stuff.

So for a busy webserver, you could disable tracking for web connection states:
iptables -t raw -A PREROUTING -d <server IP> -p tcp --dport 80 -j NOTRACK
iptables -t raw -A PREROUTING -s <server IP> -p tcp --sport 80 -j NOTRACK

Or on a machine that is routing between several internal networks and also NATing external connections you could disable tracking for the internal to internal stuff.
iptables -t raw -A PREROUTING -s 172.16.0.0/16 -d 172.16.0.0/16 -j NOTRACK

On one of our core Linux routers the number of ip_conntrack entries dropped from an average of 65-70k down to about 2k. 

Upgrading Juniper EX series switches from USB.

| No Comments | No TrackBacks

This is for my notes. Generally I use this to upgrade the firmware of an EX4200 switch to the same JunOS version as the rest of a virtual chassis, BEFORE connecting the VC cables.  I'm doing this from memory, so if there are any mistakes let me know.

You will need to have access to a shell. The "root" user logs into shell automatically, other users should run "start shell"

From the shell run these commands:

 # mkdir /mnt/usb

Insert the USB stick (FAT32 formatted) into the port at the back of the switch. It should have the firmware on it, generally in a file named something like "jinstall-ex-10.3R3.4-domestic-signed.tgz"

 # mount -t msdosfs /dev/da1s1 /mnt/usb"

Note that tab completion works on Junos, to save some typing just enter the frist few characters of the filename and hit "tab"

 # cp /mnt/usb/jinstall-ex-10.3R3.4-domestic-signed.tgz /var/tmp 

If the switch is running firmware older than 10.3 you should upgrade the loader too:

 # cp /mnt/usb/jloader-ex-3242-11.3I20110326_0802_hmerge-signed.tgz /var/tmp
 # umount /mnt/usb
 # cli
At the ">" cli prompt: If the firmware on the switch is older than 10.3 you will need to install the updated jloader
 > request system software add /var/tmp/jloader-ex-3242-11.3I20110326_0802_hmerge-signed.tgz
then:
 > request system software add /var/tmp/jinstall-ex-10.3R3.4-domestic-signed.tgz reboot

Dual IP stack for twice the love

| No Comments | No TrackBacks
Several weeks too late for world IPV6 day: My first directly connected dual stack router (no tunnel). In this case a Juniper MX-80 connected to TW Telecom.
mfarver@RTR2> ping 2001:470:1f0e:d23::1
16 bytes from 2001:470:1f0e:d23::1, icmp_seq=0 hlim=62 time=46.032 ms
16 bytes from 2001:470:1f0e:d23::1, icmp_seq=1 hlim=62 time=45.363 ms
16 bytes from 2001:470:1f0e:d23::1, icmp_seq=2 hlim=62 time=44.284 ms

With rainbows, black is the most worrisome color

| No Comments | No TrackBacks
This has been at the back of my mind for years now.  Every company I've worked with stored MD5 hashed credit card numbers in the DB so "search by CC number" works.  As far as I know no one has released a rainbow table of all of the common credit card number prefixes hashed, but its getting way too easy to do so.  33.1 billions hashes per second.... yikes.

http://blog.zorinaq.com/?e=42

Back of the envelope, this system could calculate hashes for all of the Mastercard or Visa prefixes in 20 days or so.. faster if it concentrates on the most common BIN numbers.  Discover cards would take less than a week.

So reconsider if you really need a "search by CC number" function.  If you do switch to a more computationally expensive hash.   Consider adding a site specific salt so an attacker needs to generate a unique set of tables for your site.

Earth, Wind, Water and a Fire Alarm

| No Comments | No TrackBacks
Odd experience at our older colo facility:

We were onsite in the cage when the building fire alarms sounded. I did what most reasonable persons would: figured it was a false alarm but put down tools and walked out the front door.

The datacenter's technical staff had a little trouble.  They told us immediately that it was a test, and not to worry.  But usually tests involve technicians from the alarm company and there was no service trunks outside.  The techs continued to try to reach the facilities manager on the phone.  A gentleman turned up claiming to be the building owner, and said the fire department was on the way.  The fire department shows up, and the first question they ask is "What is this place?"  When we explained it was a datacenter they asked for further clarification.  In one of the most crowded datacenter corridors in the world, the fire department had no special training on dealing with datacenter fires, and didn't even know the buildings within their area.  The fire department leaves after a few minutes.  The alarm continues sounding for another 15 minutes before resetting on its own.

We later find out the issues was a false detect from an under floor smoke detector.  Seems that some additional training in reading the fire panel and when to evacuate the datacenter might be in order for the DC operator.  I was also a little surprised the fire department didn't inspect under the floor, or even walk around. 




Giving the Finger to Biometrics

| No Comments | No TrackBacks
I've been touring datacenters for my employer, trying to decide if any of the options in Austin are well suited to our needs.  Until recently Austin has been overshadowed by far greater connectivity in Dallas.  Austin has been popular for Disaster recovery purposes, having a very low incidence of disasters.  (Minimal tornadoes, inland enough to avoid hurricanes and seismically stable geology.)  Connectivity is improving, and the MetCenter project near the airport boasts 11 major providers in the campus.  DataFoundry is building their huge Texas One facility just south of the MetCenter, with 100MW of design capacity.  

Most datacenters today boast biometric security as a major feature.  Biometrics have quite a few issues, and have been defeated quite readily.  Still, when combined with another authentication method they can effective.  

Both datacenters I toured featured Biometric authentication combined with proximity cards and listed such on their marketing brochures.  I was disappointed to notice one of them only requiring biometric authentication for the customers.  Employees, whose credentials generally allow them wider access to the facility, only needed to be in possession of their proxcard to access the data floor.  The facilities manager argued that their security system, which displays the employees photo next to the relevant security camera image every time a door is opened, adequately compensated for the lack of two factor authentication.  I'm not sure if this is the case, since both the security guard and the NOC engineers did not seem to be paying much attention to the feeds.

The second facility, I was pleased to note, required two factor authentication of both employees and customers before entering the datacenter floor, and for access to other critical areas.

Extinguishing the ddWRT firewall

| No Comments | No TrackBacks
I was trying to use OpenVPN with ddWRT to make some cheap simple VPN adapters.  The user had H.323 Video conference stations at 3 locations, each behind a inexpensive home router.  The user wanted to be able to connect easily site to site.  Now H.323 and the NAT used on home routers doesn't work very well together, H.323 requires each site connect to the other on arbitrary UDP ports.  Very messy.  Even configuring the VC stations to use fixed ports and opening those ports through the routers didn't work well, nor did UPNP.  All methods the video stream itself subject to interception by an attacker.

On each network I installed a Buffalo WHR-HP-54 router I got at Fry's for about $70.  The latest version of ddWRT has a "VPN" edition that includes OpenVPN and some special configuration pages.  

The problem was that ddWRT wants to serve its intended use as a NATing router, and this conflicts with using OpenVPN in a point to point configuration.  The default configuration of the router was NATing some of the VPN traffic and prevented it from working.  Since these VPN devices were behind the firewall I only needed routing, not NAT.  The fix was to disable all of the standard iptables rules.  I couldn't find a way to turn this off via the control panel, but I did find a way to override it.  Go to the Administration->Commands page and type these lines into the text box:

iptables -t nat -F
iptables -F INPUT
iptables -F FORWARD
Then press "Save Firewall." This wipes all of the firewall rules on startup, if you need any of your own rules just add them after the commands above. This renders ddWRT into a simple router instead of a firewall and OpenVPN works perfectly. The OpenVPN setup turned out to be pretty easy, I'll explain in a later post.

The Condemned

| No Comments | No TrackBacks
When the earth was created, the powers above,
Gave each man a job to work at and love.
He made doctors and lawyers and plumbers and then,
He made carpenters, singers, and confidence men.
And when each had a job to work as he should,
He looked them all over and saw it was good.

He then sat down to rest for a day,
When a horrible groan chanced to come his way.
The Lord then looked down and his eyes opened wide,
For a motley collection of bums stood outside.
"And what do you want?" the creator asked them,
"Help us," they cried out, "A job for us men".
"We have no profession," they cried in dismay,
"And even the jails have turned us away".
Said the Lord, "I've seen many things without worth,
But here I find gathered the scum of the earth!"

The Lord was perplexed, and then he was mad,
For the jobs were all gone, there was none to be had.
Then he spoke aloud in a deep angry tone,
"Forever and ever ye mongrels shall roam,
Ye shall freeze in the summer and sweat when it's cold,
Ye shall work on equipment that's dirty and old,
Ye shall crawl under raised floors, and there cables lay,
Ye shall be called out at midnight and work through the day,
Ye shall work on all holidays, and not make your worth,
Ye shall be blamed for all downtime that occurs on the earth,
Ye shall watch all the glory go to software and sales,
Ye shall be blamed by them both if the system then fails.
Ye shall be paid nothing out of sorrow and tears,
Ye shall be forever cursed, and called SYSTEMS ENGINEERS!"

What, me worry?

| No Comments | No TrackBacks
This is a nice example of the advantages of thinking things through, especially when dealing with systems that have large negative consequences.

End of shift - one of the operators is leaving with his backpack slung over a shoulder. The other strap happens to catch the Halon release button by the door.

No problem. Pulling the button off did not cause a Halon release.

Putting the button back however...

I be, you be, Yubikey

| No Comments | No TrackBacks
Cool, and simple.  The Yubikey is a one-time password generator in a USB key.  Plug in in and it emulates a USB keyboard.  Touch the button on it and it will type out a one-time use password.  Basically the same system as "rolling code" garage door openers.

The hardware should be pretty cheap having no battery like the RSA SecurID tokens, just a microcontroller.   It claims to be somewhat resistant to phishing, though I cannot see how that works. 

The YubiKey
It works seamlessly with any hardware and operating system combination supporting USB keyboards such as Windows, MacOS, Linux and others. The Key generates and sends unique time-variant authentication codes by emulating keystrokes through the standard keyboard interface. The computer to which the Key is attached receives this authentication code character by character just as if it were being typed in from the keyboard - yet it's all performed automatically. This process allows the Key to be used with any application or Web-based service without any need for special client computer interaction or drivers.

The YubiKey differs from traditional authentication tokens based on time-variant codes in that it needs no battery and therefore does not rely on an absolute time generated by an accurate time source. No battery means unlimited shelf life, no synchronization and customer support issues, and enables significant cost reduction.

Link