From the Mind of Mark Farver

Juniper JunOS USB partition table rescan

2012-04-16T14:03:24Z

As an update to my upgrading Juniper EX switches from USB. Upgrading from a USB thumb drive works well, but if you don't have one handy you might try to use another "dual use" mass storage device like a phone or Kindle. Problem is the delay while that device initializes causes JunOS to fail to read the partition table. You will have a /dev/da1 entry, but no /dev/da1s1:

umass1: Amazon Amazon Kindle, rev 2.00/1.00, addr 3
da1 at umass-sim1 bus 1 target 0 lun 0
da1:  Removable Direct Access SCSI-2 device
da1: 40.000MB/s transfers
da1: Attempt to query device size failed: NOT READY, Medium not present
(da1:umass-sim1:1:0:0): READ CAPACITY. CDB: 25 0 0 0 0 0 0 0 0 0
(da1:umass-sim1:1:0:0): CAM Status: SCSI Status Error
(da1:umass-sim1:1:0:0): SCSI Status: Check Condition
(da1:umass-sim1:1:0:0): UNIT ATTENTION asc:28,0
(da1:umass-sim1:1:0:0): Not ready to ready change, medium may have changed
(da1:umass-sim1:1:0:0): Retrying Command (per Sense Data)

The workaround seems to be to issue a "camcontrol rescan da1" to rescan the parition table.

This issue only shows up on some versions of Junos, and the camcontrol command is also only available on some versions of Junos.

Disabling ip_conntrack to save memory, restore happiness

2012-01-04T23:00:43Z

I found a new trick in iptables I wasn't aware of. Turns out iptables tracks all connections, even ones that are not going to be NAT'd. On a busy webserver or router this means iptables is wasting resources tracking connections when it should be just routing.

The raw table in iptables can be used to set a NOTRACK flag on a packet, and that packet will not traverse the ip_conntrack stuff.

So for a busy webserver, you could disable tracking for web connection states:

iptables -t raw -A PREROUTING -d  -p tcp --dport 80 -j NOTRACK
iptables -t raw -A PREROUTING -s  -p tcp --sport 80 -j NOTRACK

Or on a machine that is routing between several internal networks and also NATing external connections you could disable tracking for the internal to internal stuff.

iptables -t raw -A PREROUTING -s 172.16.0.0/16 -d 172.16.0.0/16 -j NOTRACK

On one of our core Linux routers the number of ip_conntrack entries dropped from an average of 65-70k down to about 2k.

Upgrading Juniper EX series switches from USB.

2011-12-06T18:35:13Z

This is for my notes. Generally I use this to upgrade the firmware of an EX4200 switch to the same JunOS version as the rest of a virtual chassis, BEFORE connecting the VC cables. I'm doing this from memory, so if there are any mistakes let me know.

You will need to have access to a shell. The "root" user logs into shell automatically, other users should run "start shell"

From the shell run these commands:

 # mkdir /mnt/usb

Insert the USB stick (FAT32 formatted) into the port at the back of the switch. It should have the firmware on it, generally in a file named something like "jinstall-ex-10.3R3.4-domestic-signed.tgz"

 # mount -t msdosfs /dev/da1s1 /mnt/usb"

Note that tab completion works on Junos, to save some typing just enter the frist few characters of the filename and hit "tab"

 # cp /mnt/usb/jinstall-ex-10.3R3.4-domestic-signed.tgz /var/tmp

If the switch is running firmware older than 10.3 you should upgrade the loader too:

 # cp /mnt/usb/jloader-ex-3242-11.3I20110326_0802_hmerge-signed.tgz /var/tmp

 # umount /mnt/usb

 # cli

At the ">" cli prompt: If the firmware on the switch is older than 10.3 you will need to install the updated jloader

 > request system software add /var/tmp/jloader-ex-3242-11.3I20110326_0802_hmerge-signed.tgz

then:

 > request system software add /var/tmp/jinstall-ex-10.3R3.4-domestic-signed.tgz reboot

Dual IP stack for twice the love

2011-07-24T15:59:12Z

Several weeks too late for world IPV6 day: My first directly connected dual stack router (no tunnel). In this case a Juniper MX-80 connected to TW Telecom.

mfarver@RTR2> ping 2001:470:1f0e:d23::1
16 bytes from 2001:470:1f0e:d23::1, icmp_seq=0 hlim=62 time=46.032 ms
16 bytes from 2001:470:1f0e:d23::1, icmp_seq=1 hlim=62 time=45.363 ms
16 bytes from 2001:470:1f0e:d23::1, icmp_seq=2 hlim=62 time=44.284 ms

LLDP: Your guide to the land of lost servers

2011-07-22T01:29:19Z

We often use LLDP (Link Layer Discovery Protocol) to find out which switches are uplinked to other switches. Each switch sends identification packets to the opposing switch identifying its name and the power number it is broadcasting from.

Juniper EX Example

user@core1> show lldp neighbors    
Local Interface    Parent Interface    Chassis Id          Port info     System Name
ge-2/0/7.0         -                   00:26:f2:50:85:c0   47           data_sw1 
ge-2/0/14.0        ae15.0              00:26:f5:b1:3d:40   1            data_sw2 
ge-2/0/2.0         ae9.0               b4:39:d7:89:48:40   1            pub_sw1
ge-1/0/12.0        ae31.0              b4:39:d3:1a:2c:c0   1            pub_sw2 
ge-2/0/12.0        ae31.0              b4:39:d2:1a:2c:c0   2            pub_sw2

Makes it a little simpler to audit the wire maps. I wondered if it would be possible to have servers broadcast as well.

Turns out they can. RHEL6 ships with lldpd already ready to go. Just install the package and start the daemon. We are running CentOS 5 still, and I wasn’t able to find a recent package for that. Eventually I found a src RPM for lldpd 0.4.2 and just updated it. You can find the updated version here.

HP Procurve Example

SW2# show lldp info remote

 LLDP Remote Devices Information
  LocalPort | ChassisId                 PortId PortDescr SysName               
  --------- + ------------------------- ------ --------- ----------------------
  2         | 50 c5 8f b7 33 c0         36     vme.0     pub-sw1              
  19        | 78 2b cf 16 14 f0         78 ... eth2      cache-1.example.com
  20        | 78 2b cf 15 a0 2f         78 ... eth2      web-3.example.com
  21        | 78 2b cf 16 7c 24         78 ... eth2      web-2.example.com
  22        | 78 2b cf 15 bf 9e         78 ... eth2      web-1.example.com

You can query LLDP info manually as noted above, or you can get it via SNMP MIB: lldpRemoteSystemsData 1.0.8802.1.1.2.1.4. Interestingly it looks like you might be able to get remote IP information this way. The lldpd daemon also supports CDP for Cisco environments and a few other features that we don’t use. Windows hosts can broadcast the same data using the haneWIN LLDP Agent. Adding LLDP support is a simple add to your Puppet or Chef recipes, and makes a handy backup to manual wiremaps that your network team will appreciate.

With rainbows, black is the most worrisome color

2011-06-27T17:36:05Z

This has been at the back of my mind for years now. Every company I've worked with stored MD5 hashed credit card numbers in the DB so "search by CC number" works. As far as I know no one has released a rainbow table of all of the common credit card number prefixes hashed, but its getting way too easy to do so. 33.1 billions hashes per second.... yikes.

http://blog.zorinaq.com/?e=42

Back of the envelope, this system could calculate hashes for all of the Mastercard or Visa prefixes in 20 days or so.. faster if it concentrates on the most common BIN numbers. Discover cards would take less than a week.

So reconsider if you really need a "search by CC number" function. If you do switch to a more computationally expensive hash. Consider adding a site specific salt so an attacker needs to generate a unique set of tables for your site.

iftop: its tops...

2011-05-29T00:10:37Z

New tool I just found today and am kinda disappointed I never noticed before: iftop.. it works exactly how you would expect.

We're watching you, little PDU

2011-05-14T18:22:26Z

SNMP OID values for APC8941 PDUs we found useful when setting up Zenoss and cacti monitoring.

Total Volt Amps:

.1.3.6.1.4.1.318.1.1.12.1.18.0

Temperature (with optional AP9335T Temperature Sensor)

.1.3.6.1.4.1.318.1.1.26.10.2.2.1.8 in C

.1.3.6.1.4.1.318.1.1.26.10.2.2.1.7 in F

Relative Humidity (with optional AP9335TH Temp and Humidity Sensor):

.1.3.6.1.4.1.318.1.1.26.10.2.2.1.10

There are a bunch more things you can check, Temp status, breaker and power outlet status as well. Make sure you get the latest PowerNet MIB form APC.

Earth, Wind, Water and a Fire Alarm

2011-04-29T05:58:10Z

Odd experience at our older colo facility:

We were onsite in the cage when the building fire alarms sounded. I did what most reasonable persons would: figured it was a false alarm but put down tools and walked out the front door.

The datacenter's technical staff had a little trouble. They told us immediately that it was a test, and not to worry. But usually tests involve technicians from the alarm company and there was no service trunks outside. The techs continued to try to reach the facilities manager on the phone. A gentleman turned up claiming to be the building owner, and said the fire department was on the way. The fire department shows up, and the first question they ask is "What is this place?" When we explained it was a datacenter they asked for further clarification. In one of the most crowded datacenter corridors in the world, the fire department had no special training on dealing with datacenter fires, and didn't even know the buildings within their area. The fire department leaves after a few minutes. The alarm continues sounding for another 15 minutes before resetting on its own.

We later find out the issues was a false detect from an under floor smoke detector. Seems that some additional training in reading the fire panel and when to evacuate the datacenter might be in order for the DC operator. I was also a little surprised the fire department didn't inspect under the floor, or even walk around.

Giving the Finger to Biometrics

2011-01-30T00:06:59Z

I've been touring datacenters for my employer, trying to decide if any of the options in Austin are well suited to our needs. Until recently Austin has been overshadowed by far greater connectivity in Dallas. Austin has been popular for Disaster recovery purposes, having a very low incidence of disasters. (Minimal tornadoes, inland enough to avoid hurricanes and seismically stable geology.) Connectivity is improving, and the MetCenter project near the airport boasts 11 major providers in the campus. DataFoundry is building their huge Texas One facility just south of the MetCenter, with 100MW of design capacity.

Most datacenters today boast biometric security as a major feature. Biometrics have quite a few issues, and have been defeated quite readily. Still, when combined with another authentication method they can effective.

Both datacenters I toured featured Biometric authentication combined with proximity cards and listed such on their marketing brochures. I was disappointed to notice one of them only requiring biometric authentication for the customers. Employees, whose credentials generally allow them wider access to the facility, only needed to be in possession of their proxcard to access the data floor. The facilities manager argued that their security system, which displays the employees photo next to the relevant security camera image every time a door is opened, adequately compensated for the lack of two factor authentication. I'm not sure if this is the case, since both the security guard and the NOC engineers did not seem to be paying much attention to the feeds.

The second facility, I was pleased to note, required two factor authentication of both employees and customers before entering the datacenter floor, and for access to other critical areas.

Changing your mind every 3000 miles

2010-12-20T01:58:02Z

I was trying to explain the Monty Hall problem to Alyssa the other night and failing miserably.

Imagine that the set of Monty Hall's game show Let's Make a Deal has three closed doors. Behind one of these doors is a car; behind the other two are goats. The contestant does not know where the car is, but Monty Hall does.
The contestant picks a door and Monty opens one of the remaining doors, one he knows doesn't hide the car. If the contestant has already chosen the correct door, Monty is equally likely to open either of the two remaining doors.
After Monty has shown a goat behind the door that he opens, the contestant is always given the option to switch doors. What is the probability of winning the car if she stays with her first choice? What if she decides to switch?

The correct answer is the contestant should always change their mind, by doing so they increase the odds of winning from 1 in 3 to 2 in 3. This is pretty counterintuitive, and even experts get it wrong. But it makes more sense if you draw it out:

Keep your choice:

C G G Initial Config
X     Your choice
  H   Host's Choice (he could choose either goat w/same results)
X     Keep your choice (WIN)

C G G Initial Config
  X   Your choice
    H Host's Choice
  X   Keep your choice (LOSE)

C G G Initial Config
    X Your choice
  H   Hosts Choice
    X Keep your choice (LOSE)

Change your choice:
C G G Initial Config
X     Your choice
  H   Hosts Choice (he could choose either goat w/same results)
    X Change your choice (LOSE)

C G G Initial Config
  X    Your choice
    H  Hosts Choice
X     Change your choice (WIN)

C G G Initial Config
    X Your choice
  H   Hosts Choice
X     Change your choice (WIN)

More information on the Dr Math FAQ

Extinguishing the ddWRT firewall

2010-11-19T00:01:58Z

I was trying to use OpenVPN with ddWRT to make some cheap simple VPN adapters. The user had H.323 Video conference stations at 3 locations, each behind a inexpensive home router. The user wanted to be able to connect easily site to site. Now H.323 and the NAT used on home routers doesn't work very well together, H.323 requires each site connect to the other on arbitrary UDP ports. Very messy. Even configuring the VC stations to use fixed ports and opening those ports through the routers didn't work well, nor did UPNP. All methods the video stream itself subject to interception by an attacker.

On each network I installed a Buffalo WHR-HP-54 router I got at Fry's for about $70. The latest version of ddWRT has a "VPN" edition that includes OpenVPN and some special configuration pages.

The problem was that ddWRT wants to serve its intended use as a NATing router, and this conflicts with using OpenVPN in a point to point configuration. The default configuration of the router was NATing some of the VPN traffic and prevented it from working. Since these VPN devices were behind the firewall I only needed routing, not NAT. The fix was to disable all of the standard iptables rules. I couldn't find a way to turn this off via the control panel, but I did find a way to override it. Go to the Administration->Commands page and type these lines into the text box:

iptables -t nat -F
iptables -F INPUT
iptables -F FORWARD

Then press "Save Firewall." This wipes all of the firewall rules on startup, if you need any of your own rules just add them after the commands above. This renders ddWRT into a simple router instead of a firewall and OpenVPN works perfectly. The OpenVPN setup turned out to be pretty easy, I'll explain in a later post.

DIY crack: "Where's my patching cement?"

2010-10-27T08:52:20Z

As these things go, my guilty pleasure is a pretty benign one. About the only TV I can sit down to anymore is home improvement shows. And not Tim the Toolman but old episodes of "This Old House," "Hometime," "Holmes on Homes," whatever. My local PBS and a few secondary off air channels provide a trickle of mostly repeats on the weekends. When I had cable Discovery provided a few more, and DIY was a premium channel with in house shows that were new as good as the Public Broadcasting originals.

So here's my question: Netflix has licensed hundreds of old TV shows to fill out their "Watch Instantly" system. Viewers are getting a sweet taste of what we've known we've wanted for decades. A la carte, on demand programming with an immense back catalog. Despite thousands of TV shows available in its system, there isn't a single home improvement, DIY or similar show in Netflix's catalog. A few episodes of PBS's "Frontline" and some Mythbusters are about the closest.

In fairness, shows like Hometime aren't produced by the network. They get some initial funding from the network for first broadcast rights which usually just covers the cost of production. Profits come from merchandising, and video sales. It has been said that tremendous sales of Hometime's Log Cabin series were largely responsible for the show's executive producer Dean Johnson being able to keep the Log Cabin itself after construction as well as build at least two other luxury homes for himself (also on camera, natch.)

I don't know how many people would agree, but that is my suggestion for Netflix.. find a way to buy rights to these shows cheap and I can live vicariously, without ripping down more walls in my house. My wife will thank you.

The Condemned

2010-10-09T23:52:17Z

When the earth was created, the powers above,
Gave each man a job to work at and love.
He made doctors and lawyers and plumbers and then,
He made carpenters, singers, and confidence men.
And when each had a job to work as he should,
He looked them all over and saw it was good.

He then sat down to rest for a day,
When a horrible groan chanced to come his way.
The Lord then looked down and his eyes opened wide,
For a motley collection of bums stood outside.
"And what do you want?" the creator asked them,
"Help us," they cried out, "A job for us men".
"We have no profession," they cried in dismay,
"And even the jails have turned us away".
Said the Lord, "I've seen many things without worth,
But here I find gathered the scum of the earth!"

The Lord was perplexed, and then he was mad,
For the jobs were all gone, there was none to be had.
Then he spoke aloud in a deep angry tone,
"Forever and ever ye mongrels shall roam,
Ye shall freeze in the summer and sweat when it's cold,
Ye shall work on equipment that's dirty and old,
Ye shall crawl under raised floors, and there cables lay,
Ye shall be called out at midnight and work through the day,
Ye shall work on all holidays, and not make your worth,
Ye shall be blamed for all downtime that occurs on the earth,
Ye shall watch all the glory go to software and sales,
Ye shall be blamed by them both if the system then fails.
Ye shall be paid nothing out of sorrow and tears,
Ye shall be forever cursed, and called SYSTEMS ENGINEERS!"

Never answer a question you wouldn't ask.

2010-09-26T01:04:07Z

Engineering humor:

A man is walking through a field, when he spots a hot air balloon low above the trees. The balloon nears, and the man in the balloon calls out, "Hello! Can you tell me where I am?"

The man in the field pauses for a second, then says, "You are in a hot air balloon, about thirty feet off of the ground, above the field in which I am walking."

The man in the balloon replies, "You must be an engineer."

The man in the field, who is an engineer, responds, "Why yes, I am. How did you know?"

The man in the balloon says, "Well, everything you told me is technically correct, but is not useful to me."

The engineer asks, "Would you, by any chance, be an executive?"

The man in the balloon, who is indeed an executive, replies, "Yes. What gave it away?"

The engineer responds, "Well, you obviously have no idea of where you are or where you are going, you are in just as bad of a position now as you were before you talked to me, but now it's my fault."