NetBSD and the Intel ATM Driver
On the PCs, the kernel sources should be available in
/big/netbsd-sys/1.1/ijet-sys-stable. The IJET driver is found within
this tree in /big/netbsd-sys/1.1/ijet-sys-stable/dev/pci/jetland.
At some point it may move into the i386 specific sub-tree. Kernel config
files are in /big/netbsd-sys/1.1/ijet-sys-stable/arch/i386/conf and the
compilation directories are /big/netbsd-sys/1.1/ijet-sys-stable/arch/i386/
Loading a Kernel
Assume you want to run the kernel in /big/netbsd-sys/1.1/ijet-sys-stable/
arch/i386/compile/IJET.N48.CREDIT/netbsd, the steps are:
- % cd /big/netbsd-sys/1.1/ijet-sys-stable/arch/i386/compile/IJET.N48.CREDIT
- % cp netbsd /netbsd.new
-- copy the kernel into root directory and rename it
- % cd /
- % rm netbsd
-- delete the old kernel binary
- % ln netbsd.new netbsd
- % reboot
-- link renamed binary as kernel and reboot machine
Building a Kernel
We are running NetBSD -current. To build a new kernel, the steps are
- % cd /big/netbsd-sys/1.1/ijet-sys-stable/arch/i386/conf
- -- edit/create a kernel config file -- For example, we'll use
the IJET config file.
- % /usr/sbin/config.new IJET
-- at some point config.new will be moved to config, and config will
become config.old. So if config.new disappears, you'll know what
- % cd ../compile/IJET
- % make clean && make depend && make
- % cp netbsd /netbsd.IJET
- % cp /netbsd /netbsd.OLD
-- always a good idea to keep a working kernel around
- % rm /netbsd
- % ln /netbsd.IJET /netbsd
-- we make /netbsd a hard link to the actual kernel. This makes it
easier to keep track of which kernel we're really using.
- % reboot
-- hopefully the new kernel boots successfully. If it fails to even
get to the command level, see the section
Booting an alternate kernel.
The directory /big/netbsd-sys/1.1/ijet-sys-stable/dev/pci/jetland/france is the current
location of the driver utilities. This directory is located in the
kernel source tree because it is highly dependent on some of the
driver include files which currently change on a frequent basis.
Assuming you're running a standard IJET kernel and you want the
default setup, the only command you should have to issue is
`hostname`.up; where hostname is the simple hostname of
the machine you are on.
The following commands are executed by this file:
Setarp.c contains the various mappings from IP addresses to
VCid. This should probably be read in via some file.
Currently, our mapping of IP addresses to machines is done by
using the subnet 192.70.210 and appending the final octet of the host's
ethernet IP address. For example, hephaestus has the normal IP address
of 22.214.171.124, so its ATM IP address is 126.96.36.199. Obviously,
should we get addresses that conflict, we'll have to move to a different
- % ./debugmask 0
- debugmask takes as an optional single argument a string
consisting of the various debuglevels you wish to make active.
The debuglevels are [0-5][a-z]. The argument -all may
also be used. A null argument turns off all debugging. See the
debugging section for more details.
- % ./loaducode
- simply loads the microcode contained in ucode.dat into the adaptor.
This is a mandatory step.
- % ./newsetup
- Registers some configuration variables with the driver. The number
of RxTx VCs, message buffers, etc can be set. This is also a
mandatory step prior to ifconfig'ing the device
up. This will probably be renamed back to setup in the
- % /sbin/route delete -net 192.80.210
- Solely as a precaution that the net has not been previously
registered. In the past, the switch has been broadcasting itself
as a gateway for this network; which prevented us from registering
any other devices on that net.
- % /sbin/ifconfig ijet0 some_address -arp
- Finally tell the driver to register the device as up. Currently,
we can not configure the device down. This will hopefully be resolved
We currently have two different switches. A Fore switch and a
BNR switch. I'll describe how to set up each of these.
The BNR switch is known as Kato. Telnet'ing to kato will
put you at a command prompt. The following commands and command
scripts are typical:
- cd "/afs/cs/project/vcnectar/harvard"
- Most of the command scripts are located in this directory.
- < loadup.scr
- This script takes some time to run as it performs diagnostics on
each of the port cards it finds.
- vc2 3, 45, 0, 4, 68, 0
- Creates a bi-directional VC from port 3, VC 45, no flow control to
port 4, VC 68, no flow control.
- vc1 3, 45, 0 3, 45, 0
- This would just create a loopback VC on port3.
- Shows the status of all the created paths.
To turn off credit cells on a port use "nofc portNo" at the prompt and turn
credit on use "yesfc portNo". Note, our driver does not currently
issue credit, so we should set the switch up to send out credit cells
but not to expect them.
The correct way to make a
loopback VC with flow control only on the link to the switch, one would
use 'vc1 3, 45, 1, 3, 45, 0'. Similarly you should use the 'vc1' command
twice if you want a bidirectional VC between two machines with flow control
only on the input link to the switch.
The FORE switch is known as Boopsie. Telnet'ing to boopsie will
bring up the login prompt. The login name for administration and
configuration is asx and the password is coreyisgod.
Login will put you at a command line interface called AMI (ASX - Management
Interface). The following points are general to the command line interface:
- Typing '?' at any prompt will display a list of commands at that level.
- Typing 'help command' will display help on that command.
- Type 'up' to go up the menu one level.
- Commands with a '>' at the end indicate a sub-menu. Typing the command
name will take you down to that menu.
Setting up PVCs
To setup pvcs you have to be at the 'configuration:vcc>' menu. The command
syntax to setup pvcs is:
new iport ivpi ivci oport ovpi ovci
For example to set up a pvc between ports a1 and a2 with vpi, vci = 0, 50, you
would say 'new a1 0 50 a2 0 50'. This only creates a uni-directional pvc. You
have to explicitly create another pvc in the other direction using the command
'new a2 0 50 a1 0 50'.
There are other optional parameters that you can give to the command used to
create pvcs. They are used to specify peak bandwidth, cdvt, and tag/drop
policy. Typing 'new' at the configuration:vcc> menu without any options brings
up the list of options.
To delete a pvc use the command:
delete iport ivpi ivci oport ovpi ovci
Turning signalling ON/OFF
The following commands can be used to turn signalling on/off:
- Typing delete port vpi at the configuration:spans> menu would
turn SPANS off on that port. Vpi is typically 0.
- Typing new port vpi at the configuration:spans> menu would
bring up SPANS on that port. Vpi is again typically 0. There are other
optional parameters like AAL4/5, cdvt and tag/drop.
- Typing delete port vpi at the configuration:uni30> menu would
turn UNI signalling off on that port. Vpi is typically 0.
- Typing new port vpi at the configuration:uni30> menu would
bring up UNI on that port. Vpi is again typically 0. There are other
optional parameters like ILMI, AAL4/5 etc.
The BNR switch supports monitoring the performance of the VCs. There are several things that one could monitor including bandwidth, queue size, credit received from the downstream node, etc. These performance measures are displayed on a real-time X display on the monitoring workstation. The architecture of the monitoring software provided by Harvard is like this: A real-time X display program binds to a well-known UDP socket and recieves measured samples from the switch and displays it on the workstation. The switch has to be configured with the IP address of the workstation and also the VC parameters that have to be monitored.The following explains how to setup the workstation and the switch for monitoring.
Monitoring workstation setup
You need to do the following to startup the display program on the monitoring workstation
- % cd /afs/cs/project/vcnectar/harvard/graph
- % ./graph_report&
You need to do the following to startup the monitoring software on the switch
- % telnet kato
- grSrv "IP address of the workstation"
-- e.g., grSrv "188.8.131.52"
- grBw port, ivc, slowdown, avg (To graph bandwidth for example)
-- e.g., grBw 12, 1, 7, 10
The following monitoring commands are available to graph various parameters.
Note that the "ivc" is the internal VC number of the switch. This can be
obtained by typing "lsvcs" at the command prompt. Also note that the meaning
of the parameter "port" is different for different graphing commands --- for
some it is the input port and for some it is the output port of the same ivc.
The "slowdown" is a number in the range 0 ... 31, which specifies the number
of samples to skip over before taking a measurement and "avg" is the number of
measurements to average before transmitting the result. Values of 7 for slow
down and 10 for avg work best for us.
- grBw port, ivc, slowdown, avg (Bandwidth, port is the input port)
- grCcnt port, ivc, slowdown, avg (Output cell counter, port is the
- grBcnt port, ivc, slowdown, avg (Buffered cell count port is the
- grCMBcnt port, ivc, slowdown, avg (Buffered count on the input side
of the credit manager, should differ from grBcnt by atmost one. Port is the
- grDrops port, ivc, slowdown, avg (Plots 1 for dropped cell 0
otherwise. Port is the input port)
- grCredit port, ivc, slowdown, avg (Credits received from downstream
node, port is the output port)
- grPortBw port (Total output bandwidth of a port. Port is the output
Currently there are two basic mechanisms that are employed to
facilitate the debugging process. The first is the standard
kernel printf() function. When printf() is called
within the kernel it prints the message to the console, and
additionally writes it to /var/log/messages. However, the
internal console buffer is only 4k, so if the message isn't
written to the file soon enough, information is overwritten.
The second method of writing debugging messages makes use of a
much larger internal circular buffer. Messages are written to this
buffer, and read by a user level process. An example would be:
The -f means to go ahead and wait for messages to arrive
in the buffer; printing them out as they occur. The debugmask
is a bitmask specifying which debug messages you're interested in.
Within the kernel, these various debug levels are denoted by using the
macros debug[0-5]() and debug[a-z](). The debugmask
program sets the appropriate kernel logging mask. For example,
debugmask 34 would turn debugging on for events 3 and 4 only.
- % debugmask -all
- % showlog -f
If you've built a new kernel and you find it doesn't boot properly,
there are a couple of methods for fixing the problem.
Hopefully, you've saved a working kernel under another name (e.g.,
/netbsd.OLD). If so, you should be able to interrupt
the boot process by hitting any key just after the boot blocks
have been loaded. Unfortunately, since these machines are so fast,
the timing loop that waits for input is very small. Hopefully this
will be fixed someday. If the boot process halts, you can then
just type the name of the alternate kernel.
If you can't seem to get the boot process to halt, or if you haven't
saved a stable kernel elsewhere, you have to place a new kernel in
the root partition from a fixit floppy. Currently, I have
a copy of the fixit floppy in /usr/distrib/floppies on
the host marilyn. The file kernel.fs is a filesystem
image...to place it onto a floppy use the following command.
You should then be able to boot with this floppy. Just hit return
at all the prompts, until you reach a shell prompt. The command
copy_kernel should install a new kernel on your root
filesystem. On all our existing machines, the root filesystem is
on disk wd0. In particular, it is in the wd0a
- % dd bs=8k if=kernel.fs of=/dev/fd0a
An incomplete list of known bugs:
- 48*N Bug -- Packets which are a multiple of 48 bytes
are not handled properly. They end up merging with the
next packet. An End-Of-Pdu bit is not set until a packet
arrives which is not a multiple of 48 bytes. Currently,
there is a slow software workaround which pulls individual
packets out of the merged buffer. However, there is no
guarantee that a non 48 byte packet will ever arrive or
that we will not run out of resources before it does.
If we are sending only between PCs which have this bug, we
can add a one word header to prevent 48N packets from ever
being sent. This does, however, make us incompatible with
other interfaces. To enable this feature, define the
identifier N48HACK either in the file
if_ijet.c or via an options line in the kernel configuration
- Spurious CRC errors are occasionally seen on the P90's.
While some of these are legitimate errors, many are due to
problems with the FIFOs overflowing when transmitting/receiving
at full rate. Tends to happen more with larger packet sizes.
The alphas seem to be able to send back-to-back packets fast
enough to cause the adaptor similar problems.
Here is a partial list, in no particular order, either of fixes that
will be applied to the driver or new features that will be implemented:
- Change ijetifioctl() to properly reset the board.
- Change credit processing to be on a per VC basis. Currently,
either all VCs use credit or none do.
- Allow credit to be updated via an ioctl(). Currently, if we run
out of credit, we will wait forever for another credit cell to
arrive. But the switch only sends out credit after a certain
amount of data has arrived. If we just happen to lose those
credit cells, the switch does not resend them. A simple ioctl()
will allow us to recover from this deadlock.
- Prashant is attempting to get Vince to work with our
adapter. This requires a different interface than the simple IP
Mounting the MSDOS file system
The DOS partition of the disk is accessible from NetBSD by mounting
the filesystem as root with:
# mount_msdos /dev/wd0f /mnt
Upon finishing, you can unmount the volume with
# umount /mnt
making sure that you are not in the /mnt tree or using any files
contained therein. Files can be copied back and forth between
different filesystems, but certain details have to be remembered.
DOS filenames are formatted as 8.3; 8 character primary name w/
a 3 character extension. I believe all filenames are folded
to uppercase when copying into DOS, and folded to lowercase
when moving to NetBSD. I've successfully untar'd files into
a DOS partition, but have had problems when simply trying to rename
directories. For this reason, it is probably a wise idea to
occasionally run scandisk from within DOS after modifying
Random things that probably aren't of much interest to anyone but me.
I'll put this in a separate file at some time.
- Boot off the kernel floppy. Switch to the install floppy. Run
the install program. Regardless of what NetBSD thinks the
cyl/tracks/sectors is, use what pfdisk (under dos) says it is.
In fact, use pfdisk to set up the NetBSD partition. If the
install program asks if you wish to overwrite the partition table,
you've screwed up. ^C and try again.
- Boot off the kernel floppy. Use this same floppy to copy a kernel
onto the disk.
- Reboot from the hard disk. Use ifconfig/route to put ourselves
on the net.
- Ftp a distribution/packages over from some machine that has them.
Do Set_tmp_dir, Extract foo.
- Get tarpkgs and packages as well.
- Change the following files:
- hostname.device, hosts
- myname, mygate, resolv.conf, localtime, rc.local
- copy master.pswd from some standard host, and group
- add /usr/local/bin/tcsh to shells