Station Control using the LCU

Software Level

Control processes in a LOFAR station are collected in groups called software levels. The user does not need to manually start and stop a number of processes, but each level contains processes for certain actions in the station, and appropriate processes are started / stopped when entering a given level. Users can query and set the software level using the swlevel command. When software level is changed in steps larger than one, the system goes through the intermediate levels as well, and starts/stops processes as needed. Processes belonging to each level are listed in the help message of the swlevel command.

0: Stop all lofar software
1: Run Lofar daemons and PVSS
2: Run Lowlevel hardware drivers
3: Run Calculation services
4: Run Hardware and software monitors
5: Run System Health Management
6: MAC is controlling the software
Software levels 0 . . . 3 are relevant in single station use, the processes belonging to the higher levels 4. . .6 are intended to be used in ILT (International LOFAR Telescope) mode.

After entering software level 2, or higher if the initial level was below 2, one needs to wait for a register update to complete. Status of the registers can be displayed with the command

rspctl –regstat,
which repeatedly prints several lines of text to the screen. The registers are up to date when all lines consist only of the characters ’.’ and ’*’, after which the rspctl process can be stopped with ctrl-c.

In short, beam forming requires software level 3, whereas other processes making use of RSP boards and TBBs require software level 2. All processes belonging to a software level below the current one are always available, so everything necessary for the single station use covered by this document is available if the system is running at software level 3.

Beam Control

Station beamforming uses the subband data produced by the polyphase filters, and produces a number of beamformed subbands called beamlets. Beams with larger spectral width can be produced by forming several beamlets with same pointing direction but different centre frequency. A LOFAR station can form up to 244 beamlets, each of which can have its own pointing direction and centre frequency. The user has large freedom in selecting the frequencies and pointing directions, but certain technical restrictions need to be taken into account:

1. A RCU has three inputs, only one of which can be active at a time. Inside a receiver the signal will go through an analogue band pass filter, where only one filter at a time can be selected. Due to these limitations, the user must select a single receiver band (a combination of an input connector and a band pass filter) for each RCU.

2. Subbands 0 – 511 are available in principle. However, the receiver passband, i.e. passband of the analogue filter in RCUs, is always narrower than the full receiver band. A few lowest and highest subbands should thus not be used. Especially subband 0 should not be used, because it has contribution from both lowest and highest ends of the receiver band and it is vulnerable to bias voltage in the receivers.

3. Third restriction arises when using the HBA array, which makes use of the analogue beamformers inside the HBA tiles. Because the tile beam can point only to one direction at a time, all digital pointing directions must be close enough to each other, so that they fall within the tile beam. Width of a tile beam is approximately 30 degrees at 150 MHz, and it is inversely proportional to the frequency. The system does not make any sanity checks about the difference between the analogue and digital beam directions, and it is on the user’s responsibility to allocate the beams properly.

A single-station user has access to the station beamforming via the beamctl command. The command can be used when the system is running at software level 3. The beamctl command can handle several tasks, such as selecting receivers and receiver modes, which are accessible via the rspctl command as well. However, beamctl is the preferred way to control also the receivers, because it automatically starts and stops the receivers as needed, and automatically starts up the system slowly enough to avoid problems with high rush-in currents.

Coordinate systems

The present BeamServer, as well as the beamctl command, accept most coordinate systems that are defined in casacore.

RSP Control

Local user control of most signal processing devices in a LOFAR station, including most functionality of the RSP boards, the RCUs, and the HBA tile beamformers, is collected under the rspctl command.

The command allows the user to query and set several parameters, some of which are included as options in the beamctl command as well. When possible, it is advisable to use the beamctl command instead of rspctl, because the beamctl command takes automatically care of e.g. starting and stopping the relevant RCUs, and always starts up the receivers slowly enough to avoid problems with high rush-in currents. One should also notice that an allocated beam causes the system to regularly update all beamforming weights, and manual settings with rspctl may thus be rapidly reset by the system.5

If the beamctl command is being used to allocate beams, the user does not usually need to set any parameters with the rspctl command. However, rspctl allows the user to query several parameters, which may be interesting for checking the system status and for trouble shooting.

The rspctl command is first needed already before giving any beamctl commands for checking the register status with

rspctl –regstat
After allocating beams, it is useful to check that the RCUs are on and

running in correct mode with the command

rspctl –rcu
and to check status of the RSP boards with the command rspctl –status.
The command

rspctl –subbands
can be used to check which subbands are actually received from each RCU. The output is not completely self-explanatory; for even-numbered RCUs the subband numbers are multiplied by two, and for odd- numbered RCUs the subband numbers are multiplied by two and the result is incremented by 1. As an example, subband 10 is denoted as 20 in output of even-numbered RCUs and as 21 in output of odd-numbered RCUs. rspctl –subbands also prints the total number of subbands produced by each RCU, as value of the variable ”subbands”.

Spectra from individual RCUs (antenna elements) can be monitored with the command

rspctl –statistics,
which repeatedly plots power spectra of all receivers.6 It is also possible to plot the signal powers of each beamlet by using the command

rspctl –statistics=beamlet
or array covariance matrix at a selected subband with

rspctl –xcsubband= rspctl –xcstatistics.
The above commands also allow the signal statistics to be recorded in files.  Users who want to send raw beamlet data to CEP or some other network device can switch the UDP data stream off/on and query its status with

rspctl –datastream[=0|1|2|3].
Only the options 0 (off) and 1 (on) are used in international stations. The options 2 and 3 are used in core stations, where the ring of RSP boards can be split in two halves, and data output from both of these need to be controlled separately.

If the user wants to use the receivers without forming beams, they need to be started manually with a couple of rspctl commands. In order to guarantee that the beam server does not reset the user-defined settings, one should begin with switching to software level 2, which disables the beam server,

swlevel 2,
and then wait for register update, i.e. wait until the output of

rspctl –regstat
consists only of the characters ’*’ and ’.’.

The current receiver mode can be queried with

rspctl –rcu.
If one wants to switch to one of the receiver modes 1 – 4, the mode can be changed with

rspctl –rcumode=1|2|3|4.
Do not use the command rspctl –rcumode=5|6|7! This command will switch on the 48 V power supply of all HBA tiles simultaneously. The resulting rush-in current drawn from the station power supply drops its voltage down to 44 V which, in turn, causes freezing RSP boards. In order to switch to the higher receiver modes, which use the HBA array, one should use the script poweruphba.sh 5|6|7, which performs the startup slowly enough to maintain a high enough voltage. The script should be available at all stations.

After the previous commands the receivers are running in correct mode, but the data stream from the receivers to the RSP boards still needs to be started with

rspctl –rcuenable[=1|0].
If running in receiver mode 5, the received spectrum will be inverted, and one might be interested in inverting it back with

rspctl –specinv=[1|0].

Notice that this command is not needed if the receiver mode is selected with beamctl, which automatically inverts the spectrum if receiver mode 5 is selected.

After these commands one can check that the system is running properly with the commands

rspctl –rcu
rspctl –status
rspctl –statistics.
The rspctl command allows the user to set and query numerous parameters.

Back to the I-LOFAR Observer’s Guide

Station Signal Processing Pipeline

1. Antennas

LOFAR station is a wide-band radio receiver, which operates at the frequency range from 10 MHz to 240 MHz. The full reception band is divided into low band (10 MHz – 90 MHz) and high band (110 MHz – 240 MHz). The frequencies in between low band and high band are not used due to RFI from FM radio transmitters.

The station signal processing has been originally designed assuming that three different antenna arrays would be used. These arrays would have been the low-band-low (LBL) antenna array, the low-band-high (LBH) antenna array, and the high-band antenna (HBA) array. However, the present standard stations do not have the LBL antennas, and the LBH antenna array is thus referred to as the low-band antenna (LBA) array.

Antennas used in LOFAR stations are briefly introduced in following two sections. Notice that the antenna array configurations are configurations of full antenna arrays. It is always possible to manually select only a subset of a full antenna array to be used e.g. in beamforming. Positions of individual antenna elements and antenna array definitions of a station are stored in station configuration files.

2. Receiver Control Units (RCUs)

The analogue signals from LBA elements and HBA tiles are transferred in coaxial cables to a LOFAR station cabinet, where each of the cables is connected to a receiver unit. A RCU performs input selection, followed by amplification and filtering of the analogue input signal. The conditioned analog signal is sampled with a 12-bit A/D converter. The A/D converter produces real signal samples at either 200 MHz or 160 MHz sampling frequency. Combinations of filter passband and sample clock frequency are selected so that the selected frequency band always aliases around zero frequency, without frequency aliasing inside the passband. With one of the available combinations the signal spectrum will be inverted, but it can be inverted back in subsequent processing steps.

The receiver units are numbered starting from zero. X-polarisation cables are connected to even-numbered RCUs and Y-polarisation cables are connected to odd-numbered ones.  The only deviation from this rule is the LBA outer field, whose X-polarisation cables are connected to odd-numbered RCUs and Y- polarisation cables to the even-numbered ones. A receiver unit has three inputs: LBL, LBH, and HBA. The LBL and LBH connectors have an 8 V bias voltage. The HBA X-polarisation connectors have the 48 V bias voltage for power supply. The HBA Y-polarisation connectors have a 3.3 V bias voltage only when communicating with the HBA tiles.

A core station or a remote station with 96 LBA elements and 48 HBA tiles has 96 RCUs. The LBA inner array is connected to the LBH inputs and the LBA outer array to the LBL inputs. Despite the naming convention in RCUs, both LBA inner and LBA outer use the same kind of antenna elements. The HBA tiles are connected to the HBA inputs.

Due to their larger number of HBA tiles, international stations have 192 RCUs. The 96 LBA elements of an international station are connected to the LBH inputs of the RCUs, and the 96 HBA tiles are connected to the HBA inputs. In a basic installation of an international station the LBL inputs are left empty, thus facilitating later installation of an additional antenna array.

3. Remote Station Processing Boards (RSPs)

After frequency band selection and A/D conversion in the RCUs, the received signals are transferred to remote station processing boards. The RSP boards perform all digital signal processing that is done at the station, and send data to central processor (CEP).

The discrete signal samples arriving from a RCU to a RSP board are first buffered in a FIFO buffer. In order to compensate for differences in signal delays in the coaxial cables, buffer length can be adjusted independently for each individual RCU. The buffer is followed by a polyphase filter, which divides the wide band input signal into so-called subbands.

A polyphase filter is a novel FFT-based implementation of a bandpass filter bank, which divides the real input signal into 1024 complex subband signals. Because the real input contains two identical (mirrored) copies of the signal spectrum – one at positive and the other at negative frequencies – the lowest 512 subbands are complex conjugates of the 512 highest ones. The whole receiver passband is thus covered by the 512 lowest subbands, which are used in further processing. Depending on sample clock frequency, subband width is either 200 MHz/1024 = 195.3125 kHz, or 160 MHz/1024 = 156.250 kHz.

After the polyphase filter the signal bandwidth is small enough to facilitate phased-array beamforming1, which produces beamformed subband signals called beamlets. Although 512 subbands are produced in the polyphase filter, only 244 beamlets can be formed by the station signal processing. Subbands used in beamforming are selected immediately after the polyphase filter. The beamforming is performed in a ring, where a process handling signal from one RCU receives a partial beamforming result, adds the contribution from its own RCU, and passes the result forward to the process handling signal from the next RCU. In order to facilitate the beamforming, the RSP boards of a LOFAR station are connected together as a ring. After a full cycle in the ring, the beamformed sample is complete and ready to be transmitted to central processing.

The beamforming ring is divided into four separate lanes, each of which has its start and end point in a different RSP board. Beamformed data are output from those RSP boards that are endpoints of the beamforming lanes. In core stations the beamforming lanes can be split into two halves, allowing independent beamforming with the HBA0 and HBA1 arrays. Because 244 beamlets can be formed with both arrays, the splitting allows core stations to form 488 beamlets simultaneously, but each of them with only 24 HBA tiles. Four additional output RSP boards are defined in the split mode, i.e. data are being output from eight RSP boards simultaneously.

In addition to the beamforming, the RSP boards can calculate so-called subband statistics , beamlet statistics , and array covariances (crosslet statistics) . Likewise the beamlets, the co- variances are also formed in the ring of RSP boards, and their calculation is divided into four lanes. All these crosslet lanes start from and end to the same RSP board, which also outputs the data. The crosslet output board is usually different from the beamlet output boards. The RSP boards can also send either raw A/D converter samples or subband data to ring buffers in transient buffer boards.

An RSP board has four antenna processors (AP). Each AP processes two orthogonal polarisations from its connected antenna. Each RSP board thus has 8 digital inputs for the signals from the RCUs, which defines the number of RSP boards in a station: core stations and remote stations have 12 RSP boards, whereas international stations have 24 RSP boards.

4. Transient Buffer Boards (TBBs)

Transient buffer boards are not part of the main signal processing chain, but their function is to store a short period of raw voltage data in a ring buffer. Writing to the buffer can be stopped when a triggering event is detected, after which the data can be read from the buffer.

TBBs are physically connected to the RSP boards, which provide the raw data that is buffered to the boards. Two kinds of data can be buffered: in ”transient” mode real signal samples are copied to TBBs immediately after the FIFO buffers in RSP boards, whereas in ”subbands” mode complex subband data produced by the polyphase filter are copied to the TBBs. TBB memory is divided into pages, which are large enough for 1024 12-bit data samples in ”transient” mode. Because the subband data is in 16-bit format, only 487 of the 512 subbands can be recorded in ”subbands” mode.

One TBB is capable of recording data from 16 RCUs, and each TBB is thus connected to two RSP boards. Core stations and remote stations thus have 6 TBBs, whereas international stations have 12 TBBs.

5. Local Control Unit (LCU)

Local Control Unit (LCU) is a computer running the Redhat Linux OS. All station control happens via the LCU, which runs a number of control processes that communicate with the different processing boards of the station. LCU also receives station clock signals from GPS and a rubidium standard. Users can log on to the LCU via a ssh connection, and control the whole station from the command line. Single-station control is covered in more detail in Section 3, and relevant commands are listed in the command reference.

Back to the I-LOFAR Observer’s Guide

Station Handover

Once you have completed your shift as I-LOFAR Chief Observer, please follow the following steps.

  1. Send an email to observer[at]astron.nl to inform ASTRON the station is ready to be handed over to international mode. CC observer[at]lofar.ie
  2. Send an email to observer[at]lofar.ie summarising the observations, mention the mode used and if other tests were run (e.g. RFI and TBB tests). You must also mention who the next I-LOFAR Chief Observer in this email.

Back to I-LOFAR Observer’s Guide

I-LOFAR User’s Guide

Welcome to the I-LOFAR User’s Guide. Below you will find details on how to log in to the I-LOFAR Control Computer (ICC) and the Local Control Computer (LCU) and how to run a variety of observation scripts. The ICC is located in the I-LOFAR Control Room, while the LCU is located in the International LOFAR Telescope (ILT) Container.

Please note that you are using a complex scientific instrument. In order not to cause any damage to the station, follow the procedures described below.

  1. Station Signal Processing Pipeline
  2. Logging In
  3. Station Control from the LCU
  4. Station Control from the ICC
  5. Data Formats
  6. Data Archives
  7. Local Processing Clusters
  8. Observer’s Log
  9. Station Handover

Further Reading

1. Making Observations With Mode 357 (document missing)
2. Observing with the Rawling Array (LOFAR-UK UK608)