Data Formats
I-LOFAR produces very large data streams which are stored in different file types depending on the configuration of the software and hardware control parameters. The type of experiment being run determines the file type most suitable. In some cases it is not possible to make different file types simultaneously.
Data Formats
Data are output from RSP boards and TBBs in two different ways: some data are being transmitted to central processing, whereas options for writing data to LCU hard drive are available as well. Raw voltage level beam formed data are the only data product that cannot be written directly to files, but the station can only output it via a network connection. For TBB data there are options for both local data recording and CEP transmission, but the file recording option is only for testing purposes, and not fast enough for recording significant amounts of TBB data. Subband statistics, beamlet statistics, and array covariances (visibilities) can be recorded only to LCU hard drive.
I-LOFAR provides 4 main data files detailed below.
1. Subband-Statistics (SST)
The 1-second average power, for each subband, for each of the 96 receiver units is recorded to a separate file. 96 files are produced with N records (where N is the number of seconds), where each record is 512 subbands in size.
The file name contains a timestamp, the string ”sst” to indicate that the file contains subband statistics data, and the RCU number. As an example, the file
20171205_105031_sst_rcu000.dat
was recorded December 5 2017 10:50:31 UT from RCU 0. The data are written as 64-bit little-endian floats without any header. One data point per integration period is produced.
2. Beamlet Statistics (BST)
The 1-second average power, for each beamlet for each polarisation is recorded and two files are produced. The two correspond to the X and Y polarisation, with N records (where N is the number of seconds), where each record is B beamlets in size. There are either 244×16-bit, 488×8-bit or 976×4-bit beamlets. The data are written as 64-bit little-endian floats without any header.
The file name contains a timestamp, the string ”bst” to indicate that the file contains beamlet statistics data, number of the RSP board, and ’X’ or ’Y’ indicating the polarisation. As an example, the file
20171205_105947_bst_02Y.dat
was recorded December 5 2017 10:59:47 UT from RSP board 2, and it contains Y-polarisation data.
Each of the selected RSP boards will output 244 data samples per integration period. They are written to the files one integration period at a time, always starting from the beamlet number 0.
Notice that the boards will output 244 beamlet samples, even though only 61 beamlets per board are actually complete beamforming results. The remaining 183 data points are intermediate values from middle of the beamforming ring, and probably not what the user is interested in. The actual final beamforming products from each beamforming lane are the following:
In order to select only the final beamforming products, one needs to pick the beamlets 0 – 60 from the output of lane 0, beamlets 61 – 121 from the output of lane 1, beamlets 122 – 182 from lane 2 or beamlets 183 – 243 from lane 3 . Usually the RSP board numbers match with the lane numbers, i.e. lane 0 is output from RSP board 0 etc.
3. Array Covariance Cubes (ACC) and Cross-correlation Statistics (XST)
ACC files are also referred to as Antenna Cross-Correlation files. These files contain visibility data for all the baseline pairs within the array. The data are written as 64-bit little-endian floats without any header. The data is averaged over a time period of 1 second. After 512 seconds the program resets and a new file is created.
If the station has N RCUs, an N × N complex array covariance matrix will be produced from each integration period. Denoting the covariance of receivers i and j with xi,j, the data points of single integration period are written to the file in the order
Re(x0,0), Im(x0,0), Re(x0,1), Im(x0,1), Re(x0,2), Im(x0,2), . . . , Re(xNrcu,Nrcu ), Im(xNrcu,Nrcu ), where Re and Im denote the real and imaginary parts of the complex data values.
If array covariances are recorded for station calibration the files will be different. In the file name, the timestamp will be followed with ”acc” (”array covariance cube”), and dimensions of the covariance matrices stored in the file. As an example, the file
20171209_104029_acc_512x192x192.dat
was recorded December 9 2017 10:40:29 UT, and it contains 192 × 192 covariance matrices from 512 sub- bands. 192 is the number of LBA elements in the station (LBA was used in the experiment). Covariance matrices from each subband are recorded subsequently with increasing subband number, i.e. first 192 x 192 = 36864 complex samples are from subband 0, the next 36864 samples from subband 1, etc.
XST files are similar to ACC files, however each second a single subband is recorded, rather than stepping through them. Furhtermore unlike the ACC files data is recorded continuously. It is important to note that it is not possible to record both ACC and XST files simultaneously.
The file name contains a timestamp and the string ”xst” to indicate that it contains crosslet statistics (covariance, visibility) data. As an example, the file
20171205_114317_xst.dat
was recorded 2017 December 05 11:43:17 UT.
4. Transient Buffer Boards (TBB)
TBBs provide a snapshot of the running data-streams from the antennas and operates in parallel with the normal streaming data processing. Each TBB stores up to 32 GB of data, enough for 8 dual polarized antennas (individual dipoles for LBA or tiles for HBA) either before or after the conversion to subbands (the polyphase filter and FFT). 5 seconds of raw data is recorded at the full time resolution of 5 ns.