What is a “fifo overflow”?

A "Fifo overflow" message is emitted when a component in the diagram has not been able to read all the data coming from an upstream component.

Let's recall how components exchange data! (see also Reader types)

Imagine a diagram containing 3 components, with 1 Writer and 2 Readers (whose inputs are of type FifoReader in our example). When the upstream component (the writer) emits data on its output, it stacks the data into a circular buffer associated to the output. The downstream components' inputs read the data from the stack as shown in the picture below:

Fifo overflow

Assuming that the readers' inputs are of type FifoReader, when a reader requests data on its input, it either:

  • falls asleep until a new piece of data arrives: this happens if the reader has already processed all the data from the writer.
  • or gets the oldest data from the Fifo that it has not processed yet.


The above picture demonstrates the following situation:

  • the Writer component stacks its samples of data in the circular buffer as soon as he can.
  • Reader_1 has already read and processed all the available samples that have been emitted by the Writer. Thus, it enters a sleeping state, and will be woken up immediately when the Writer will provide a new piece of data.
  • Reader_2 is late for some reason (not enough CPU resources, not enough bandwidth, ...): It has only read and processed the samples 1, 2, 3, and 4, it is now processing sample 5, and trying to catch up. When Reader_2 will have finished with sample 5, it will request a new data, get sample 6, process sample 6, request new data, get sample 7, etc...

Samples 1, 2, 3 and 4 are filled with light gray stripes in the picture to indicate that they have been processed by all the downstream components, and can now be overwritten by the writer without any problems when it reaches the end of the circular buffer.


If the lack of resources for Reader_2 is temporary, the circular buffer read by FifoReader inputs can be used to absorb temporary overloads of downstream components.

If the lack of resources for Reader_2 is permanent and the Writer generates data faster than Reader_2 can process it, after some time the Writer will fill entirely the circular buffer with data that has not been processed by Reader_2, then will have to overwrite such data: this will result in data loss for Reader_2.

When an upstream component (the Writer) overwrites data that has not been processed yet by all the downstream components connected to the output (Reader_1 and Reader_2), Fifo overflow warning messages are generated.

This indicates 2 problems:

  • at least one of the downstream components missed data, which is not an expected behaviour for a FifoReader input.
  • the downstream components which are late will always process (when they can) the oldest data available in the buffer, hence an induced latency between the components equal to the duration of the buffer.


There are several ways to deal with such a problem :

  • the first thing is to identify which component is late and why: lack of computation resources? lack of bandwidth? can it be blocked by any other process?
  • if the lack of resources is momentary, you can increase the size of the circular buffer (this is the FifoSize property of the component's output). This will reduce the chances of a data sample be missed by the downstream component. However if the lack of resources is permanent, the downstream component's latency will increase.
  • in case of a lack of CPU or bandwidth, a solution can be to reduce the incoming frame rate: edit the properties of the downstream component input and raise the subsampling property (a subsampling of 3 will make the component process one data out of 3 from the buffer). It is also possible to edit the properties of the upstream component output and raise the subsampling property there: the upstream component will provide less data to the buffer, but be aware that this will also reduce the sample rate for other downstream components that may accept a full sample rate.
  • otherwise, it is possible to change the input reader type to other kinds than FifoReader, such as LastOrNextReader or Wait4NextReader.

Vector management: Vector manipulator, Vector scale and offset

There are many components in RTMaps for vector manipulation.
  • Vectors scale and offset : enables the user to apply a linear transformation for all elements in the vector. The scale property is the multiplying factor and the offset property is an offset to the original value. In a formula : output[i] = input[i] * scale + offset
  • Vectorizer and Devectorizer : The first component concatenates N elements in a vector while the second splits a vector into N scalars.
  • Vector manipulator : allows to extract and re-organize vectors elements. The “dest vector elements” property must be filled as if you wanted to print pages from a document. For example, “0;1-4;6-8” builds a vector of 6 elements taken at indexes 0, 1, 4, 6, 7 and 8 of the input vector.

The RTMaps Updater

With RTMaps, you can check for updates very easily. Go in the Help menu, then click on [Check for Updates].
RTMaps starts immediately to scan installed archives and shows a console dialog box.

retrieving_installed_archive

Then RTMaps opens a window containing 2 tabs :

  • The “Search” tab : containing all the installed packages (.pck)
  • The “Installed” tab : here you can search for a particular packet. Click on the Update button and you should also see a list of packages that can be upgraded.

: Package not installed (you don’t have this package yet)
: Package already installed (you already have this package)
: Package that can be upgraded (new version available)
: Package that can be downgraded (the online version is older than yours)
: Package that can’t be installed (conflict with dependencies)

: Package marked for installation
: Package marked for reinstallation
: Package marked for downgrade
: Package marked for upgrade

 

In this example, the pck_rtmaps_qt_widgets is a new package available for download (upgrade). When you click on this package, it becomes green if everything is ok, or red if something is wrong.

When you click on the Next button, a Summary screen appears with all the packages that are ready to download, including dependencies.

The next screen simply asks you for a place to download the packages. If the directory is fine for you, just click on the “Download and install” button. RTMaps should reboot with your brand-new packages !

NB : You can configure update sites in Help [Preferences] [RTMaps options] [Update Sites]

Performance diagnostic

Sometimes you want to know the performance of a particular component. Or just to find where your bottleneck is. There is a couple of tools in RTMaps to help you in those cases.

Let’s take a simple example (see the diagram below). In this diagram we have a gradients and edges algorithm that we want to monitor. The Get_Processing_Duration component allows us to measure the computational time by subtracting the time of issue difference. Here in our case the DataViewer shows a 8 ms computational time.

performance_schema

Let’s have a look to the three monitors panels :

  • FIFO monitor
  • Performance monitor
  • Threads monitors

They can be activated via Window [Monitors] RTMaps menu.

The performance monitor indicates CPU usage of each component. Here we can see that the gradient and edges algorithm takes 21% of the total CPU resources (400% for a 4-cores architecture).

Performance monitor

 

The FIFO monitor shows the FIFO size, occupation and occupation rate. When everything goes fine, the FIFO is always almost empty.

fifo_monitor

 

Other components can also be used to perform a diagnostic : the Sample Rate component which displays in the console how many samples it has received in a second.

Note also that the data viewer has an info field which contains timestamp and time of issue.

New viewers and generators

Included with RTMaps 4.1, our new widget viewers and generators are available. Some replace the previous ones (Oscillo, Data Viewer), others are brand-new ones (Gauge, Led, Text Viewer, Slider, Potentiometer).

The widget properties are organized in two categories : the common properties and the custom properties. Some properties can only be changed in the property panel, but most of the important ones can be changed dynamically from the widget.

Common properties

Each widget have at least three options :

  • Always on top : Force the widget to stay on top of the screen. It can not be overlaid by another window, except the ones with this very same attribute. This property ensures you that the widget remains visible during the simulation. This property is enabled by default.
  • Moveable | Resizable : Enable the widget to be moved and resized. To resize the widgets, use the tiny grips at each corner. To move the widget, grab the title bar with the mouse as usual. On some widgets which don’t have focus on their content, you can use the whole widget zone to click and move the widget. The Moveable | Resizable property is enabled by default. Disable it if you want to prevent a perfectly placed windows from moving with a unwanted click and move.
  • Pause: If disabled, the widget runs normally. Turn it on to pause your widget.This property is disabled by default.

The pause property is generally reachable via a pause button on the title bar. It turns to green when on. The other properties are set via the contextual menu icon or by a right click on the title bar. Both method show a contextual menu with the properties. On small widgets, only the right click is activated, there are no icons on the title bar.

common properties

Figure 1 : The common properties showed on a data viewer

The data viewer

Role : Displays almost any kind of data in a tree way.

Let’s show this widget with a simple diagram:

simple example with data viewer

Figure 2 : Simple diagram with data viewer

Here are two input components : one Random integers and one Random Steam8. The data viewer is set by default with only one entry. Adding an entry is done by manually setting the “nb Inputs” property to 2 int the property panel. Let’s run the diagram, here is the data viewer component ‘s output:

full data viewer

Figure 3 : Data viewer widget

Each input has two fields :

  • Info : general info on the data : Time of issue, timestamp, vector size, buffer size, type and some optional fields like misc and frequency (if the data contains one of them).
  • Data : The data, shaped into a vector form.

Those two fields are collapsible so that you can choose what you want to display. In most cases the info field is collapsed and the data field is expanded by default. In the properties panel you have display modifier properties :

  • Show Info : Set it to true to make the Info node appear.
  • Show Data : Set it to true to make the Data node appear.
  • Compact Mode : Set it to true to enable the compact mode. In that mode, the data viewer tries to minimize screen space. It often hides the Info and Data leaf, displaying only the data without any additional information. Note that in most cases the two previous options have no effects when the compact mode is enabled.
  • Hex Display : property that force the data viewer to print all the values in hexadecimal.

In the case of a vector with a variable size during the simulation, the data viewer shows old data in light gray. In the figure 3, the random stream8 produces random data with random vector size. The current vector has 3 elements (the vector size) and can hold up to 10 elements (the buffer size). The elements from 3 to 9 are outdated, so they are printed in “light gray”.

The oscilloscope widget

Role : Represents data in an oscilloscope

Here is a simple diagram : a waveform generator (in sinusoidal mode) and a random integers component.

oscillo_diagram

Figure 4 : Simple diagram with oscilloscope widget

oscillo

Figure 5 : The oscilloscope widget

In the oscilloscope widget, there is one part dedicated to drawing and two panels:

  • The Common panel : Here the user controls the time scale (horizontal), the time offset and also the oscilloscope mode:
    • Mode scroll : the oscilloscope prints data as they arrive to it. This is the default mode.
    • Mode triggered : the oscilloscope starts printing when the input goes beyond or below a threshold (with a positive slope or a negative one depending on the slope property). When you decide to go in trigger mode, you have to select the triggered channel, the threshold value and the slope (positive, negative, any).
  • The Channels panel : Here the user can change the vertical scale and the offset. For each channel we have :
    • An autoset button : enables the oscillo to select the optimal vertical scale based on all the values already received.
    • An offset potentiometer (the small one) : changes offset
    • A vertical scale potentiometer (the bigger one) : changes the vertical scale.

Note that you can directly edit the value of a potentiometer by double clicking on the label. Once the label is showed as editable you can write your own value. Press enter to valid.

The text viewer widget

Role : Displays text in a dedicated console.

The text viewer can display almost any form of text (UTF8, Ascii). Here is a simple diagram (figure 6) : we use a random stream8 to feed the text viewer.

text_viewer_diagram

Figure 6 : Simple diagram with text viewer

In the properties you can change the background color and the font color (even during diagram execution). Default colors are black for the background and white for the font. You can also set the “Maximum Block Count” property. If different from 0, the text viewer will limit the display to n blocks. This enables the widget to limit the memory load if you intend to use it for a long time.

text_viewer

Figure 7 : The text viewer widget

The Led widget

Role : Two states Led.

In the led’s properties, you can change the on color and the off color, even during execution.
The led has only one input. A zero input will produce the off color. Any other values will produce the on color.

led

Figure 8 : The LED widget

The Gauge widget

Role : A speedometer

This widget is a classical gauge widget. You can change the max speed (used by the scale) in the properties. The gauge has two inputs, one for the speed (via the needle) and one for the engine speed (via the LCD number)

gauge

Figure 9 : The gauge widget

The potentiometer widget

Role : Enables the user to choose dynamically a value through a potentiometer.

The potentiometer has two modes :

  • On event (default) : The potentiometer outputs values only when the internal value changes.
  • Periodic : The potentiometer outputs values at each sampling period (can be changed in the properties panel)

You can set the minimum and maximum values in the properties panel.

Tips : A double click on the wheel will “reset” the potentiometer to its middle value. This is very useful when the min and the max are opposite values : the double click set the 0 value.

potentiometer

Figure 10 : The potentiometer widget

The slider widget

Role : Enables the user to choose dynamically a value through a slider.

Slider

Figure 11 : The slider widget

The slider can be horizontal (default) or vertical. Change it via the property Orientation.
You can also set the minimum and maximum values in the properties panel.

Keyboard shortcuts

These shortcuts can be found in Help [Key Assist] !


With focus on the RTMaps Studio main window:

  • Ctrl + D : show / hide Inputs/Outputs labels
  • Ctrl + N : New diagram
  • Ctrl + Shift + N : Force new diagram (don't ask for saving changes)
  • Ctrl + O : Open diagram
  • Ctrl + S : Save diagram
  • Ctrl + M : Load most recent diagram. This is an extremely interesting shortcut. Especially when you work often with the same diagram (typical example : component debugging)
  • Ctrl + Shift + M : Force loading the most recent diagram -> no proposal to save changes (e.g. Restore current diagram)
  • Ctrl + K : Register most recent package
  • Ctrl + Shift + K : Unregister most recent package
  • Ctrl + Shift + U : Unregister all packages
  • Ctrl + P : Show / Hide Parse window
  • Ctrl + L : Show / Hide Components list window
  • Ctrl + R : Run
  • Ctrl + T : Shutdown
  • Ctrl + E : Register/Unregister package



With focus on the VCR window:

  • Ctrl + R : Run
  • Ctrl + T : Shutdown
  • Ctrl + Enter : Start/Stop recording
  • Ctrl + space : Pause / Unpause
  • Ctrl + up : Speed up 10%
  • Ctrl + down : Speed down 10%
  • Ctrl + right : Play (100% timespeed)
  • Ctrl + left : Play backwards (-100% timespeed)

 

 

Image management

RTMaps offers tools to deal with images : color space conversion, JPEG compression / decompression, and a muxer to display multiple images in only one.

Color space conversion

In the Image processing package, there is a RGB -> YUV and a YUV -> RGB component.

If what you want to do is more than that, you can go for the image_processing_opencv package, which has a Colorspace Converter component. This component has two properties : the input colorspace and the output colorspace.

Compression / Decompression JPEG

In the Image processing package, there are two components : JPEG compressor and JPEG decompressor.

Realize an image puzzle - Show several images in the same window !

In the opencv package, you have a Video Muxer component. You can choose the number of inputs, and parameters for each image : width and height for the output image, positions of the input images, etc...

Here are the signification of the properties :

Muxer

How to perform a step by step replay ?

Those buttons are only activated when at least one of the player's outputs is in autopause.

To edit the properties of a specified output, just click on the output of the component (orange circle in the figure just below).

 

output_component

The output properties should show up in the properties panel. Edit the properties then set the autopause property to true. Now the RTMaps time will freeze each time a new data is emitted by those Player outputs.

player_output_properties

Do not forget to set the "Time lag" property on the Player to "infinite".Indeed, the "Time lag" property of the Player defines the delay the Player wait until it is allowed to drop lines in the .rec file in order to catch up. In such case it emits warning messages in the console like "Player can't stand real-time replay. Skipping data...".

How do I deploy my application (without the studio)?

Once your diagram is running as you expected, you can deploy your application anywhere using the RTMaps Runtime (you still need a valid Runtime licence). Here are a couple of ways to do that.

The simplest way : command line execution

Assuming your diagram is called my_diagram.rtd

In a console, type :

rtmaps_runtime --run --console my_diagram.rtd

NB 1 : The --console is optional, it just shows the RTMaps console so that you can still interact with RTMaps.

NB 2 : Your computer have to be able to find the rtmaps_runtime, so execute this line in the bin/ RTMaps folder, or add the bin/ folder to the path of your system.

This is a great simple way to deploy your application, but it has some limitations : you may want to have a different behaviour on your diagram and on your deployed application. For example, you may want to hide viewers which are for debugging purpose only.

The rtm script way

Still assuming your diagram is called my_diagram.rtd, you can export it using the File [Export] menu in RTMaps. Specify your diagram, and choose a destination with an .rtm extension (for example my_diagram.rtm).

The my_diagram.rtm is written in the RTMaps building script language. You can have a look by editing the my_diagram.rtm file. All the instructions are those showed by the RTMaps console when you do actions in the RTMaps studio. For example, let’s have a look to the following three lines (very simple diagram, one webcam connected to an image viewer).

Webcam Webcam_1
ImageViewer_VMR9 ImageViewer_VMR9_2
Webcam_1.outputMAPSImage -> ImageViewer_VMR9_2.input

The first line actually creates a webcam and puts it on the diagram.
The second line creates an image viewer and puts it on the diagram.
The third line connects the webcam’s output to the image viewer’s input.

By adding the command “run” to the end of the file, your script is ready to be executed in a console :

rtmaps_runtime --run  my_diagram.rtm

NB : This script naturally shows the console. So if you want to hide it, just add a --no-console in the script.

This solution is not really better than the first because any changes of the original diagram would break the script. So you would have to export your diagram again and again anytime you modify it.

The modular way : script with loaddiagram

This is the best solution to deploy and keep modifying your diagram at the same time!

Create or edit your my_diagram.rtm and just type loaddiagram <<my_diagram.rtd>>.

Add a run just after and that’s all ! The first line loads the diagram you already created, and the second runs the diagram.

This solution is definitely better because any changes in the original diagram would be automatically reported in the script, and you can still add some custom commands just after. For example :

kill ImageViewer_VMR9_2

to prevent the display of the image in the case of our simple webcam diagram. So our final script looks like :

loaddiagram <<my_diagram.rtd>>
run
kill ImageViewer_VMR9_2

Multiple inputs reading policies

When many writer components are connected to the same reader component, many questions come to mind! How do I keep sources synchronized? What do I have to do if one of my inputs is slower than the others? Actually there are many ways to connect multiple components to one reader component.

Definitions

Asynchronous reading on N inputs

In that case one input will be processed at a time. The component will wait for any of the inputs. When the input X has finished to write some data, the component reads this input X, and starts waiting again.

Triggered reading

One of the inputs (the input X) is responsible for clocking the component. It means when the input X has finished writing some data, the component reads input X and all others inputs (like a sampling reader).

Resampling

The component reads all inputs at the same time (sampling reader for all the inputs).

Synchro reading

The component processes only samples that have the same timestamp, with a given tolerance.
If the tolerance is zero, only data with the exact same timestamp will be read!

Sorted reading - expert only

The component reorders the samples read from inputs so that they are really read in ascending order relative to their timestamps.

Examples

The video muxer

Video muxer

Figure 1: Video muxer example


Here we have a Webcam and a Random Image configured with the same resolution. The Webcam delivers 30 images per second, and the Random Image is set to deliver 2 images per second. These two image streams are fed to an OpenCV Video Muxer component that concatenates the two images side by side. An image viewer shows the resulting image and a sample rate checks the output frame rate.

The default configuration for the OpenCV Muxer is triggered mode on the first input. So 30 times per second the video muxer reads the webcam, and then reads the random image. The image viewer shows very smooth images and the sample reader indicates 30 samples /sec.

Now what happens if if we change the trigger to the second input? The video muxer reads the random image two times per second, and then reads the webcam input. So on the video muxer output we have only 2 samples per second! The video is definitely not smooth.

Now let’s use the synchronized mode for the video muxer. In that case, we have no output! The sample rate indicates 0 samples / second. Why? Because our inputs are not synchronized as they have different timestamps (it’s almost impossible they have the same). So we have no video here! To solve this problem, we have to set a synchro tolerance. With a synchro tolerance of 20ms, we get our (smooth) video again!

Finally, let’s use the last mode: asynchronous. In that case the video muxer reacts every time an input is ready. It means every time the webcam is ready and every time the random image is ready. On the output we have 32 samples per second and a smooth video!


The pattern recognition example

Pattern diagram

Figure 2: pattern diagram


In that example, we will perform a face detection. We use a pattern recognition component which produces a red bounding box (rectangle). To superimpose the bounding box with the image we use an Overlay Drawing component.

The default behaviour of the Overlay Drawing is asynchronous reading, but it really acts as a trigger reading actually. Indeed, when a bounding box from pattern recognition is available for reading, the Overlay Drawing component stores it in memory. So when the webcam component outputs an image, the overlayDrawing reads it and uses the bounding box in memory to perform the overlay.

In conclusion, the bounding box is always late relative to the image. So the bounding rectangle appears to be shifted relative to the face when there is global image motion.

Tracking without synchro

Figure 3: Face moving from right to left. Bounding box is shifted


To fix the problem, we have to change the reader type by selecting “Synchronized” in the OverlayDrawing properties. Now the component checks timestamps (it’s actually doing a synchro start reading).

Now even when the face is moving fast, the bounding box keeps being synchronized with it.

Tracking with synchro

Figure 4: Bounding box is correctly placed