[Interop-dev] Fwd: [netjsonconfig] Motivation of changes in 0.6.0 alpha

[Federico Capoano]

Updates regarding one of the NetJSON implementations.

---------- Forwarded message ---------
From: Federico Capoano <(spam-protected)>
Date: Wed, May 31, 2017 at 12:55 PM
Subject: [netjsonconfig] Motivation of changes in 0.6.0 alpha
To: OpenWISP <(spam-protected)>, Edoardo Putti <
(spam-protected)>, Ritwick D'Souza <(spam-protected)>


Hi everyone,

recently I started introducing some important changes in netjsonconfig
<https://github.com/openwisp/netjsonconfig>, the library used to generate
configuration of routers.

This change should not affect openwisp2 users negatively, I'm already
testing the current master on 3 instances and already fixed a few bugs.

Nevertheless, the change is affecting negatively the 2 GSoC students who
have started to work on the AirOS and Raspbian backends, so I will try to
explain why this change is needed.

*Backward conversion*

One of the features that is missing in OpenWISP 2 is the possibility to
easily store the full configuration of devices in the controller.

Since OpenWRT/LEDE allows merging a remote configuration with the local
one, that's the default mode we are using now in OpenWISP 2 (it was also
the only mode available in OpenWISP 1).

This method has its own disadvantages:

   - operators have to remember what is in their local configuration, in
   order to write a remote configuration that works well with the local one
   - some packages use anonymous UCI blocks by default, which are
   impossible for openwisp to override in the merge mode, hence creating
   duplicate configurations

These disadvantages are holding down OpenWISP 2 from becoming easier to
use. OpenWISP 2 is very powerful, but hard to use properly. Which is a
shame, because it means many people will give up and use some kind of home
baked solution with its own class of bugs, reinventing the wheel: no good
framework and ecosystem of tools for low cost open source networks can
emerge for such a situation.

That's why we need to add a way to store the full configuration of routers
in the controller and this needs to happen automatically: here where
backward conversion in netjsonconfig comes into play.

Without the backward conversion, we cannot even use the AirOS or Raspbian
backends because those don't have a way to easily merge different
configurations like OpenWRT/LEDE (uci import --merge), users would need to
manually enter the full configuration of their devices in openwisp2: that
will never happen!

*The new implementation*

I've been designing and planning this feature since at least a year, but
never got the time to implement because there were more urgent matters. Now
is the right time to introduce it because it's very much needed.

The new implementation adds a few new concepts in the codebase:

   - intermediate data structure
   - converters
   - parsers

Everything revolves around the data structures: the *NetJSON
<http://netjson.org/> configuration dictionary* and the *intermediate data
structures*.

Remember what *Linux Torvalds* said?
*"Bad programmers worry about the code. Good programmers worry about data
structures and their relationships."*

This intermediate data structure represents the native configuration of the
backend in a python data structure (lists and dictionaries).

This data structure is a convenient way to simplify conversion in both
direction, parsing and rendering, and will be different for each backend.

Take a look at this diagram:
[image: netjsonconfig backward conversion.jpg]
*Converter* classes take care of converting between NetJSON and
intermediate data structure (and vice versa):

   - to_intermediate converts the NetJSON configuration dictionary to the
   intermediate data structure
   - to_netjson (not pushed to the repository yet) converts the
   intermediate data structure to NetJSON configuration dictionary

These two main methods can then call helper methods in order to split
complex logic in smaller methods that are easier for the human mind to
grasp.

*Renderers* take care of rendering the intermediate data structure to the
native format.

*Parsers* perform the opposite operation of Renderers: they take care of
parsing native format and build the intermediate data structure.

In case of OpenWRT and AirOS, we can get away with just 1 renderer (with
just 1 jinja2 tempate) and 1 parser. In the case of Raspbian, we may need
to have a render and a template for each format that needs to be rendered
(eg: /etc/network/interfaces, ini files, dnsmasq, iptables, ecc).

*Advantages*

   - Cleaner, well defined workflow - also easier to document and learn
   - Less boilerplate code in those cases where the native configuration of
   the router is centralized in one format (OpenWRT / AirOS / PfSense / vyos),
   because most of the logic goes in converters while there can be only 1
   renderer and 1 parser
   - Last but not least, makes it possible to implement backward conversion
   without going crazy with messy spaghetti code. Our sanity is very important
   not only for our work but for our life!

*Disadvantages*

Developers using lower level features of netjsonconfig in their programs
and GSOC students working on the netjsonconfig backends will have to
rewrite part of their code. A bit of more work needed, but in my opinion a
worthy price to pay for a cleaner and more robust implementation.

*How to upgrade your code*

Proceed as follows:

   - the logic that before was contained in the renderers, should most
   probably go into the converters
   - design an intermediate data structure with the following features:
      - closely represents your native configuration format (or formats)
      - can be easily rendered with jinja2
      - can be easily built when parsing the native format (or formats)
   - simplify your renderers so that you have 1 renderer and 1 template for
   each format supported

*Conclusion*

That's it. Sorry for the length. I hope the explanation is clear enough.

Federico
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.funkfeuer.at/pipermail/interop-dev/attachments/20170531/ae634e88/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: netjsonconfig backward conversion.jpg
Type: image/jpeg
Size: 48136 bytes
Desc: not available
URL: <http://lists.funkfeuer.at/pipermail/interop-dev/attachments/20170531/ae634e88/attachment-0001.jpg>