[Interop-dev] NetJSON feedback from Gluon/Freifunk [was: hello new subscribers]

[Nemesis]

Let's change subject.

I'm CCing Simon Wunderlich as the part of the local topology might
interest him as well.

On 05/31/2015 07:57 PM, Matthias Schiffer wrote:
> On 05/31/2015 07:05 PM, Nemesis wrote:
>> Hey Matthias,
>>
>> first of all, thank you for your precious feedback.
>>
>> On 05/31/2015 06:16 PM, Matthias Schiffer wrote:
>>> On 05/31/2015 03:09 PM, Nemesis wrote:
>>>> On 05/26/2015 02:16 AM, Matthias Schiffer wrote:
>>>>> On 05/19/2015 06:18 PM, L. Aaron Kaplan wrote:
>>>> [...]
>>>>>> But... this being said, now it's your turn for the new members to this list to speak up!
>>>>>> Do you want to briefly introduce yourselves?
>>>>>> What brought you here? What motivates you? What do you want to see done? What do you want to contribute?
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>> Aaron.
>>>>> Hello interop-dev!
>>>>>
>>>>> I'm Matthias (NeoRaider on Github / in IRC) from the Freifunk project of
>>>>> Lübeck, Germany. Frederico suggested me to join the list at the Wireless
>>>>> Community Weekend in Berlin last week.
>>>>>
>>>>> I'm one of the core developers of the Gluon firmware framework used by
>>>>> many German Freifunk communities. As we're also developing visualization
>>>>> systems in and around Gluon, we're in the constant process of inventing,
>>>>> refining and extending information formats like those NetJSON tries to
>>>>> define. The other Gluon core developer Nils, who has joined this list as
>>>>> well, is the original author of the ffmap and MeshViewer projects used
>>>>> by many Freifunk communities for visualization. Long story short, we'd
>>>>> love to make our tools more interoperable by making Gluon emit NetJSON
>>>>> and add NetJSON support to ffmap/MeshViewer :)
>>>> Welcome you all, I think we are all very happy to have you on board.
>>>>> You can see parts of our current data formats at [1] and [2] (look for
>>>>> the entry "68:72:51:20:db:cc" to see the current version, most nodes
>>>>> aren't updated to the latest Gluon version yet). Some information in our
>>>>> current format is redundant as we're trying to provide backwards
>>>>> compatiblity for at least a few Gluon versions. We're also currently
>>>>> replacing the alfred/batadv-vis-internal topology information format
>>>>> with an own extensible JSON format (to make it independent of alfred and
>>>>> generally more flexible).
>>>>>
>>>>> As you can see, many parts of our "nodeinfo" and "statistics" are
>>>>> already similar to the "Network Device Configuration" and "Device
>>>>> Monitoring" NetJSON objects. For the parts that aren't, I hope we'll
>>>>> find time soon to think up proposals how to extend NetJSON to support
>>>>> all of our usecases :)
>>>>>
>>>>> Matthias
>>>>>
>>>>>
>>>>> [1] http://metameute.de/~freifunk/alfred/nodeinfo.json
>>>>> [2] http://metameute.de/~freifunk/alfred/statistics.json
>>>> Amazing to notice you came up with a format that is not too different
>>>> from the NetworkDevice, which is a synthesis of the data one can get
>>>> from ubus, nodewatcher-agent, openwifi map and the nodeshot API.
>>>>
>>>> Well the next thing is to start using it in the real world. We'll be
>>>> doing that in the next couple of months, hopefully if other people do it
>>>> we can gather feedback and talk about it in person at the battlemesh and
>>>> CCC.
>>>>
>>>> Federico
>>>>
>>> We now have the first few nodes using our new topology format online.
>>> The current output can be found at
>>>
>>> http://metameute.de/~freifunk/alfred/neighbours.json
>>>
>>> As you can see, our format has some important differences from the
>>> Network Graph format in NetJSON: each "record" only shows the topology
>>> as seen from a single node, all nodes' records together are required to
>>> see the whole topology. This difference is natural as in the metrics we
>>> use (batman-adv TQ, WLAN signal/noise levels), only information about a
>>> node's direct neighbourhood is available. It doesn't match the Network
>>> Routes format well either, as our format doesn't contain information
>>> about the chosen routes to distant nodes either.
>> By looking at that JSON document it seems to me that it could be
>> converted to a NetworkGraph object. So it could be either be returned by
>> the firmware as an alternative format or it could be converted by a parser.
>> But I might be wrong, I should try implementing it to be sure.
>>
>>> Some other notes about differences between our formats and the NetJSON
>>> draft:
>> One premise: I see that many points are about missing info, or info that
>> is not interesting for you.
>> That should not be a big issue, because you can add custom properties if
>> needed, or you might omit optional data if it's not interesting. I'll
>> explain more in detail below.
>>
>>> * Our formats always contain a node_id attribute, so for
>>> statistics/neighbours the corresponding nodeinfo can be found
>> What does the format represent exactly? The topology as seen by some nodes?
> When I talk about nodeinfo, statistics and neighbours.json, I mean the
> three JSON files I sent links for. "nodeinfo" is our "Device
> Configuration" format, "statistics" corresponds to "Device Monitoring",
> and "neighbours" contains the topology as seen be a single node, which
> is only the direct neighbourhood of the node in our batman-adv based
> networks.

Sorry, I should have just re-checked.

>> In order to understand thoroughly I also need to ask: what kind of id is
>> node_id exactly? A database id or some kind of hash? How is it generated?
> It is an arbitrary unique stable identifier that is included in all our
> JSON formats. We currently use the node's primary MAC address (without
> the colons) as the node ID in Gluon, but a random string generated when
> a node is flashed for the first time would work as well.

thx

>>> * The Network Graph format doesn't contain any information about the
>>> specific interfaces a link belongs to on nodes with multiple interfaces,
>>> that might be especially interesting when there are different interface
>>> types (different WLAN bands/channels, wired, VPN tunnel, ...)
>> This data might be added in custom members or in the properties object.
>>
>> If more people find this useful, we can have a chat with routing
>> protocol devs to know if their software would be able to output a
>> complete NetworkGraph object including this type of data, so that we
>> might define this data as part of the spec.
> It would be useful to have a standard for a "local topology" format that
> can be generated on a single node. Our nodes are already sending
> compressed JSON over alfred, we'd like to switch to NetJSON for our
> "nodeinfo" and "statistics" formats when the specs have been written.

I see. Some kind of vector-state specific format?

> A "local topology" as contained in our "neighbours" format isn't just a
> partial "Network Graph", at least not in the way the "Network Graph"
> looks in the current draft. The current "Network Graph" format has a
> list of nodes with "router IDs" and a list of links between these nodes.
> But in a layer 2 routing protocol like batman-adv, we only know the MAC
> address of a neighbour's interface, but not the "router ID"/"node ID" of
> the neighbour the interface belongs to. So we might see a node on two
> different interfaces as two different MAC addresses and not even know
> that it is the same node. Only when the local topology of both nodes is
> available, this information ensues.

I see.

> NetJSONs "Network Graph" format makes sense when you know the whole
> topology and want to visualize it (or do whatever analysis you want).
> Processing the "local topology" information of all nodes into a single
> file similar to a "Network Graph" is what we do in our ffmap-backend.
>
> I think haveing standardized "local topology" format in NetJSON would be
> useful as well, so we can generate more or less compatible data for
> different routing protocols.

Fair enough, it sounds like a good proposal to me.

I would prefer to keep track of these proposals on github.

Would you mind opening an issue here?
https://github.com/interop-dev/netjson/issues

I am asking you to do it because at the moment if one looks at the
github issue tracker will think I'm talking to myself :-D

If you don't have time, let me know and I'll do it.

But if you do have time, it would be nice if you could propose a draft
JSON structure, then we can collectively improve it if needed.

>
>>
>>> * I find it a bit strange that Network Routes can be included in a
>>> Device Configuration. Is this supposed to contain static routes only?
>> Yes.
>>
>> At the moment this is also the only part of DeviceConfiguration I'm
>> notreally satisfied with and I would like to ammend.
>>
>>> Otherwise, including this into Device Monitoring would be more appropriate
>> Definitely
>>
>>> * I won't go much into detail about differences between nodeinfo and
>>> Device Configuration. Much information in the Device Configuration
>>> example looks much too verbose in my eyes. Especially the interface
>>> configuration parts contain many attributes that aren't very useful
>>> (have just been copied from the netifd output?)
>> It has been inspired by ubus and nodewatcher-agent.
>>
>> But now that you pointed it out you helped me realize something really
>> important.
>>
>> If we go back in the archive (i'm actually looking for the emails but
>> can't find them), we'll find out that we wrote that in the
>> DeviceConfiguration and DeviceMonitoring all (or most of) the blocks
>> should be optional.
>>
>> But that is not written in the spec so I should definitely add it right
>> now to clarify, what do you think?
>> I left out the details of DeviceConfiguration and DeviceMonitoring from
>> the spec because I wanted to get some more feedback before writing them
>> down.
>>
>>> * The traffic information in our statistics format is supplied by
>>> batman-adv over the ethtool interface. It contains some interesting data
>>> like the amount of management traffic, and discerns forwarded and
>>> locally transmitted/received traffic, but no interface-specific information
>> I see, is the statistic format one of the previous files you linked?
> Yes.
>
>>> * The monitoring format shouldn't contain any information about
>>> interfaces not used for meshing. This is important so "monitoring"
>>> doesn't turn into "surveillance" when someone uses a single OpenWrt
>>> system both as a community mesh node and a private router (which is
>>> something we plan to support with Gluon some time in the future)
>> I think the implementation of the format is in charge of deciding what
>> to do with it.
>>
>> For example: I work in a municipal wifi setting, we mostly don't use
>> mesh on our nodes, but we do need a format that describes this data.
>>
>> If your firmware doesn't need this it can simply omit it, but I believe
>> this misunderstanding is caused by the fact that the specification
>> doesn't clearly states that all the blocks are optional.
>>
>> What do you think?
>>
>> Just to clarify: the format should not dictate/impose to
>> expose/send/collect sensitive data. It just describes how to represent
>> data, each implementation will decide how to use it. The important thing
>> is that we find a way to output and parse this data in a standard (and
>> possibly easy) way.
> Makes sense.
>
>> Keep up the good work
>> Federico
>>
>>

Awesome productive sunday.

Federico

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: OpenPGP digital signature
URL: <http://lists.funkfeuer.at/pipermail/interop-dev/attachments/20150531/292a509d/attachment.sig>