[Interop-dev] hello new subscribers

[Matthias Schiffer]

On 05/31/2015 07:05 PM, Nemesis wrote:
> Hey Matthias,
> 
> first of all, thank you for your precious feedback.
> 
> On 05/31/2015 06:16 PM, Matthias Schiffer wrote:
>> On 05/31/2015 03:09 PM, Nemesis wrote:
>>> On 05/26/2015 02:16 AM, Matthias Schiffer wrote:
>>>> On 05/19/2015 06:18 PM, L. Aaron Kaplan wrote:
>>> [...]
>>>>> But... this being said, now it's your turn for the new members to this list to speak up!
>>>>> Do you want to briefly introduce yourselves?
>>>>> What brought you here? What motivates you? What do you want to see done? What do you want to contribute?
>>>>>
>>>>>
>>>>> Thanks,
>>>>> Aaron.
>>>> Hello interop-dev!
>>>>
>>>> I'm Matthias (NeoRaider on Github / in IRC) from the Freifunk project of
>>>> Lübeck, Germany. Frederico suggested me to join the list at the Wireless
>>>> Community Weekend in Berlin last week.
>>>>
>>>> I'm one of the core developers of the Gluon firmware framework used by
>>>> many German Freifunk communities. As we're also developing visualization
>>>> systems in and around Gluon, we're in the constant process of inventing,
>>>> refining and extending information formats like those NetJSON tries to
>>>> define. The other Gluon core developer Nils, who has joined this list as
>>>> well, is the original author of the ffmap and MeshViewer projects used
>>>> by many Freifunk communities for visualization. Long story short, we'd
>>>> love to make our tools more interoperable by making Gluon emit NetJSON
>>>> and add NetJSON support to ffmap/MeshViewer :)
>>> Welcome you all, I think we are all very happy to have you on board.
>>>
>>>> You can see parts of our current data formats at [1] and [2] (look for
>>>> the entry "68:72:51:20:db:cc" to see the current version, most nodes
>>>> aren't updated to the latest Gluon version yet). Some information in our
>>>> current format is redundant as we're trying to provide backwards
>>>> compatiblity for at least a few Gluon versions. We're also currently
>>>> replacing the alfred/batadv-vis-internal topology information format
>>>> with an own extensible JSON format (to make it independent of alfred and
>>>> generally more flexible).
>>>>
>>>> As you can see, many parts of our "nodeinfo" and "statistics" are
>>>> already similar to the "Network Device Configuration" and "Device
>>>> Monitoring" NetJSON objects. For the parts that aren't, I hope we'll
>>>> find time soon to think up proposals how to extend NetJSON to support
>>>> all of our usecases :)
>>>>
>>>> Matthias
>>>>
>>>>
>>>> [1] http://metameute.de/~freifunk/alfred/nodeinfo.json
>>>> [2] http://metameute.de/~freifunk/alfred/statistics.json
>>> Amazing to notice you came up with a format that is not too different
>>> from the NetworkDevice, which is a synthesis of the data one can get
>>> from ubus, nodewatcher-agent, openwifi map and the nodeshot API.
>>>
>>> Well the next thing is to start using it in the real world. We'll be
>>> doing that in the next couple of months, hopefully if other people do it
>>> we can gather feedback and talk about it in person at the battlemesh and
>>> CCC.
>>>
>>> Federico
>>>
>> We now have the first few nodes using our new topology format online.
>> The current output can be found at
>>
>> http://metameute.de/~freifunk/alfred/neighbours.json
>>
>> As you can see, our format has some important differences from the
>> Network Graph format in NetJSON: each "record" only shows the topology
>> as seen from a single node, all nodes' records together are required to
>> see the whole topology. This difference is natural as in the metrics we
>> use (batman-adv TQ, WLAN signal/noise levels), only information about a
>> node's direct neighbourhood is available. It doesn't match the Network
>> Routes format well either, as our format doesn't contain information
>> about the chosen routes to distant nodes either.
> 
> By looking at that JSON document it seems to me that it could be
> converted to a NetworkGraph object. So it could be either be returned by
> the firmware as an alternative format or it could be converted by a parser.
> But I might be wrong, I should try implementing it to be sure.
> 
>> Some other notes about differences between our formats and the NetJSON
>> draft:
> 
> One premise: I see that many points are about missing info, or info that
> is not interesting for you.
> That should not be a big issue, because you can add custom properties if
> needed, or you might omit optional data if it's not interesting. I'll
> explain more in detail below.
> 
>> * Our formats always contain a node_id attribute, so for
>> statistics/neighbours the corresponding nodeinfo can be found
> 
> What does the format represent exactly? The topology as seen by some nodes?

When I talk about nodeinfo, statistics and neighbours.json, I mean the
three JSON files I sent links for. "nodeinfo" is our "Device
Configuration" format, "statistics" corresponds to "Device Monitoring",
and "neighbours" contains the topology as seen be a single node, which
is only the direct neighbourhood of the node in our batman-adv based
networks.

> 
> In order to understand thoroughly I also need to ask: what kind of id is
> node_id exactly? A database id or some kind of hash? How is it generated?

It is an arbitrary unique stable identifier that is included in all our
JSON formats. We currently use the node's primary MAC address (without
the colons) as the node ID in Gluon, but a random string generated when
a node is flashed for the first time would work as well.

> 
>> * The Network Graph format doesn't contain any information about the
>> specific interfaces a link belongs to on nodes with multiple interfaces,
>> that might be especially interesting when there are different interface
>> types (different WLAN bands/channels, wired, VPN tunnel, ...)
> 
> This data might be added in custom members or in the properties object.
> 
> If more people find this useful, we can have a chat with routing
> protocol devs to know if their software would be able to output a
> complete NetworkGraph object including this type of data, so that we
> might define this data as part of the spec.

It would be useful to have a standard for a "local topology" format that
can be generated on a single node. Our nodes are already sending
compressed JSON over alfred, we'd like to switch to NetJSON for our
"nodeinfo" and "statistics" formats when the specs have been written.

A "local topology" as contained in our "neighbours" format isn't just a
partial "Network Graph", at least not in the way the "Network Graph"
looks in the current draft. The current "Network Graph" format has a
list of nodes with "router IDs" and a list of links between these nodes.
But in a layer 2 routing protocol like batman-adv, we only know the MAC
address of a neighbour's interface, but not the "router ID"/"node ID" of
the neighbour the interface belongs to. So we might see a node on two
different interfaces as two different MAC addresses and not even know
that it is the same node. Only when the local topology of both nodes is
available, this information ensues.

NetJSONs "Network Graph" format makes sense when you know the whole
topology and want to visualize it (or do whatever analysis you want).
Processing the "local topology" information of all nodes into a single
file similar to a "Network Graph" is what we do in our ffmap-backend.

I think haveing standardized "local topology" format in NetJSON would be
useful as well, so we can generate more or less compatible data for
different routing protocols.

> 
>> * I find it a bit strange that Network Routes can be included in a
>> Device Configuration. Is this supposed to contain static routes only?
> 
> Yes.
> 
> At the moment this is also the only part of DeviceConfiguration I'm
> notreally satisfied with and I would like to ammend.
> 
>> Otherwise, including this into Device Monitoring would be more appropriate
> 
> Definitely
> 
>> * I won't go much into detail about differences between nodeinfo and
>> Device Configuration. Much information in the Device Configuration
>> example looks much too verbose in my eyes. Especially the interface
>> configuration parts contain many attributes that aren't very useful
>> (have just been copied from the netifd output?)
> 
> It has been inspired by ubus and nodewatcher-agent.
> 
> But now that you pointed it out you helped me realize something really
> important.
> 
> If we go back in the archive (i'm actually looking for the emails but
> can't find them), we'll find out that we wrote that in the
> DeviceConfiguration and DeviceMonitoring all (or most of) the blocks
> should be optional.
> 
> But that is not written in the spec so I should definitely add it right
> now to clarify, what do you think?
> I left out the details of DeviceConfiguration and DeviceMonitoring from
> the spec because I wanted to get some more feedback before writing them
> down.
> 
>> * The traffic information in our statistics format is supplied by
>> batman-adv over the ethtool interface. It contains some interesting data
>> like the amount of management traffic, and discerns forwarded and
>> locally transmitted/received traffic, but no interface-specific information
> 
> I see, is the statistic format one of the previous files you linked?

Yes.

> 
>> * The monitoring format shouldn't contain any information about
>> interfaces not used for meshing. This is important so "monitoring"
>> doesn't turn into "surveillance" when someone uses a single OpenWrt
>> system both as a community mesh node and a private router (which is
>> something we plan to support with Gluon some time in the future)
> 
> I think the implementation of the format is in charge of deciding what
> to do with it.
> 
> For example: I work in a municipal wifi setting, we mostly don't use
> mesh on our nodes, but we do need a format that describes this data.
> 
> If your firmware doesn't need this it can simply omit it, but I believe
> this misunderstanding is caused by the fact that the specification
> doesn't clearly states that all the blocks are optional.
> 
> What do you think?
> 
> Just to clarify: the format should not dictate/impose to
> expose/send/collect sensitive data. It just describes how to represent
> data, each implementation will decide how to use it. The important thing
> is that we find a way to output and parse this data in a standard (and
> possibly easy) way.

Makes sense.

> 
> Keep up the good work
> Federico
> 


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://lists.funkfeuer.at/pipermail/interop-dev/attachments/20150531/d165d770/attachment.sig>