enoch/n2n
1
0
Fork 0
mirror of https://github.com/ntop/n2n.git synced 2024-09-20 00:51:10 +02:00
Code Issues Actions Packages Projects Releases 1 Wiki Activity
n2n/doc/ManagementAPI.md

187 lines
6.3 KiB
Markdown
Raw Normal View History

Cleanup and Documentation for JSON management API (#856) * Reimplement JSON mgmt with clear separation of read/write actions * Reduce boilerplate by using a table driven command definition for json mgmt commands * Port tools to use new json api * Add a basic authentication for json mgmt commands * If a auth key is given, it must match * Add auth key to management scripts * Add a flag bitfield to clearly turn the tag param into a options list * Allow simple pass-through of any command from n2nctl * Convert the n2nctl to use an object oriented interface * Handle sigpipe in the n2nhttpd - this happens if the remote client disconnects unexpectely * Remove some repetition from the server * Use the correct options to allow reuseaddr * Dont generate a scary message on ctrl-c * Convert n2nhttpd to use object based RPC * Use the same longopt for both tools * Pass any extra args through to the RPC * Add some documentation for the scripts in the repository * Spelling fix * Add documentation for the JSON reply mangement API
2021-10-17 22:16:42 +02:00
# Management API
This document is focused on the machine readable API interfaces.
Both the edge and the supernode provide a management interface UDP port.
These interfaces have some documentation on their non machine readable
commands in the respective daemon man pages.
Default Ports:
- UDP/5644 - edge
- UDP/5645 - supernode
JSON Reply Management API - feature parity with old management interfaces (#861) * Ensure that recent code additions pass the linter * Include some of the more obviously correct lint fixes to edge_utils.c * Refactor edge JSON api into its own source file * Use shorter names for static management functions * Implement a JSON RPC way of managing the verbosity * Tidy up help display in n2nctl script * Make note of issue with implementing the stop command * Implement a JSON RPC call to fetch current community * Make n2nhttpd time value be more self-contained * Make n2nhttpd order more closely match the existing management stats output * Wire up status page to the verbosity setting * Add JSON versions of the remainder of the edge management stats * Add new file to cmake * Properly define management handler * Only update the last updated timestamp after a successful data fetch * Function and types definition cleanup * Force correct type for python scripts mgmt port * Implement initial JSON API for supernode * Fix whitespace error * Use helper function for rendering peers ip4 address * Proxy the auth requirement back out to the http client, allowing normal http auth to be used * Ensure that we do not leak the federation community * Use the same rpc method name and output for both edge and supernode for peers/edges * Allow n2nctl to show raw data returned without resorting to tricks * Make n2nctl pretty printer understandable with an empty table * Use the full name for supernodes RPC call * Use same RPC method name (but some missing fields) for getting communities from both edge and supernode * Add *_sup_broadcast stats to edge packet stats output * Refacter the stats into a packetstats method for supernode RPC * Even if I am not going to prettyprint the timestamps, at least make all the timestamps on the page the same unit * Simplify the RPC handlers by flagging some as writable and checking that in the multiplexer * Remove invalid edges data * Avoid crash on bad data to verbose RPC * Avoid showing bad or inconsistant protocol data in communities RPC * Minor clarification on when --write is handled * Make linter happy * Fix changed method name in n2nhttpd * Move mainloop stop flag into the n2n_edge_t structure, allowing access from management commands * Implement edge RPC stop command * Move mainloop stop flag into the n2n_sn_t structure, allowing access from management commands * Implement supernode RPC stop command * Allow multiple pages to be served from mini httpd * Extract common script functions into a separate URL * Handle an edge case in the python rpc class With a proper tag-based demultiplexer, this case should be a nop, but we are single-threaded and rely on the packet ordering in this library. * Add n2nhttpd support to query supernode using urls prefixed with /supernode/ * Handle missing values in javascript table print * Add another less filtering javascript key/value renderer * Add a supernode.html page to the n2nhttpd * Address lint issue * Mention the second html page on the Scripts doc * Remove purgable column from supernode edges list - it looks like it is rarely going to be set * Add a simple one-line example command at the top of the API documentation * Acknowledge that this is not the most efficient protocol, but point out that it was not supposed to be * Make it clear that the n2nctl script works for both edge and supernode * Fight with inconsistant github runner results * Turn off the /right/ coverage generator
2021-10-23 07:20:05 +02:00
A Quick start example query:
`echo r 1 help | nc -w1 -u 127.0.0.1 5644`
Cleanup and Documentation for JSON management API (#856) * Reimplement JSON mgmt with clear separation of read/write actions * Reduce boilerplate by using a table driven command definition for json mgmt commands * Port tools to use new json api * Add a basic authentication for json mgmt commands * If a auth key is given, it must match * Add auth key to management scripts * Add a flag bitfield to clearly turn the tag param into a options list * Allow simple pass-through of any command from n2nctl * Convert the n2nctl to use an object oriented interface * Handle sigpipe in the n2nhttpd - this happens if the remote client disconnects unexpectely * Remove some repetition from the server * Use the correct options to allow reuseaddr * Dont generate a scary message on ctrl-c * Convert n2nhttpd to use object based RPC * Use the same longopt for both tools * Pass any extra args through to the RPC * Add some documentation for the scripts in the repository * Spelling fix * Add documentation for the JSON reply mangement API
2021-10-17 22:16:42 +02:00
## JSON Query interface
JSON Reply Management API - feature parity with old management interfaces (#861) * Ensure that recent code additions pass the linter * Include some of the more obviously correct lint fixes to edge_utils.c * Refactor edge JSON api into its own source file * Use shorter names for static management functions * Implement a JSON RPC way of managing the verbosity * Tidy up help display in n2nctl script * Make note of issue with implementing the stop command * Implement a JSON RPC call to fetch current community * Make n2nhttpd time value be more self-contained * Make n2nhttpd order more closely match the existing management stats output * Wire up status page to the verbosity setting * Add JSON versions of the remainder of the edge management stats * Add new file to cmake * Properly define management handler * Only update the last updated timestamp after a successful data fetch * Function and types definition cleanup * Force correct type for python scripts mgmt port * Implement initial JSON API for supernode * Fix whitespace error * Use helper function for rendering peers ip4 address * Proxy the auth requirement back out to the http client, allowing normal http auth to be used * Ensure that we do not leak the federation community * Use the same rpc method name and output for both edge and supernode for peers/edges * Allow n2nctl to show raw data returned without resorting to tricks * Make n2nctl pretty printer understandable with an empty table * Use the full name for supernodes RPC call * Use same RPC method name (but some missing fields) for getting communities from both edge and supernode * Add *_sup_broadcast stats to edge packet stats output * Refacter the stats into a packetstats method for supernode RPC * Even if I am not going to prettyprint the timestamps, at least make all the timestamps on the page the same unit * Simplify the RPC handlers by flagging some as writable and checking that in the multiplexer * Remove invalid edges data * Avoid crash on bad data to verbose RPC * Avoid showing bad or inconsistant protocol data in communities RPC * Minor clarification on when --write is handled * Make linter happy * Fix changed method name in n2nhttpd * Move mainloop stop flag into the n2n_edge_t structure, allowing access from management commands * Implement edge RPC stop command * Move mainloop stop flag into the n2n_sn_t structure, allowing access from management commands * Implement supernode RPC stop command * Allow multiple pages to be served from mini httpd * Extract common script functions into a separate URL * Handle an edge case in the python rpc class With a proper tag-based demultiplexer, this case should be a nop, but we are single-threaded and rely on the packet ordering in this library. * Add n2nhttpd support to query supernode using urls prefixed with /supernode/ * Handle missing values in javascript table print * Add another less filtering javascript key/value renderer * Add a supernode.html page to the n2nhttpd * Address lint issue * Mention the second html page on the Scripts doc * Remove purgable column from supernode edges list - it looks like it is rarely going to be set * Add a simple one-line example command at the top of the API documentation * Acknowledge that this is not the most efficient protocol, but point out that it was not supposed to be * Make it clear that the n2nctl script works for both edge and supernode * Fight with inconsistant github runner results * Turn off the /right/ coverage generator
2021-10-23 07:20:05 +02:00
A machine readable API is available for both the edge and supernode. It
takes a simple text request and replies with JSON formatted data.
Cleanup and Documentation for JSON management API (#856) * Reimplement JSON mgmt with clear separation of read/write actions * Reduce boilerplate by using a table driven command definition for json mgmt commands * Port tools to use new json api * Add a basic authentication for json mgmt commands * If a auth key is given, it must match * Add auth key to management scripts * Add a flag bitfield to clearly turn the tag param into a options list * Allow simple pass-through of any command from n2nctl * Convert the n2nctl to use an object oriented interface * Handle sigpipe in the n2nhttpd - this happens if the remote client disconnects unexpectely * Remove some repetition from the server * Use the correct options to allow reuseaddr * Dont generate a scary message on ctrl-c * Convert n2nhttpd to use object based RPC * Use the same longopt for both tools * Pass any extra args through to the RPC * Add some documentation for the scripts in the repository * Spelling fix * Add documentation for the JSON reply mangement API
2021-10-17 22:16:42 +02:00
The request is in simple text so that the daemon does not need to include any
complex parser.
The replies are all in JSON so that the data is fully machine readable and
the schema can be updated as needed - the burden is entirely on the client
to handle different replies with different fields. It is expected that
any client software will be written in higher level programming languages
where this flexibility is easy to provide.
Since the API is over UDP, the replies are structured so that each part of
the reply is clearly tagged as belonging to one request and there is a
clear begin and end marker for the reply. This is expected to support the
future possibilities of pipelined and overlapping transactions as well as
pub/sub asynchronous event channels.
The replies will also handle some small amount of re-ordering of the
packets, but that is not an specific goal of the protocol.
JSON Reply Management API - feature parity with old management interfaces (#861) * Ensure that recent code additions pass the linter * Include some of the more obviously correct lint fixes to edge_utils.c * Refactor edge JSON api into its own source file * Use shorter names for static management functions * Implement a JSON RPC way of managing the verbosity * Tidy up help display in n2nctl script * Make note of issue with implementing the stop command * Implement a JSON RPC call to fetch current community * Make n2nhttpd time value be more self-contained * Make n2nhttpd order more closely match the existing management stats output * Wire up status page to the verbosity setting * Add JSON versions of the remainder of the edge management stats * Add new file to cmake * Properly define management handler * Only update the last updated timestamp after a successful data fetch * Function and types definition cleanup * Force correct type for python scripts mgmt port * Implement initial JSON API for supernode * Fix whitespace error * Use helper function for rendering peers ip4 address * Proxy the auth requirement back out to the http client, allowing normal http auth to be used * Ensure that we do not leak the federation community * Use the same rpc method name and output for both edge and supernode for peers/edges * Allow n2nctl to show raw data returned without resorting to tricks * Make n2nctl pretty printer understandable with an empty table * Use the full name for supernodes RPC call * Use same RPC method name (but some missing fields) for getting communities from both edge and supernode * Add *_sup_broadcast stats to edge packet stats output * Refacter the stats into a packetstats method for supernode RPC * Even if I am not going to prettyprint the timestamps, at least make all the timestamps on the page the same unit * Simplify the RPC handlers by flagging some as writable and checking that in the multiplexer * Remove invalid edges data * Avoid crash on bad data to verbose RPC * Avoid showing bad or inconsistant protocol data in communities RPC * Minor clarification on when --write is handled * Make linter happy * Fix changed method name in n2nhttpd * Move mainloop stop flag into the n2n_edge_t structure, allowing access from management commands * Implement edge RPC stop command * Move mainloop stop flag into the n2n_sn_t structure, allowing access from management commands * Implement supernode RPC stop command * Allow multiple pages to be served from mini httpd * Extract common script functions into a separate URL * Handle an edge case in the python rpc class With a proper tag-based demultiplexer, this case should be a nop, but we are single-threaded and rely on the packet ordering in this library. * Add n2nhttpd support to query supernode using urls prefixed with /supernode/ * Handle missing values in javascript table print * Add another less filtering javascript key/value renderer * Add a supernode.html page to the n2nhttpd * Address lint issue * Mention the second html page on the Scripts doc * Remove purgable column from supernode edges list - it looks like it is rarely going to be set * Add a simple one-line example command at the top of the API documentation * Acknowledge that this is not the most efficient protocol, but point out that it was not supposed to be * Make it clear that the n2nctl script works for both edge and supernode * Fight with inconsistant github runner results * Turn off the /right/ coverage generator
2021-10-23 07:20:05 +02:00
Note that this API will reply with a relatively large number of UDP packets
and that it is not intended for high frequency or high volume data transfer.
It was written to use a low amount of memory and to support incremental
generation of the reply data.
Cleanup and Documentation for JSON management API (#856) * Reimplement JSON mgmt with clear separation of read/write actions * Reduce boilerplate by using a table driven command definition for json mgmt commands * Port tools to use new json api * Add a basic authentication for json mgmt commands * If a auth key is given, it must match * Add auth key to management scripts * Add a flag bitfield to clearly turn the tag param into a options list * Allow simple pass-through of any command from n2nctl * Convert the n2nctl to use an object oriented interface * Handle sigpipe in the n2nhttpd - this happens if the remote client disconnects unexpectely * Remove some repetition from the server * Use the correct options to allow reuseaddr * Dont generate a scary message on ctrl-c * Convert n2nhttpd to use object based RPC * Use the same longopt for both tools * Pass any extra args through to the RPC * Add some documentation for the scripts in the repository * Spelling fix * Add documentation for the JSON reply mangement API
2021-10-17 22:16:42 +02:00
With a small amount of effort, the API is intended to be human readable,
but this is intended for debugging.
## Request API
The request is a single UDP packet containing one line of text with at least
three space separated fields. Any text after the third field is available for
the API method to use for additional parameters
Fields:
- Message Type
- Options
- Method
- Optional Additional Parameters
The maximum length of the entire line of text is 80 octets.
### Message Type
This is a single octet that is either "r" for a read (or query) method
call or "w" for a write (or change) method call.
To simplify the interface, the reply from both read and write calls to the
same method is expected to contain the same data. In the case of a write
call, the reply will contain the new state after making the requested change.
### Options
The options field is a colon separated set of options for this request. Only
the first subfield (the "tag") is mandatory. The second subfield is a set
of flags that describe which optional subfields are present.
If there are no additional subfields then the flags can be omitted.
SubFields:
- Message Tag
- Optional Message Flags (defaults to 0)
- Optional Authentication Key
#### Message Tag
Each request provides a tag value. Any non error reply associated with this
request will include this tag value, allowing all related messages to be
collected within the client.
Where possible, the error replies will also include this tag, however some
errors occur before the tag is parsed.
The tag is not interpreted by the daemon, it is simply echoed back in all
the replies. It is expected to be a short string that the client chooses
to be unique amongst all recent or still outstanding requests.
One possible client implementation is a number between 0 and 999, incremented
for each request and wrapping around to zero when it is greater than 999.
#### Message Flags
This subfield is a set of bit flags that are hex-encoded and describe any
remaining optional subfields.
Currently, only one flag is defined. The presence of that flag indicates
that an authentication key subfield is also present.
Values:
- 0 - No additional subfields are present
- 1 - One additional field, containing the authentication key
#### Authentication Key
A simple string password that is provided by the client to authenticate
this request. See the Authentication section below for more discussion.
#### Example Options value
e.g:
`102:1:PassWord`
### Example Request string
e.g:
`r 103:1:PassWord peer`
## Reply API
Each UDP packet in the reply is a complete and valid JSON dictionary
containing a fragment of information related to the entire reply.
### Common metadata
There are two keys in each dictionary containing metadata. First
is the `_tag`, containing the Message Tag from the original request.
Second is the `_type` whic identifies the expected contents of this
packet.
### `_type: error`
If an error condition occurs, a packet with a `error` key describing
the error will be sent. This usually also indicates that there will
be no more substantial data arriving related to this request.
e.g:
`{"_tag":"107","_type":"error","error":"badauth"}`
### `_type: begin`
Before the start of any substantial data packets, a `begin` packet is
sent. For consistency checking, the method in the request is echoed
back in the `error` key.
e.g:
`{"_tag":"108","_type":"begin","cmd":"peer"}`
For simplicity in decoding, if a `begin` packet is sent, all attempts
are made to ensure that a final `end` packet is also sent.
### `_type: end`
After the last substantial data packet, a final `end` packet is sent
to signal to the client that this reply is finished.
e.g:
`{"_tag":"108","_type":"end"}`
### `_type: row`
The substantial bulk of the data in the reply is contained within one or
more `row` packets. The non metadata contents of each `row` packet is
defined entirely by the method called and may change from version to version.
JSON Reply Management API - feature parity with old management interfaces (#861) * Ensure that recent code additions pass the linter * Include some of the more obviously correct lint fixes to edge_utils.c * Refactor edge JSON api into its own source file * Use shorter names for static management functions * Implement a JSON RPC way of managing the verbosity * Tidy up help display in n2nctl script * Make note of issue with implementing the stop command * Implement a JSON RPC call to fetch current community * Make n2nhttpd time value be more self-contained * Make n2nhttpd order more closely match the existing management stats output * Wire up status page to the verbosity setting * Add JSON versions of the remainder of the edge management stats * Add new file to cmake * Properly define management handler * Only update the last updated timestamp after a successful data fetch * Function and types definition cleanup * Force correct type for python scripts mgmt port * Implement initial JSON API for supernode * Fix whitespace error * Use helper function for rendering peers ip4 address * Proxy the auth requirement back out to the http client, allowing normal http auth to be used * Ensure that we do not leak the federation community * Use the same rpc method name and output for both edge and supernode for peers/edges * Allow n2nctl to show raw data returned without resorting to tricks * Make n2nctl pretty printer understandable with an empty table * Use the full name for supernodes RPC call * Use same RPC method name (but some missing fields) for getting communities from both edge and supernode * Add *_sup_broadcast stats to edge packet stats output * Refacter the stats into a packetstats method for supernode RPC * Even if I am not going to prettyprint the timestamps, at least make all the timestamps on the page the same unit * Simplify the RPC handlers by flagging some as writable and checking that in the multiplexer * Remove invalid edges data * Avoid crash on bad data to verbose RPC * Avoid showing bad or inconsistant protocol data in communities RPC * Minor clarification on when --write is handled * Make linter happy * Fix changed method name in n2nhttpd * Move mainloop stop flag into the n2n_edge_t structure, allowing access from management commands * Implement edge RPC stop command * Move mainloop stop flag into the n2n_sn_t structure, allowing access from management commands * Implement supernode RPC stop command * Allow multiple pages to be served from mini httpd * Extract common script functions into a separate URL * Handle an edge case in the python rpc class With a proper tag-based demultiplexer, this case should be a nop, but we are single-threaded and rely on the packet ordering in this library. * Add n2nhttpd support to query supernode using urls prefixed with /supernode/ * Handle missing values in javascript table print * Add another less filtering javascript key/value renderer * Add a supernode.html page to the n2nhttpd * Address lint issue * Mention the second html page on the Scripts doc * Remove purgable column from supernode edges list - it looks like it is rarely going to be set * Add a simple one-line example command at the top of the API documentation * Acknowledge that this is not the most efficient protocol, but point out that it was not supposed to be * Make it clear that the n2nctl script works for both edge and supernode * Fight with inconsistant github runner results * Turn off the /right/ coverage generator
2021-10-23 07:20:05 +02:00
Each `row` packet contains exactly one complete JSON object. The row replies
may be processed incrementally as each row arrives and no batching of multiple
packets will be required.
Cleanup and Documentation for JSON management API (#856) * Reimplement JSON mgmt with clear separation of read/write actions * Reduce boilerplate by using a table driven command definition for json mgmt commands * Port tools to use new json api * Add a basic authentication for json mgmt commands * If a auth key is given, it must match * Add auth key to management scripts * Add a flag bitfield to clearly turn the tag param into a options list * Allow simple pass-through of any command from n2nctl * Convert the n2nctl to use an object oriented interface * Handle sigpipe in the n2nhttpd - this happens if the remote client disconnects unexpectely * Remove some repetition from the server * Use the correct options to allow reuseaddr * Dont generate a scary message on ctrl-c * Convert n2nhttpd to use object based RPC * Use the same longopt for both tools * Pass any extra args through to the RPC * Add some documentation for the scripts in the repository * Spelling fix * Add documentation for the JSON reply mangement API
2021-10-17 22:16:42 +02:00
e.g:
`{"_tag":"108","_type":"row","mode":"p2p","ip4addr":"10.135.98.84","macaddr":"86:56:21:E4:AA:39","sockaddr":"192.168.7.191:41701","desc":"client4","lastseen":1584682200}`
## Authentication
Some API requests will make global changes to the running daemon and may
affect the availability of the n2n networking. Therefore the machine
readable API include an authentication component.
Currently, the only authentication is a simple password that the client
must provide.
Reference in New Issue Copy Permalink