Commit Graph

217 Commits

Author SHA1 Message Date
Hamish Coleman
860b27d97c Add event publishing for adding and removing P2P peers 2022-02-05 20:06:31 +00:00
Hamish Coleman
a40ed15d93 Add pubsub system to edge, with one example TEST channel 2022-02-05 20:06:31 +00:00
Hamish Coleman
3fd1dc5fa8 keep private enum definition local to the management code 2022-02-05 20:06:31 +00:00
Hamish Coleman
ace3dbc44f Move all supernode management handling into one source file 2022-02-05 20:06:31 +00:00
Hamish Coleman
a8e2704b41 Move all management packet handling for edge into the one source file 2022-02-05 20:06:31 +00:00
Hamish Coleman
4f568b03c1
Autoconf simplification (#927)
* Convert zstd feature to default disable

* All autoconf test use one standard template

To simplify the testing, cross-compilation and repeatable build process,
no configure options are automatically probed for - they all default to
off and are all using the same template.

The --with-x options should be deprecated and replaced with --enable-x
because there is no syntax checking for --with options in autoconf.

There are still some differences between the config options, but this
should provide a starting point.

* Remove unused code from the autoconf

* Remove warnings from default build

* Avoid calling port mapping functions if none are enabled

* Start with all builds in neutral config

* Add more missing code guards

* Adjust code guard location to placate cmake
2022-01-10 19:58:15 +00:00
Logan oos Even
b657881a92
differentiated miniupnp includes (#912) 2021-12-24 10:32:22 +01:00
Logan oos Even
e6e8cb038a
added port forwarding (upnp and natpmp) (#905)
* UPnP port redirection is supported.

* compile fixes

* compile fix

* optimize reconnection code

* prepared upnp threadification to counter main loop stall at supernode change

* NAT-PMP port forwarding support, temporarily merge codes to resolve conflicts.

* make compile fix

* prepared threadification in more detail

* adopted threadification to new file setup

* cleaned up

* renamed functions and data structures

* fixes

* differentiated between miniupnp and natpmp and added corresponding lib support to makefile

* name

* commented unused header includes

* comments

* license

* fixes

* fixes

* fixes

* NAT-PMP is already available.

* added CLI parameter to disable port forwarding if required

* preliminary made use of multithreading

* adjusted log level

* added man page documentation

* def'ed conf

* made pmpnat adjustments

Co-authored-by: fengdaolong <fengdaolong@gmail.com>
2021-12-23 12:27:55 +01:00
Logan oos Even
89571292f4
allowed supernode MAC address to be set manually (-m) (#886) 2021-11-01 17:19:46 +05:45
Logan oos Even
f4e7e513a8
added MAC address based supernode selection (#885) 2021-11-01 16:35:05 +05:45
Logan oos Even
3b187b4ac8
updated JSON API password handling '--management-password <pw>' (#869) 2021-10-24 15:22:36 +05:45
Logan oos Even
c61b62ab1b
choose supernode selection strategy at run-time '--select-rtt' (#864) 2021-10-24 12:53:25 +05:45
Hamish Coleman
80b33cd1a9
Basic C Code lint checker and shell checker (#859)
* Factor build packages out into a more maintainable list

* Create a location for scripts to live

* Provide a make target to return the source dir as close as reasonable to the original distributed state

* Add a code lint step, checking the coding style

* Change test harness as recommended by shellcheck

* Ensure we actually have the linter tool installed

* Use the correct directory for cmake to run the tests

* Adjust for the older uncrustify in the current github ubuntu-latest

* Make one file pass the linter

* Integrate the lint with the existing test workflow

* Add files with minimal changes needed to the linter

* Add more files with minimal changes needed to the linter

* Dont build binaries if we fail the lint test

* Update the phony targets with the lint steps

* Ensure the flake8 package is installed in the new lint workflow job

* Use the makefile to drive the packages needed to install for linting

* No need to add dependancies on lint, just rely on the workflow status to show failure

* Update the scripts dir README to reflect current assumptions

* Rename and briefly document the indent.sh script

* Fix the ignore to ignore the right Makefile

* Rename the test_harness script to make it clear it is a shell script

* Provide a master lint make target and add a shell script lint tool

* Elminate stray tabs

* Drop include/auth.h from linter - there are inconsistant results with function definitions when using the current uncrustify rules
2021-10-24 01:21:18 +05:45
Hamish Coleman
ae502d9181
JSON Reply Management API - feature parity with old management interfaces (#861)
* Ensure that recent code additions pass the linter

* Include some of the more obviously correct lint fixes to edge_utils.c

* Refactor edge JSON api into its own source file

* Use shorter names for static management functions

* Implement a JSON RPC way of managing the verbosity

* Tidy up help display in n2nctl script

* Make note of issue with implementing the stop command

* Implement a JSON RPC call to fetch current community

* Make n2nhttpd time value be more self-contained

* Make n2nhttpd order more closely match the existing management stats output

* Wire up status page to the verbosity setting

* Add JSON versions of the remainder of the edge management stats

* Add new file to cmake

* Properly define management handler

* Only update the last updated timestamp after a successful data fetch

* Function and types definition cleanup

* Force correct type for python scripts mgmt port

* Implement initial JSON API for supernode

* Fix whitespace error

* Use helper function for rendering peers ip4 address

* Proxy the auth requirement back out to the http client, allowing normal http auth to be used

* Ensure that we do not leak the federation community

* Use the same rpc method name and output for both edge and supernode for peers/edges

* Allow n2nctl to show raw data returned without resorting to tricks

* Make n2nctl pretty printer understandable with an empty table

* Use the full name for supernodes RPC call

* Use same RPC method name (but some missing fields) for getting communities from both edge and supernode

* Add *_sup_broadcast stats to edge packet stats output

* Refacter the stats into a packetstats method for supernode RPC

* Even if I am not going to prettyprint the timestamps, at least make all the timestamps on the page the same unit

* Simplify the RPC handlers by flagging some as writable and checking that in the multiplexer

* Remove invalid edges data

* Avoid crash on bad data to verbose RPC

* Avoid showing bad or inconsistant protocol data in communities RPC

* Minor clarification on when --write is handled

* Make linter happy

* Fix changed method name in n2nhttpd

* Move mainloop stop flag into the n2n_edge_t structure, allowing access from management commands

* Implement edge RPC stop command

* Move mainloop stop flag into the n2n_sn_t structure, allowing access from management commands

* Implement supernode RPC stop command

* Allow multiple pages to be served from mini httpd

* Extract common script functions into a separate URL

* Handle an edge case in the python rpc class

With a proper tag-based demultiplexer, this case should be a nop,
but we are single-threaded and rely on the packet ordering in this
library.

* Add n2nhttpd support to query supernode using urls prefixed with /supernode/

* Handle missing values in javascript table print

* Add another less filtering javascript key/value renderer

* Add a supernode.html page to the n2nhttpd

* Address lint issue

* Mention the second html page on the Scripts doc

* Remove purgable column from supernode edges list - it looks like it is rarely going to be set

* Add a simple one-line example command at the top of the API documentation

* Acknowledge that this is not the most efficient protocol, but point out that it was not supposed to be

* Make it clear that the n2nctl script works for both edge and supernode

* Fight with inconsistant github runner results

* Turn off the /right/ coverage generator
2021-10-23 11:05:05 +05:45
Hamish Coleman
e6fcf1c55b
Cleanup and Documentation for JSON management API (#856)
* Reimplement JSON mgmt with clear separation of read/write actions

* Reduce boilerplate by using a table driven command definition for json mgmt commands

* Port tools to use new json api

* Add a basic authentication for json mgmt commands

* If a auth key is given, it must match

* Add auth key to management scripts

* Add a flag bitfield to clearly turn the tag param into a options list

* Allow simple pass-through of any command from n2nctl

* Convert the n2nctl to use an object oriented interface

* Handle sigpipe in the n2nhttpd - this happens if the remote client disconnects unexpectely

* Remove some repetition from the server

* Use the correct options to allow reuseaddr

* Dont generate a scary message on ctrl-c

* Convert n2nhttpd to use object based RPC

* Use the same longopt for both tools

* Pass any extra args through to the RPC

* Add some documentation for the scripts in the repository

* Spelling fix

* Add documentation for the JSON reply mangement API
2021-10-18 02:01:42 +05:45
Hamish Coleman
4438f1aa2a
added mingw test platform (#829)
* Provide a minimal reimplementation of our autoconf, to try windows builds

* Try building with windows

* Fix thinko in spelling

* Ensure shell script runs inside a shell

* Add a hack to aid include discovery

* Just keep adding tech debt...

* Assume that we will have slashes in some of the replacement strings and avoid that char with sed

* Restore one slash

* Hack around the tools makefile interdependancy bug

* A correct cflags include hack for each compile dir

* Ensure we link against winsock (note, even though this says 32bit, it should link the 64bit library ... I think)

* Bad link ordering if we dont use LDLIBS

* Remove unused make variable

* Remove makefile duplication using inheritance (this does mean you can no longer cd tools; make, but must do make tools)

* Add missing library for win32

* Show OS variable

* Make hack autoconf more robust for tests on non gitlab runners

* Remove no longer used substitutions from hack autoconf

* Add missing include path to tools under win32

* Build the win32 subdir when the compiler is Msys

* The different subdirs have different dependancies

* Ensure we can find the include files

* Fix library link ordering

* Ensure the tools dir can find the special win32 lib

* Deal with the differing basic type sizes on both linux/64bit and windows/64bit

* Document the steps to mimic the github windows/mingw build locally - to allow for simpler debugging

* Ensure branch name in instructions matches my test branch name

* Clarify the shell needed to build with mingw

* Since the makefile depends on knowing the OS, raise a fatal error if we cannot determine this

* Handling different compile environments is hard.

- Linux: sane and reasonable results for both uname -s (=Linux) and
  uname -o (=GNU/Linux)
- Windows/Mingw: insane results for uname -s
  (=MSYS_NT-$MAJOR.$MINOR-$BUILDNR) but sane results for uname -o (Msys)
- Macos: sane results for uname -s (=Darwin) but does not support
  uname -o at all

* Revamp the way that Mingw is detected

* Avoid attempting to generate gcovr report when running under windows

* Whoops, isolate the right step

* Fix spelling mistake

* win32/Makefile: Remove unused setting and add comment

* ensure that all win32 includes use the same expected path

* Allow simpler cross compilation by letting configure pass the CC and AR environment through

* Avoid multiple '_CRT_SECURE_NO_WARNINGS redefined' warnings

* Convert to a consolidated CONFIG_TARGET variable to select any different compile options

* Use the more generic printf defines to avoid warnings on mingw

* Update mingw build docs

* English better for reader happy make

* Address a number of mingw compiler warnings

* Fix Visual C compile

* Be sure to document some of the hacky nature of the mingw build
2021-10-06 00:52:15 +05:45
Logan oos Even
db5adc2038
fine-tuned REGISTER cookie handling (#837) 2021-10-03 18:19:29 +05:45
Logan oos Even
bc65a7f59e
added forwarded REGISTER cookie type (#836) 2021-10-03 05:24:33 +05:45
Logan oos Even
ebb357245a
added a flag field to QUERY_PEER (#835) 2021-10-02 23:17:45 +05:45
Logan oos Even
2d34545304
compile fix (#834) 2021-09-30 12:39:31 +05:45
Logan oos Even
16d248d921
fixed MinGW compile (#830) 2021-09-30 02:20:48 +05:45
Hamish Coleman
b735ad6b9e
added test framework and code coverage reporting (#797)
* Add a simple test framework

* Add a code coverage report example oneliner

* Move the coverage report into a separate directory

* Add a github action to run tests and publish a branch with the coverage report

* Fix: Missing job separator

* Fix: remember to actually run configure

* Fix: Gotta autogen before I configure

* Dont try to upload coverage report unless this is a push

* Clearly show the git ref tested in the coverage report

* Add a test for the various transforms

* Add tests for the elliptic curve and pearson hash

* Ensure we ignore new generated output

* Remove unneeded boilerplate from the compression tests

* Add an example of a test of the encoded wire packets

* Ensure that correctly testable data is output even when zstd is not compiled

* Factor test runner out into its own script and attempt to add it to the cmake file

* Tell cmake about a new object file

* Stop trying to make Cmake work...

* Stop trying to make cmake work, round 2

* In the middle of a thousand lines of cmake output was one important one - windows could not find assert() - try again

* Try again to plumb the tests into cmake

* Add missing library to our superset install line

* Fix build error when libcap-dev is installed

* Switch to using artifact uploads instead of pages to store/show the coverage report

* Fix artifact upload yaml

* Upload coverage report to codecov

* Fix codecov - clearly it doesnt do a recursive search for coverage files

* Fix codecov - my hopeful use of a list of directories didnt work

* Fix codecov - unfortunately, it doesnt just consume the coverage data and needs us to generate the gcov output

* Fix codecov - nope, it still doesnt recursively search

* Fix codecov - it really helps if I run the gcov data generator

* Add a simple matrix build

* Fix older ubuntu versions of gcovr that do not support the '--html-title' option

* Ensure we use gcover options that are identical on older ubuntu

* Improve coverage generation and required build packages
2021-09-27 15:26:06 +05:45
Logan oos Even
5a93fdf166
created space for more REGISTER type cookies
... to allow better differentiation between several (future) REGISTER / ACK transmission paths
2021-09-27 02:29:29 +05:45
Logan oos Even
dd41689bb9
added supernode version and uptime information to edge management port output (#796) 2021-09-17 02:34:09 +05:45
Hamish Coleman
370ea9bd37
addressed all but one of the remaining compile warnings (#793)
* Fix the mismatched char, uint8_t warnings

* Ensure we leave room for zero termination and then ensure the string is terminated

* GCC has a warning if it thinks your could overflow your snprintf buffer, but dont check for said overflow
2021-09-16 14:45:46 +05:45
Logan oos Even
9c22d28562
relocated 'detect_local_ip_address()' (#791) 2021-09-16 12:20:35 +05:45
Hamish Coleman
6b353141b4
addressed warnings (#790) 2021-09-15 22:08:45 +05:45
Logan oos Even
b3bf36f867
separated 'detect_local_ip_address()' function (#781) 2021-09-13 11:33:27 +05:45
Logan oos Even
7f1fe9a499
allowed edge to optionally auto-detetct local IP address ('-e auto') for advertising as preferred (#776) 2021-09-12 00:22:05 +05:45
Logan oos Even
7fa0cc1e0a
added preferred socket handling (-e) (#765) 2021-09-09 14:07:29 +05:45
Logan oos Even
b6e865aac7
trimmed open_socket() down by one parameter (#768) 2021-09-06 17:03:15 +05:45
Logan oos Even
e13c9a01a5
rearranged thread-related code (#761) 2021-08-30 14:23:31 +05:45
Logan oos Even
c21accc2a1
enabled edge to bind to a specified IP address (-b) (#753) 2021-08-28 03:08:23 +05:45
Logan oos Even
e09f3a4875
added federation support to user/password auth scheme (#741) 2021-07-31 22:17:07 +05:45
skyformat99
6f63f401fe
updated for compilation with VS2015 (#739) 2021-07-26 12:53:05 +05:45
Logan oos Even
c35129b8e2
added 'reload_communities' command to supernode management port (#740) 2021-07-24 03:46:50 +05:45
Logan oos Even
84d0991977
fortified user/pw auth scheme (#731) 2021-07-21 02:52:38 +05:45
Logan oos Even
39aa50b4cd
added manual MAC/IP address spoofing protection override at supernode (-M) (#716) 2021-06-21 17:03:07 +05:45
Logan oos Even
1586ae62a7
made n2n react more swiftly to (D)DNS changes (#705) 2021-06-07 17:57:49 +05:45
Logan oos Even
3c6ab56d3f
allowed (temporarily) initially non-resolving supernode names (#701) 2021-06-05 17:54:32 +05:45
Logan oos Even
c55e3e3246
multithreaded supernode name resolution (#645) 2021-06-04 03:52:12 +05:45
Logan oos Even
a28327a0af
added user-password edge authentication (#673) 2021-06-04 03:11:22 +05:45
Logan oos Even
667b7df083
added memrnd() (#685) 2021-04-19 00:47:37 +05:45
randomize00
cfa92d28d8
add callback function for hin2n to protect socket (#684) 2021-04-14 11:31:50 +05:45
Sven Roederer
23c18e349f
drop non portable SOL_TCP / fix MacOS v2 (#680)
* Revert "Fix compilation on MacOS"

in order to another solution

* src: use IPPROTO_TCP in place of SOL_TCP

* use the more specific IPPROTO_TCP type for setsockopt
* fixes build on MacOS

src/edge_utils.c:891:31: error: use of undeclared identifier 'SOL_TCP'; did you mean 'FPP_TCP'?
        setsockopt(eee->sock, SOL_TCP, TCP_NODELAY, &value, sizeof(value));

src/edge_utils.c:894:31: error: use of undeclared identifier 'SOL_TCP'
        setsockopt(eee->sock, SOL_TCP, TCP_CORK, &value, sizeof(value));
                              ^
src/edge_utils.c:894:40: error: use of undeclared identifier 'TCP_CORK'
        setsockopt(eee->sock, SOL_TCP, TCP_CORK, &value, sizeof(value));

* include/n2n.h: drop redefiniton of SOL_TCP for WIN32, as we don't use it anywhere now

* src: only use TCP_CORK on LINUX

* it's LINUX only and fails on Win32 and MacOS (man 7 tcp)
2021-04-05 19:30:53 +02:00
Alfredo Cardigliano
53e930db10 Fix compilation on MacOS 2021-03-29 09:28:13 +02:00
Logan oos Even
7c3951a10f
revised tcp implementation (#674) 2021-03-26 20:45:31 +05:45
Logan oos Even
0287837241
skewed supernode selection for the benefit of current supernode in no-pong case (#663) 2021-03-16 23:24:39 +05:45
Logan oos Even
aa1f4311f9
adapted packet format (#662)
* adapted packet format
2021-03-16 17:24:59 +05:45
Logan oos Even
23a9c871f8
increased resilience to supernode failure (#660) 2021-03-09 04:46:42 +05:45