# sbws.core package

## Submodules

### sbws.core.cleanup module

Util functions to cleanup disk space.

`sbws.core.cleanup.``gen_parser`(sub)

  Helper function for the broader argument parser generating code that adds in all the possible command line arguments for the cleanup command.

  Parameters

    sub (argparse._SubParsersAction) – what to add a sub-parser to

`sbws.core.cleanup.``main`(args, conf)

  Main entry point in to the cleanup command.

  Parameters

    - args (argparse.Namespace) – command line arguments

    - conf (configparser.ConfigParser) – parsed config files

### sbws.core.generate module

`sbws.core.generate.``gen_parser`(sub)

`sbws.core.generate.``main`(args, conf)

### sbws.core.scanner module

Measure the relays.

`sbws.core.scanner.``create_path_relay`(relay, dest, rl, cb, relay_as_entry=True)

`sbws.core.scanner.``dispatch_worker_thread`(*a, **kw)

`sbws.core.scanner.``dumpstacks`()

`sbws.core.scanner.``error_no_circuit`(circ_fps, nicknames, reason, relay, dest, our_nick)

`sbws.core.scanner.``error_no_helper`(relay, dest, our_nick='')

`sbws.core.scanner.``force_get_results`(pending_results)

  Try to get either the result or an exception, which gets logged.

  It is call by `wait_for_results()` when the time waiting for the results was long.

  To get either the `Result` or an exception, call `get()` with timeout. Timeout is low since we already waited.

  `get` is not call before, because it blocks and the callbacks are not call.

`sbws.core.scanner.``gen_parser`(sub)

`sbws.core.scanner.``get_random_range_string`(content_length, size)

  Return a random range of bytes of length size. content_length is the size of the file we will be requesting a range of bytes from.

  For example, for content_length of 100 and size 10, this function will return one of the following: ‘0-9’, ‘1-10’, ‘2-11’, […] ‘89-98’, ‘90-99’

`sbws.core.scanner.``main`(args, conf)

`sbws.core.scanner.``main_loop`(args, conf, controller, relay_list, circuit_builder, result_dump, relay_prioritizer, destinations, pool)

  Starts and reuse the threads that measure the relays forever.

  It starts a loop that will be run while there is not and event signaling that sbws is stopping (because of SIGTERM or SIGINT).

  Then, it starts a second loop with an ordered list (generator) of relays to measure that might a subset of all the current relays in the Network.

  For every relay, it starts a new thread which runs `measure_relay` to measure the relay until there are `max_pending_results` threads. After that, it will reuse a thread that has finished for every relay to measure. It is the the pool method `apply_async` which starts or reuse a thread. This method returns an `ApplyResult` immediately, which has a `ready` methods that tells whether the thread has finished or not.

  When the thread finish, ie. `ApplyResult` is `ready`, it triggers `result_putter` callback, which put the `Result` in `ResultDump` queue and complete immediately.

  `ResultDump` thread (started before and out of this function) will get the `Result` from the queue and write it to disk, so this doesn’t block the measurement threads.

  If there was an exception not caught by `measure_relay`, it will call instead `result_putter_error`, which logs the error and complete immediately.

  Before the outer loop iterates, it waits (non blocking) that all the `Results` are ready calling `wait_for_results`. This avoid to start measuring the same relay which might still being measured.

`sbws.core.scanner.``measure_bandwidth_to_server`(session, conf, dest, content_length)
  Returns tuple

    results or None if the if the measurement fail. None or exception if the measurement fail.

`sbws.core.scanner.``measure_relay`(args, conf, destinations, cb, rl, relay)

  Select a Web server, a relay to build the circuit, build the circuit and measure the bandwidth of the given relay.

  Return Result

    a measurement Result object

`sbws.core.scanner.``measure_rtt_to_server`(session, conf, dest, content_length)

  Make multiple end-to-end RTT measurements by making small HTTP requests over a circuit + stream that should already exist, persist, and not need rebuilding. If something goes wrong and not all of the RTT measurements can be made, return None. Otherwise return a list of the RTTs (in seconds).

  Returns tuple

    results or None if the if the measurement fail. None or exception if the measurement fail.

`sbws.core.scanner.``result_putter`(result_dump)

  Create a function that takes a single argument – the measurement result – and return that function so it can be used by someone else

`sbws.core.scanner.``result_putter_error`(target)

  Create a function that takes a single argument – an error from a measurement – and return that function so it can be used by someone else

`sbws.core.scanner.``run_speedtest`(args, conf)

  Initializes all the data and threads needed to measure the relays.

  It launches or connect to Tor in a thread. It initializes the list of relays seen in the Tor network. It starts a thread to read the previous measurements and wait for new measurements to write them to the disk. It initializes a class that will be used to order the relays depending on their measurements age. It initializes the list of destinations that will be used for the measurements. It initializes the thread pool that will launch the measurement threads. The pool starts 3 other threads that are not the measurement (worker) threads. Finally, it calls the function that will manage the measurement threads.

`sbws.core.scanner.``stop_threads`(signal, frame, exit_code=0)

`sbws.core.scanner.``timed_recv_from_server`(session, dest, byte_range)

  Request the byte_range from the URL at dest. If successful, return True and the time it took to download. Otherwise return False and an exception.

`sbws.core.scanner.``wait_for_results`(num_relays_to_measure, pending_results)

  Wait for the pool to finish and log progress.

  While there are relays being measured, just log the progress and sleep `TIMEOUT_MEASUREMENTS` (3mins), which is approximately the time it can take to measure a relay in the worst case.

  When there has not been any relay measured in `TIMEOUT_MEASUREMENTS` and there are still relays pending to be measured, it means there is no progress and call `force_get_results()`.

  This can happen in the case of a bug that makes either `measure_relay()`, `result_putter()` (callback) and/or `result_putter_error()` (callback error) stall.

  Note

  in a future refactor, this could be simpler by:

  1. Initializing the pool at the begingging of each loop

  2. Callling `close()`; `join()` after `apply_async()`, to ensure no new jobs are added until the pool has finished with all the ones in the queue.

  As currently, there would be still two cases when the pool could stall:

  1. There’s an exception in `measure_relay` and another in `callback_err`

  2. There’s an exception `callback`.

  This could also be simpler by not having callback and callback error in `apply_async` and instead just calling callback with the `pending_results`.

  (callback could be also simpler by not having a thread and queue and just storing to disk, since the time to write to disk is way smaller than the time to request over the network.)

### sbws.core.stats module

`sbws.core.stats.``gen_parser`(sub)

  Helper function for the broader argument parser generating code that adds in all the possible command line arguments for the stats command.

  Parameters

    sub (argparse._SubParsersAction) – what to add a sub-parser to

`sbws.core.stats.``main`(args, conf)

  Main entry point into the stats command.

  Parameters

    - args (argparse.Namespace) – command line arguments

    - conf (configparser.ConfigParser) – parsed config files

`sbws.core.stats.``print_stats`(args, data)

  Called from main to print various statistics about the organized data to stdout.

  Parameters

    - args (argparse.Namespace) – command line arguments

    - data (dict) – keyed by relay fingerprint, and with values of `sbws.lib.resultdump.Result` subclasses

## Module contents