This release was focused largely on documentation updates for the Go library & CLI. Additionally some minor performance improvements & refactoring updates were made to cleanup the overall codebase. Importantly, the Mobsql dependency was updated to v0.9.0 which brings new parameters which can be used in Mobility Database GTFS data ingest (via RTDatabase / or database CLI subcommand) to bring more up-to-date GTFS data into the system. Note, the database schema has changed and its recommended to clear cache between version updates (see (5)). See below for a detailed list of changes for this release: 1. Library & CLI Documentation The major update in this release is the rework of both the user-facing CLI documentation and Go library documentation. For the Go library side of things: all public types, structs, and functions in the top level git.sr.ht/~mil/mobroute package now have proper docstrings. And additionally example usages have been added to the `examples/` folder. For the CLI, the userguide has been split from the CLI doc. Wherein the former provides walkthrough style examples and the latter provides the full documentation as you get from the `-h` flag for each subcommand on the `mobroute` CLI. Additionally, important to note: a new documentation website for the larger Mobroute project is available and Mobroute's documentation is published to this site at: https://mr.lrdu.org/mobroute Go documentation is available at: - Go doc: https://pkg.go.dev/git.sr.ht/~mil/mobroute?tab=versions - Go examples: https://git.sr.ht/~mil/mobroute/tree/master/item/examples And updated CLI documentation is available at: - CLI userguide: https://mr.lrdu.org/mobroute/doc/userguide_cli/ - CLI doc: https://mr.lrdu.org/mobroute/doc/doc_cli/ 2. Performance Improvement: Use OID instead of ConnID for Connections Previously for referencing connections in the routing logic after CSA, the generated 'ConnID' field was used. This caused issues when a large number of connections (or origin GTFS stop_times) were present. Use of ConnID has been replaced by using SQLite's builtin OID which points to rows individually. There is a moderate performance improvement with this update (which is particularly noticeable in the case of routing with larger feeds with excessive sized stop_times). 3. Consistently refer to StopUID rather then Stop{IDU,SIDU,SSIDU} The terminologies StopIDU, StopSIDU, and StopSSIDU were previously used throughout the routing Go & SQL logic in order to reference stop unique identifiers and their origin parent stations. This invented terminology was confusing and rather now we simply use the new terminology StopUID (short for stop unique identifier) which uniquely maps a stop ID and feed ID combination. And in the case a parent stop is used, StopParentUID is referenced. This cleanup also allows us to properly refer to 'StopUID' in the RTSchedule function. 4. Removal of Oneshot Library Functionality The library formerly offered two usage variants: Oneshot & Runtime usage functionality. Wherein the former allowed for 'one-stage' runtime initialization, load, compute, and {routing,stops,schedule} functionality batched into a single function call. While seemingly convenient, this added complexity as there are configurable params at each stage (for runtime initialization, load, compute, and routing) and encapsulating all these params into a single function just complicated maintenance. As such, oneshot functionality has been completely removed and runtime / stepwise functionality is the defacto recommended and only supported usage pattern. See the updated Go docs and examples for more information on library usage. 5. Update to Mobsql v0.9.0 & Passthrough for Update / Direct Agency URL Support As per usual the minor version tagged with this Mobroute release (v0.9.0) is correlated with the minor release for Mobsql (v0.9.0). The included update to v0.9.0 on Mobsql brings an update to the `RTDatabase` function & `database` CLI subcommand on Mobroute to allow two new params for the `loadmdbgtfs` op. The `MDBGTFSUpdate` parameter allows for ignoring caching for the Mobility Database CSV & GTFS data & enforces explicitly updating the Mobility Database to ensure the latest GTFS data is pulled per MDB. And the `MDBGTFSDirectURLs` parameter allows the user to specify that *direct* agency URLs should be used in GTFS data ingest (wherein by default cloud CI-generated bucket mirrored Mobility Database URLs are used - which may occasionally get out of date). These two new params allow end-users to more reliably get up-to-date GTFS data loaded. 5. Schema Updates A few small schema updates have been applied for naming & code structure consistency. It's recommended to clear your database & cache (rm `~/.cache/mobroute`) before updating from v0.8.* to v0.9.0. Apologies for the manual user-intervention; long-term I am considering investing in a more standard SQL migration process.
Minor patch (v0.8.4) release published as a fixup to v0.8.3 which was indaventerantly force overwritten. This patch contains retract directive for v0.8.3 which should not be used.
Minor patch release includes changes: - Allow glob searching feed id / stop id for RTStops - Add optimizations to routeparams input; implement feedsclean optimization - Handle case for NaN in dist haversine formatting calculation - Store optimized feeds array in output for diagnostics
Minor patch release to address a few small perf issues: - Make LoadStopToParent test ordering independent / sort in unit test - Add support for glob in RTStops - Added missing index for stops related to stopwalks query - Cap _vcaltoservice to static 2024-2028 year range - Add index for _ctcaltoservice on (feed_id, service_id, service_date) - Add nLimit param to RTStops - Bump mobsql to 0.8.3 (fixes indexes for stopwalks / agency)
Minor patch release with 2 small updates: - Bumps mobsql dep to v0.8.2 (which addresses HH:MM:SS / H:MM:SS bug) - Log connection path / stop details when CSA alg broken error occurs
This release of Mobroute implements some large scale performance optimizations for computed tables (generated transfers, calendar to services, and connections) in addition to adding support for customizable logging. Load & compute speed is dramatically improved for adding new GTFS feeds. The Mobsql dependency for this release is bumped to 0.8.1. See below for a detailed list of changes: 1. Transfers Computed Table RTree Perf Optimization The SQL view logic (_vtransfersgen) which is responsible for creating the computed/generated transfers table has been wholesale reworked to optimize performance. The view logic for computed transfers is essentially a cross-comparison between all stops (per GTFS feed) to find stops with a haversine distance with less then a 3km threshold to consider these transfers valid for inter-stop transfers. Calculating this view had previously been implemented using a naive self join (using a rough distance 3 km heuristic) and then computing the haversine distance between matched stops. This logic has been reworked using SQLite's RTree extension (and utilizing the new init/deinit computed table functionality in Mobsql). RTree allows for hierarchical organization of the bounding boxes for stops and has much better performance characteristics then the former self join implementation logic. The resulting set of stops distances for transfers is identical to the old logic as verified by unit tests; and the net performance gain is conservatively a 2-3x speedup. Note speed scale was formerly quadratic so you may see even greater performance improvements. 2. Calendar to Services Computed Table Perf Optimization The SQL view logic (_vcaltoservice) responsible for creating the calendar to services mapping computed table has been wholesale reworked to optimize performance. This view allows mapping a single YYYYMMDD day for a GTFS feed to its associated valid service ids for that day (which is an integral dependency for routing connections extraction prior to running the core routing algorithm). This view had previously been implemented with a simple and naive group by per feed id / service id / day which exhibited very bad performance characteristics as the size of the GTFS calendar and calendar_dates tables grew. The newly reworked logic uses plain joins (no group by) and has vastly improved performance characteristics as the size of the calendar & calendar_dates tables grow. Along with (1), these 2 performance optimizations (of _vtransfersgen and _vcaltoservice) contribute to a significantly decreased compute speed time for the grand majority of feeds. 3. Generated Connection Computed Table Perf Optimization While not as significant as the performance optimizations mentioned for (1) and (2); a smaller performance optimization was applied to the SQL view logic (_vconn) responsible for creating the connections computed table. The update applied is that connections are no longer ordered by departure time in the view itself (since this order *needs* to be applied at the time of extraction / during a routing library call in anycase since timezone logic is applied at this step and will vary per service ID). A moderate increase in performance is gained by not ordering by departure time in computed table calculation as such. 4. Ability to Set Customizable Logger in Runtime Initialization In initializing a Mobroute runtime, the user may now specify a custom logger (log.Logger) to override the default logger (which prints logs to STDERR). This allows downstream consumers & applications to customize where to send logs for load & routing calls and provides better debugging transparency.
This release of Mobroute was focused on tooling updates to add new stops, schedule, and database library functions, addition of custom GTFS loading support, and removal of GTFS feed autoload / ingestion functionality for runtime library functions. See below for detailed notes on changes: 1. Addition of RTStops Library Function A new library function RTStops has been added which allows the enduser to query stops for a given feed based on an input coordinate and maximum kilometer threshold cutoff. If the input coordinate and maximum km threshold are unspecified, this function can be used to dump a list of all stops for the given feed IDs. Similar functionality was present in early versions of Mobroute however was cut to facilitate large scale refactoring. Adding this functionality back enables transparency into the underlying GTFS data loaded for each feed; and further the stop IDs returned may be directly used with the new relatedly added RTSchedule library function (see next section). See the new CLI subcommand `mobroute stops` for testing details; further documentation for RTStops in library usage is in the works. 2. Addition of RTSchedule Library Function A new library function RTSchedule has been added which allows the enduser to query departure times for a given feed ID & stop. This allows the user to get a quick overview of departures and increases the general utility of the library as a whole. The schedule library function reuses timetable logic from the routing algorithm / CSA and essentially the schedule library function queries 'connections' in CSA's terminology directly. Similar functionality was present in early versions of Mobroute but was cut to facilitate large scale refactoring. Now that the CSA schedule SQL logic has been stabilized, adding this functionality in addition to the RTStops library function back should enable more general-purpose usecases for the Mobroute library besides just routing. See the new CLI subcommand `mobroute schedule` for testing details; further documentation for RTStops in library usage is in the works. 3. Addition of RTDatabase & RTDatabaseq Library Functions Two new library functions to interface with Mobsql (Mobroute's underlying GTFS SQLite loading & ETL interface) have been added. The new RTDatabase function allows for manipulating (loading, purging, updating) GTFS data in the SQLite database and RTDatabaseq allows for querying said GTFS data in the SQLite database respectively. These functions are essentially a 'raw' passthrough to Mobsql functionality but was important to add as (1) this way a 'single' database connection can be used for both database load & routing/stops/schedule calls and (2) by adding this passthrough, endusers/consumers of Mobroute's library only have to interface with the Mobroute and not Mobsql directly. See the new CLI subcommands `mobroute database` and `mobroute databaseq` for testing details; further documentation for RTDatabase and RTDatabaseq in library usage is in the works. 4. Custom GTFS Loading Support (via RTDatabase) Custom GTFS archives load support has been added via the newly added support in Mobsql. This functionality is exposed through the new RTDatabase library function and may be used with the 'loadcustomgtfs' op allowing the user to pass a negative feed ID to be associated with a user-specified GTFS zip archive (file:// or http:// path) & loaded to the database and for routing, stops, and schedule functionality. See the CLI documentation for the `mobroute database` subcommand for more details. 5. Library Usage Pattern Update: Runtime GTFS Autoload Removal In prior releases runtime library usage using the RTRoute function and similar would automatically 'load & compute' the necessary data for the request *upon use*. In this release, the pattern for runtime library functions (RTRoute, RTStops, RTSchedule, etc.) usage has been updated such that GTFS archives are no longer 'automatically' loaded. Rather the new runtime library pattern usage is now: 1) initialize the Mobroute Runtime, 2) use the new RTDatabase function to load the relevant GTFS data, 3) use the RTDatabase function to compute the routing-optimized tables from the GTFS data, and then 4) run the RTRoute or GTFS-dependent other function with the loaded feed ID(s). You can see the Go documentation for more details on each of these steps. If you are just interested in one-off / oneshot library routing, stops, database, or schedule requests (with automatic/implicit GTFS loading / ingestion), you may still use the Oneshot* functions for all-in-one functionality. E.g. the Oneshot functions still invoke the old pattern of 'automatic' GTFS ingestion and the CLI remains the same with this pattern and serves as the defacto reference implementation for oneshot library usage.
This release of Mobroute was focused on tooling updates in order to: make the library & CLI more friendly to end users; and to aide in the initial general public release (on F-Droid) for Transito, which is Mobroute's associated mobile app and reference implementation for runtime library usage. See below for a detailed list of changes. 1. Library: Public Interface Cleanup - Oneshot & Runtime Functions The public interface for the Mobroute library has been wholesale reworked with an eye toward external consumer usage. The largest change is that the library's API now presents both 'oneshot' and 'runtime' function variants. The defacto reference implementation for usage of Mobroute's library 'oneshot' functions is Mobroute's CLI located in the source `cli/` folder. And the defacto reference implementation for Mobroute's library 'runtime' functions is the Transito mobile application which pools a single Mobroute runtime (and database connection) for the entire app usage. The Runtime library functions (as utilized by Transito), allow a *single database connection* & runtime to be persisted between multiple library function calls. The user first creates a MobrouteRuntime (via RTInitialize) and then the user is able to perform multiple routing requests using the same runtime and database connection using the RTRoute function. While RTRoute is currently the only runtime function available apart from RTInitialize (to initialize the runtime); this pattern of using a single runtime/database connection sets things architecturally in a good place for future updates & planned functionality (such as querying stoptimes, nearby stops, and similar). See the Transito app for an example of Mobroute library's runtime functions usage. By convention all runtime library functions are and will be prefixed with 'RT'. Meanwhile, Oneshot library functions (as utilized by the Mobroute CLI), offer functions callable in a single (or 'oneshot') request. Unlike the runtime library functions, no prior setup is needed for oneshot functions and this has the benefit that the entire request (for routing requests and similar) is serializable as a single JSON object. The database connection is opened at the start of the function call and closed upon returning with oneshot functions. Do note that internally the new oneshot library functions (OneshotRoute & OneshotMobsql) use the runtime API under-the-hood and the oneshot API is largely a convenience for end-users who want to fire off a routing request with a single call and is the simpler option for integration for one-off use-cases. See the CLI for an example of Mobroute library's' oneshot function usage. By convention all oneshot library functions are and will be prefixed with 'Oneshot'. 2. Library: Initial Documentation Initial documentation for the Mobroute library has been added in the form of godoc strings and the overall presentation of the documentation has been improved if you view within godoc, pkg.go.dev, and similar. More extensive documentation is slated for a later release (see roadmap), however for basic usage of the new runtime & oneshot library functions, the initial documentation (and reference implementations of the Mobroute CLI & Transito app) added in this release should suffice. 3. CLI: Functionality & Documentation A number of large changes have been made to the Mobroute CLI. Among these changes are: (1) the CLI has been reworked to use the newly updated Oneshot Mobroute library functions, (2) the CLI documentation and functionality for both the route and passthrough Mobsql functionality has been reworked to be more friendly to endusers wrt. JSON object formatting and docstrings in -h help messages (3) YAML functionality has been removed and all requests are expected to be formatted as JSON, (4) Feed IDs are used directly instead of the old 'filter format' specification which just added complexity, and finally (5) error handling is vastly improved due to both code restructuring & the addition of route request validation (see next section). 4. Routing: Route Request Validations Validations have been added for route requests by way of utilizing struct tags via the github.com/go-playground/validator library which has been added as a dependency. This library allows for validations such as checking the passed transfer categories, output formats, feed IDs, tunable walk parameter, and similar simply by struct tag annotation. Both CLI & library calls for routing are validated upon usage. As a result, the user can now expect much friendlier & more structured error messages in the case of failed route request due to invalid route parameters. 5. Routing: Embedding of Route Request in GeoJSON Response The GeoJSON output formatter has been modified to include the original route request serialized as a JSON string embedded in the output route LineString property 'metadata' field. This was done largely for convenience & debugging in that the http://geojson.lrdu.org service which provides links for the map view output format now automatically passes through the 'original request'. This has the benefit that the map URLs (which are easily shareable) are now much easier to debug & replicate in that all that is needed to 'recreate' a map URL route is simply to run the mobroute CLI with the JSON buffer included in the map link as in: `mobroute route -rp '{}'` wherein `{}` is the JSON request. 6. Tooling: MDBID & Source References Renamed to FeedID All references to 'MDBID' and 'source' in the codebase have been replaced with the term 'FeedID'. The previously used MDBID and source terms were used interchangeably and created ambiguity as to what the difference between these two terms were. Replacing all references should make it clearer both from the database & in the Go code what exactly is being referred to. As well the term 'FeedID' should be more self-explanatory to end users then either of the former terms. This update matches a similar naming update for FeedID in Mobsql 0.6. 7. Tooling: Go SQLite Driver Portability / Simplification Since initial versions of Mobroute, there has been a hard dependency on the SQLite driver library mattn/go-sqlite3. A feature specific to this library called 'Go SQlite3 Extensions' has been historically utilized to call Go code directly from SQLite queries in the core of the routing API. This provided a good amount of convenience in that haversine distance, timezone lookup, and similar functionality didn't have to be directly implemented in SQL. However, in this release the reliance on all 'Go SQLite3 Extensions' functionality has been deliberately removed. There are a number of reasons this was done; the primary driving reason has to do with debugging. Without using these bindings, all queries executed & printed to the console can be simply replicated just by dumping the exact query in the sqlite CLI with the parameterized args. This offers a much better developer UX for debugging for both code & live debugging scenarios. Additionally, without relying on the Go/SQLite interop extension pattern specific to this library, Mobroute's SQLite driver could be very simply be switched out in the future for another Go SQLite driver. This has benefit in both testing scenarios (e.g. in the case of a bug with the mattn/go-sqlite3 library) and also opens the door to potential for future optimization wrt. the choice of which Go SQLite driver is used.
This release of Mobroute was focused on improving overall compatibility of routing functionality to work with a larger number of GTFS feeds. Many edgecase scenarios have been corrected; and given the correct provided routing parameters & specification-conforming GTFS data, users can now expect routing to work reliably in the vast majority of cases. In addition, a large number of unit test cases have been added to subcomponents of the routing system. Details on changes are below: 1. Transfer Categories System Rework & Performance Improvements The way transfers are handled, particularly for implicit transfers and generated transfers, has been vastly reworked. - Three distinct transfer categories can now be toggled: feed, implicit, and generated transfers. These three categories can be used in any combination. The defaults are to use feed and implicit transfers. The user should manually enable generated transfers when desired (e.g. if a feed does not provide its own transfers or when using >1 feed). - The implementation of implicit transfers (e.g. transfers from one stop to the same stop) now applies at the SQL loader level (rather then at the algorithm level) and the concept of 'transfer categories' is completely abstracted from the routing / CSA algorithm itself. - Performance-related: a heuristic is used to avoid a full selfjoin in determining potential computed/generated transfers. Additionally the size of the generated transfers table was decreased by avoiding duplicate entries. Both of these changes vastly decreases the amount of time to create the generated transfers table & improves overall transfers performance. 2. Parent Stop Lookup Accuracy & Performance Improvements The logic which determines looking up parent stops has been reworked entirely and now utilizes the GTFS location_type field. - The GTFS location_type field is properly respected and each case (for root stations, exit/entry/nodes, platforms, and boarding areas) is handled separately in implementation logic. The new logic is much more accurate then the previous logic which contained a number of bugs effecting GTFS feeds such as the Prague feed (MDBID 767). - The recursive CTE logic previously utilizing the parent_station field was removed, as location_type field lookup is all that is needed for accurate lookup. - Performance is vastly improved for parent stop lookups since without the recursive CTE and with utilizing location_type, lookup is a 'flat' operation. Additionally parent station lookup no longer utilizes a computed table (_cvstopparent) but rather utilizes the direct SQL view (_vstopparent) in downstream logic (since it's efficient enough to do so); and this thus also entirely removes a good amount of wall time in the initial compute cycle prior to routing. 3. Multiday Route Support Multiple day route support has been added such that connections that are obtained from multiple different adjacent schedule days can be used in combination for a single calculated target route. - Beginning a route at or near midnight in the target locale now can be expected to work and the system will pull multiple connections from different schedule days (interleaving properly) into a single unified schedule from which to determine the target route. - Previously, routes beginning around midnight would defacto fail (depending on each feed's availability of timetable data exceeding the >24hr boundary). Now instead, the routing system works in all cases where routing would previously fail in this usecase. - Returned connections are formatted in RFC3339 format to allow seeing which day each leg component time is from (in addition to timezone). 4. Timezone Handling Support Timezone handling logic has been implemented per GTFS specification and each feed's timezone is respected (independently of the routing request's input timezone). - Connections extraction logic (in the connections loader) handles normalizing all connections times to UTC times (independent of timezone) prior to ever being entered into the CSA algorithm. - Routes are formatted with times in RFC3339 format normalized to whichever timezone or UTC delta the input routing request used. - Multiple feeds with agency providers using different timezones may be utilized in a single routing request. 5. Unit Tests: SQL Views Tests, SQL Loaders Tests, and CI The overall testsuite particularly for the SQL-based components of the system (e.g. everything before hitting the CSA algorithm) has been vastly improved. - Unit tests have been introduced for transfers, stopwalks, and connections functionality. SQL 'loader' tests handle extensively testing edgecases for each of these components. Unit tests for prior mentioned timezone handling, multiday logic, transfers logic, and parent stop lookup logic is implemented. See the `dbquery_test` package. - In addition to SQL 'loader' tests, more componentized SQL view tests are implemented to test the behavior of SQL views used as subcomponents of the SQL loaders (e.g. _vstopparent, _vcaltoservice). See the `dbstubquery_test` package. - The GTFS-based routing integration tests (within `apiroute_test`) that generate results at https://ci.lrdu.org/tests_results/ have been updated to additionally validate route duration & number of transfers to make it simpler to debug routing system updates. Additionally, results for each commit are now published to https://ci.lrdu.org/tests_results_refs/ - The CI that is responsible for generating test results has been updated to send emails with build results to the new ~mil/mobroute-ci@lists.sr.ht mailing list.
This release of Mobroute was focused around usability updates to make the project more approachable to end-users; and additionally added some infrastructure to improve the testsuite. Details on changes below: 1. Output Formatters Rework; And addition of GeoJSON & Map URL Formatters A new field "output_formats" has been added to the request structure for route requests. This field enables the user to customize the format of the output returned the calculated routes. Previously the "legs" and "diagnostics" formats were always shown by default. Now it is customizable if you wish to suppress either of these fields. Additionally, several new formatters have been added. Importantly, a new GeoJSON formatter allows exporting the route rendered in GeoJSON format which is capable of being viewed in many external applications. And additionally, the new mapurl formatter provides a URL based on the http://geojson.lrdu.org service which renders arbitrary GeoJSON in the browser. The full list of supported output formatters (and description of each) now is as follows: - legs: Shows the route "legs" in format as it was previously - diagnostics: Merges the previous 'diagnostic_perf' & 'diagnostics_route' - request: Shows the raw request used for the route - geojson: Show the route as GeoJSON points (with steps information merged) - mapurl: Shows the route as a URL on geojson.lrdu.org domain with GeoJSON passed in hash 2. Migration of Views / Computed Tables to Mobroute Proper The actual (SQL) views & computed tables used in Mobroute's routing logic, previously housed in Mobsql, has been migrated to Mobroute's codebase itself. This change was enabled by an API update in Mobsql's library that allows end users / consumers of Mobsql's API to define arbitrary SQL views & computed tables. This update makes the overall codebases of Mobroute and Mobsql alot cleaner in that now Mobsql is purely a GTFS-to-SQLite ETL tool and all Mobroute application-specific-logic lives in Mobroute's repo and is parameterized in the call to initializing Mobsql's runtime. See the `db/dbschemaextra` folder for more details. 3. Calendar Service Mapping View Fix Historical routes (e.g. routes calculated for dates in the past) are now properly calculated. Prior to changes in this cycle, if you attempted to calculate a historical route (e.g. for any time in the past) the route request would fail as the underlying _vcaltoservice view only looked at dates in the present and 30 days in the future. This view has been corrected to properly respect both the underlying calendar.txt and calendar_dates.txt GTFS tables and as such now routes are properly calculated for dates in the past & >30 days in the future. This was an important change to make as this enables tests to properly function independent of 'current' GTFS data. 4. Routing API Testing: GTFS Testing Architecture Testing architecture for the routing API itself has been added in this release. Prior, the main testsuites in Mobroute revolved around testing the CSA algorithm (which is abstracted from the routing API - the latter being a higher level interface). The whole routing API covers quite a bit: from Mobility Database CSV ingestion, to GTFS import via Mobsql, to SQL view extractions, to running the CSA algorithm, and to finally formatting the response. With this release cycle, the whole API for routing is tested as an integrated unit. See the new `apiroute_test` package for details. Also 'realistic' GTFS tests (sourced from actual Mobility Database GTFS feeds snapshotted) were added in this package. The 'realistic' GTFS tests run in CI and the results are outputted in HTML format and uploaded to: http://ci.lrdu.org/tests_results/ 5. Documentation Rework Documentation for Mobroute has been reworked as a whole to be more approachable to end-users. This involved splitting prior documentation into multiple pages and a rework of the README and project description overall. The documentation overall also matches the style & fits cohesively with the related Transito project.