~louis77/pg2sqlite

05b435e48448a3479c4c6a90002f79575523d8f1 — Louis Brauer 2 months ago febd5e1
Update README and provide build script
6 files changed, 112 insertions(+), 23 deletions(-)

A .gitignore
M README.md
A VERSION
A build.sh
M go.mod
M main.go
A .gitignore => .gitignore +3 -0
@@ 0,0 1,3 @@
.idea
.vscode
build/
\ No newline at end of file

M README.md => README.md +61 -18
@@ 1,14 1,26 @@
# pg2sqlite

CLI tool to migrate a table from a PostgreSQL database to a SQLite3 database.
`pg2sqlite` can migrate tables from PostgreSQL to SQLite3.

### Installation
## Installation

### From source

```shell
$ go get -u github.com/louis77/pg2sqlite
```

### Usage
### Download pre-built binaries

| Platform | Intel/AMD | ARM |
| -------- | ----- | --- |
| MacOS | [Intel](https://pg2sqlite.surge.sh/pg2sqlite-darwin-amd64.gz) | [M1](https://pg2sqlite.surge.sh/pg2sqlite-darwin-arm64.gz) |
| Linux | [amd64](https://pg2sqlite.surge.sh/pg2sqlite-linux-amd64.gz) | [arm](https://pg2sqlite.surge.sh/pg2sqlite-linux-arm64.gz) |
| FreeBSD | [amd64](https://pg2sqlite.surge.sh/pg2sqlite-freebsd-amd64.gz) | [arm64](https://pg2sqlite.surge.sh/pg2sqlite-freebsd-arm.gz) |
| Windows | [amd64](https://pg2sqlite.surge.sh/pg2sqlite-windows-amd64.exe.zip) | [arm](https://pg2sqlite.surge.sh/pg2sqlite-windows-arm.exe.zip) |


## Usage

```shell
$ pg2sqlite -h


@@ 16,8 28,8 @@ $ pg2sqlite -h
Options:

  -h, --help                   display help information
      --pg-url                *Postgres connection string
      --sqlite-file           *Path to SQLite database
      --pg-url                *Postgres connection string (i.e. postgres://localhost:5432/mydb)
      --sqlite-file           *Path to SQLite database file (i.e. mydatabase.db)
      --table                 *Name of table to export
      --drop-table-if-exists   DANGER: Drop target table if it already exists
```


@@ 52,20 64,57 @@ CREATE TABLE daily_sales (
Does this look ok? (Y/N) y

Estimated row count: 50042260
  24s [--------------------------------------------------------------------]   0%
  24s [==>-----------------------------------------------------------------]   3%

Finished.

$ 
```

## Details
## Release history

### General
- 1.0.0
    - Initial version     

## Details

pg2sqlite works with a single connection to PostgreSQL and SQLite. To keep memory consumption low, rows are transferred
without buffering.

### Workflow

`pg2sqlite` will try to complete these steps in the following order:

1. Validate the connection to Postgres
2. Validate the existence of the specified SQLite file
3. Fetch the table schema from Postgres
4. Display a `CREATE TABLE` statement for the SQLite table
5. Ask for your confirmation (can be silenced)
6. Drop target table if it already exists
7. Estimate number of rows in the Postgres table for progress display
8. Queries source table rows and inserts them while they come in

#### `pg2sqlite` doesn't do:

`pg2sqlite` created the bare table with its columns.
No primary keys, foreign keys, constraints or indexes are created
in the SQLite table.


### TODOs

- [ ] Create SQLite file if it doesn't exist
- [ ] Silence confirmations
- [ ] Offer verification option (number of rows)


### Warnings

Be careful with the `--drop-table-if-exists` option. It *will* drop your SQLite-table without
confirmation. This is useful if you use `pg2sqlite` in a scripted context. By default, if 
the table already exists in SQLite, `pg2sqlite` will terminte the process with
an error message to avoid data loss.

### Type mapping

SQLite supports only a limited number of types aka. storage classes. Postgres' complex types like JSON, ARRAY etc. are


@@ 75,18 124,12 @@ Here is a table of explicit mappings:

|PG Type | SQLite Type|
|--------|------------|
|integer | INTEGER |
|smallint| INTEGER |
|integer, smallint | INTEGER |
|numeric|REAL|
|date|TEXT|
|array|TEXT|
|character|TEXT|
|character varying|TEXT|
|timestamp with time zone|TEXT|
|All other types|TEXT|

## LICENSE
## License

© 2021 Louis Brauer
Copyright © 2021 by Louis Brauer.

GNU GPLv3. See [LICENSE](./LICENSE).
\ No newline at end of file
Software released under GNU GPLv3 license. See [LICENSE](./LICENSE).
\ No newline at end of file

A VERSION => VERSION +1 -0
@@ 0,0 1,1 @@
1.0.0
\ No newline at end of file

A build.sh => build.sh +42 -0
@@ 0,0 1,42 @@
#!/usr/bin/env bash

builddir=build
name=pg2sqlite

declare -a goos=(  darwin darwin linux linux windows windows freebsd freebsd )
declare -a goarch=(amd64  arm64  amd64 arm64 amd64   arm     amd64   arm     )

# Compress: os file
compress () {
  if [ "$1" == "windows" ]
  then
    zip -m "$2".zip "$2"
  else
    gzip "$2"
  fi
}

if [ ! -d $builddir ]
  then
    echo "Creating build directory"
    mkdir -p $builddir
  else
    echo "Cleaning build directory"
    rm $builddir/*
fi

echo "Building pg2sqlite ..."

for K in "${!goos[@]}"; do
  current_os=${goos[$K]}
  current_arch=${goarch[$K]}
  product=$name-$current_os-$current_arch
  if [ "$current_os" == "windows" ]; then product=$product.exe; fi
  echo Build "$K" : "$product"
  env GOOS="$current_os" GOARCH="$current_arch" go build -o $builddir/"$product"
  compress "$current_os" $builddir/"$product"
done

echo "Uploading to surge"

surge $builddir pg2sqlite.surge.sh
\ No newline at end of file

M go.mod => go.mod +2 -2
@@ 1,10 1,10 @@
module pg2sqlite
module git.sr.ht/~louis77/pg2sqlite

go 1.16

require (
	github.com/gosuri/uilive v0.0.4 // indirect
	github.com/gosuri/uiprogress v0.0.1 // indirect
	github.com/gosuri/uiprogress v0.0.1
	github.com/jackc/pgx/v4 v4.11.0
	github.com/mattn/go-sqlite3 v1.14.7
	github.com/mkideal/cli v0.2.5

M main.go => main.go +3 -3
@@ 33,8 33,8 @@ import (

type argT struct {
	cli.Helper
	PGURL             string `cli:"*pg-url" usage:"Postgres connection string"`
	SLFile            string `cli:"*sqlite-file" usage:"Path to SQLite database"`
	PGURL             string `cli:"*pg-url" usage:"Postgres connection string (i.e. postgres://localhost:5432/mydb)"`
	SLFile            string `cli:"*sqlite-file" usage:"Path to SQLite database file (i.e. mydatabase.db)"`
	Tablename         string `cli:"*table" usage:"Name of table to export"`
	DropTableIfExists bool   `cli:"drop-table-if-exists" usage:"DANGER: Drop target table if it already exists" default:"false"`
}


@@ 65,7 65,7 @@ func run(ctx *cli.Context) error {
	if err != nil {
		log.Fatal(err)
	}
	fmt.Printf("Creating Table statement:\n%s\n", createTableSQL)
	fmt.Printf("Will create SQLite table with the following statement:\n%s\n\n", createTableSQL)
	if !AskYesNo("Does this look ok?") {
		log.Fatal("Cancelled")
	}