~cg/uls2sqlite

5d2a36dc9b0d5da47d53287878ff99cff175dd77 — Chris Goff 1 year, 25 days ago
Initial commit.
A  => LICENSE +13 -0
@@ 1,13 @@
Copyright (c) 2023 Chris Goff <mailbag@chrisapproved.com>

Permission to use, copy, modify, and distribute this software for any
purpose with or without fee is hereby granted, provided that the above
copyright notice and this permission notice appear in all copies.

THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
\ No newline at end of file

A  => README.md +98 -0
@@ 1,98 @@
# ULS2SQLite

This tool is designed to convert an FCC Universal Licensing System (ULS) database into an SQLite database.


## Requirements  

- Bash (or compatible shell) for helper tools
- Python 3.9 or later
- sqlite-utils Python package
- [FCC ULS Database files](https://www.fcc.gov/wireless/data/public-access-files-database-downloads)
- curl (if using the download_weekly_dbs.sh helper tool)

ULS2SQLite has been tested on [Fedora Linux 37](https://fedoraproject.org/), but may be portable to other operating systems.

## Installation  

```
git clone https://git.sr.ht/~cg/ULS2SQLite
cd ULS2SQLite/
pip install -r requirements.txt
```

## How to Use

This tool will process `.DAT` files from a daily or weekly download from the FCC website, or take a single file from a geographical data export.

Command-line arguments to specify input and output files. Use the `-i` or `--import_file` flag to specify the ULS file to be imported, and the `-o` or `--output_file` flag to specify the SQLite database filename to export to. If no SQLite database exists, it will be created.

Example using ULS `.DAT` file as input:
`python uls2sqlite.py -i HD.dat -o output.sqlite --encoding=utf-8`

Note search export files are also pipe-delimited, however they are a single file containing many ULS records. There is a helper tool included, `uls_splitter.py`, which can be run against these files to split them into separate pipe-delimited files.

Example using a single ULS file as input:

```bash
cd tools
python uls_splitter.py -i UL20236291542491.txt
cd ..
bash process_pipe_files.sh databases/my_database.sqlite
```

This will split the `UL20236291542491.txt` file into separate pipe-delimited files, and process each of them into the `my_database.sqlite` SQLite database located in the `databases` directory.

There is also a helper file, `process_zip_files.sh`, which can be used to process ZIP files downloaded from the FCC website located in the `archives` directory. This will process all the files for you in an automated manner.

Example using ZIP files downloaded to the `archives` directory:
`bash process_zip_files.sh`

### Helper Tools

There are a number of helper tools included to assist with the conversion process.

#### process_zip_files.sh
Processes batches of ZIP files downloaded from the FCC website and converts them into SQLite databases.  

Usage: `bash process_zip_files.sh`

Note: If [GNU Parallel](https://www.gnu.org/software/parallel/) is installed on your system, the script will attempt to
process ZIP files in parallel for a massive speed boost. Be aware this can consume considerable system resources!

#### process_pipe_files.sh
Processes a single pipe-delimited file downloaded from the FCC website and converts it into an SQLite database.

Processes a single pipe-delimited ULS database to a SQLite database. Run `uls_splitter.py` first to generate the necessary
pipe-delimited files.

Usage: `bash process_pipe_files.sh output.sqlite`

#### tools/download_weekly_dbs.sh
Downloads the weekly ULS databases from the URLs in `weekly_dbs_urls.txt`.

#### tools/ready_dat_files.py
Ready the .DAT files for import by making sure filenames are correct.

#### tools/uls_splitter.py
If a combined ULS database is being used, split it into separate pipe-delimited files by record type. Files are
placed in the `temp` directory.

Usage: `python uls_splitter.py -i UL20236291542491.txt`

#### tools/weekly_dbs_urls.txt
Contains list of URLs for weekly ULS databases provided by the FCC.  

## Resources

ULS Databases:  
    https://www.fcc.gov/wireless/data/public-access-files-database-downloads

ULS definitions SQL file:  
    https://www.fcc.gov/sites/default/files/public_access_database_definitions_sql_v6_0_0.txt

ULS record types  
    https://www.fcc.gov/sites/default/files/pubacc_intro_11122014.pdf

ULS definitions:
    https://www.fcc.gov/sites/default/files/uls_code_definitions_07182022.txt 
\ No newline at end of file

A  => archives/README.txt +1 -0
@@ 1,1 @@
Place downloaded FCC ULS database ZIP files in this directory.
\ No newline at end of file

A  => databases/README.txt +1 -0
@@ 1,1 @@
Directory to store the SQLite database files.
\ No newline at end of file

A  => definitions/public_access_database_definitions_sql_v6_0_0.txt +2032 -0
@@ 1,2032 @@
create table dbo.PUBACC_A2
(
      Record_Type               char(2)              null,
      unique_system_identifier  numeric(9,0)         not null,
      ULS_File_Number           char(14)             null,
      EBF_Number                varchar(30)          null,
      spectrum_manager_leasing  char(1)              null,
      defacto_transfer_leasing  char(1)              null,
      new_spectrum_leasing      char(1)              null,
      spectrum_subleasing       char(1)              null,
      xfer_control_lessee       char(1)              null,
      revision_spectrum_lease   char(1)              null,
      assignment_spectrum_lease char(1)              null,
      pfr_status		char(1)		     null
    		
)

go
create table dbo.PUBACC_AC
(
      record_type               char(2)              null,
      unique_system_identifier  numeric(9,0)         not null,
      uls_file_number           char(14)             null,
      ebf_number                varchar(30)          null,
      call_sign                 char(10)             null,
      aircraft_count            int                  null,
      type_of_carrier           char(1)              null,
      portable_indicator        char(1)              null,
      fleet_indicator           char(1)              null,
      n_number                  char(10)             null
)

go

create table dbo.PUBACC_AD
(
      Record_Type               	char(2)              null,
      unique_system_identifier  	numeric(9,0)         not null,
      ULS_File_Number           	char(14)             null,
      EBF_Number                	varchar(30)          null,
      Application_Purpose       	char(2)              null,
      Application_Status       		char(1)              null,
      Application_Fee_Exempt    	char(1)              null,
      Regulatory_Fee_Exempt     	char(1)              null,
      Source                    	char(1)              null,
      requested_expiration_date_mmdd 	char(4)              null,
      Receipt_Date              	char(10)             null,
      Notification_Code         	char(1)              null,
      Notification_date         	char(10)             null,
      Expanding_Area_or_Contour 	char(1)              null,
      Change_Type               	char(1)              null,
      Original_Application_Purpose 	char(2)              null,
      Requesting_A_Waiver       	char(1)              null,
      How_Many_Waivers_Requested 	int                  null,
      Any_Attachments           	char(1)              null,
      Number_of_Requested_SIDs  	int                  null,
      fee_control_num           	char(16)             null,
      date_entered              	char(10)             null,
      reason                    	varchar(255)         null,
      frequency_coordination_indicat 	char(1)              null,
      emergency_sta             	char(1)              null,
      overall_change_type       	char(1)              null,
      slow_growth_ind           	char(1)              null,
      previous_waiver           	char(1)              null,
      waiver_deferral_fee       	char(1)              null,
      has_term_pending_ind		char(1)		     null,
      use_of_service       		char(1)		     null
)

go


create table dbo.PUBACC_AG
(
      record_type               char(2)              null,
      unique_system_identifier  numeric(9,0)         not null,
      uls_file_number           char(14)             null,
      ebf_number                varchar(30)          null,
      action_requested		char(1)		     null,
      agreement_type		char(2) 	     null,
      agreement_name		varchar(50)	     null,
      des_ent_or_closed_bid_impact char(1)	     null,
      reserved_for_future	char(1)		     null,
      mod_to_rev		char(1)		     null						 	  	
)

go

create table dbo.PUBACC_AH
(
      record_type		char(2)		     not null,
      unique_system_identifier  numeric(9,0)         null,
      uls_file_num              char(14)             null,
      attachment_desc           varchar(60)          null, 
      attachment_file_id        char(18)             null
)

go

create table dbo.PUBACC_AM
(
      record_type               char(2)              not null,
      unique_system_identifier  numeric(9,0)         not null,
      uls_file_num              char(14)             null,
      ebf_number                varchar(30)          null,
      callsign                  char(10)             null,
      operator_class            char(1)              null,
      group_code                char(1)              null,
      region_code               tinyint              null,
      trustee_callsign          char(10)             null,
      trustee_indicator         char(1)              null,
      physician_certification   char(1)              null,
      ve_signature              char(1)              null,
      systematic_callsign_change char(1)             null,
      vanity_callsign_change    char(1)              null,
      vanity_relationship       char(12)             null,
      previous_callsign         char(10)             null,
      previous_operator_class   char(1)              null,
      trustee_name              varchar(50)          null
)

go

create table dbo.PUBACC_AN
(
      record_type              	char(2)              null,
      unique_system_identifier  numeric(9,0)         not null,
      uls_file_number           char(14)             null,
      ebf_number                varchar(30)          null,
      call_sign                	char(10)             null,
      antenna_action_performed  char(1)              null,
      antenna_number            int                  null,
      location_number           int                  null,
      receive_zone_code         char(6)              null,
      antenna_type_code         char(1)              null,
      height_to_tip            	numeric(5,1)         null,
      height_to_center_raat     numeric(5,1)         null,
      antenna_make              varchar(25)          null,
      antenna_model             varchar(25)          null,
      tilt                      numeric(3,1)         null,
      polarization_code         char(5)              null,
      beamwidth                 numeric(4,1)         null,
      gain                      numeric(4,1)         null,
      azimuth                   numeric(4,1)         null,
      height_above_avg_terrain  numeric(5,1)         null,
      diversity_height          numeric(5,1)         null,
      diversity_gain            numeric(4,1)         null,
      diversity_beam            numeric(4,1)         null,
      reflector_height          numeric(5,1)         null,
      reflector_width           numeric(4,1)         null,
      reflector_separation      numeric(5,1)         null,
      repeater_seq_num          int                  null,
      back_to_back_tx_dish_gain numeric(4,1)         null,
      back_to_back_rx_dish_gain numeric(4,1)         null,
      location_name             varchar(20)          null,
      passive_repeater_id       int                  null,
      alternative_cgsa_method   char(1)              null,
      path_number               int                  null,
      line_loss                 numeric(3,1)         null,
      status_code		char(1)		     null,
      status_date		datetime	     null,
      psd_nonpsd_methodology    varchar(10)          null,
      maximum_erp               numeric(15,3)        null
)

go


create table dbo.PUBACC_AP
(
      record_type               char(2)              null,
      unique_system_identifier  numeric(9,0)         not null,
      uls_file_number           char(14)             null,
      ebf_number                varchar(30)          null,
      agreement_name		varchar(50)	     null,
      action_requested		char(1)		     null,
      legal_entity_name		varchar(50)	     null,
      first_name		varchar(20)	     null,
      mi                        char(1)              null,
      last_name                 varchar(20)          null,
      suffix                    char(3)              null,
      FCC_reg_number	        char(10)	     null	
)

go
create table dbo.PUBACC_AS
(
      record_type               char(2)              not null,
      unique_system_identifier  numeric(9,0)         null,
      uls_file_number           char(14)             null,
      ebf_number                varchar(30)          null,
      callsign                  char(10)             null,
      assoc_callsign            char(10)             null,
      status_code		char(1)		     null,
      status_date		datetime	     null,
      action_performed  	char(1)              null
)
 
go

create table dbo.PUBACC_AT
(
      Record_Type               char(2)              null,
      unique_system_identifier  numeric(9,0)         not null,
      ULS_File_Number           char(14)             null,
      EBF_Number                varchar(30)          null,
      Attachment_Code           char(5)              null,
      Attachment_Description    varchar(60)          null,
      Attachment_Date           char(10)             null,
      attachment_file_name      varchar(60)          null,
      attachment_action_performed char(1)            null
)

go

create table dbo.PUBACC_BC
(
      record_type               char(2)              not null,
      unique_system_identifier  numeric(9,0)         not null,
      uls_file_number           char(14)             null,
      ebf_number                varchar(30)          null,
      callsign                  char(10)             null,
      broadcast_callsign        char(10)             null,
      broadcast_city            char(20)             null,
      broadcast_state           char(2)              null,
      parent_facility_id        int                  null,
      parent_class_code         char(2)              null,
      nonparent_type_code       char(1)              null
)


go


create table dbo.PUBACC_BD
(
      record_type               char(2)              not null,
      unique_system_identifier  numeric(9,0)         not null,
      uls_file_number           char(14)             null,
      ebf_number                varchar(30)          null,
      involves_credit           char(1)              null,
      involves_payment          char(1)              null,
      involves_close_bidding    char(1)              null,
      credits_paid              char(1)              null,
      bc_same_elig_status       char(1)              null,
      bc_diff_elig_status       char(1)              null,
      bc_dnq_elig_status	char(1)              null,
      payments                  char(1)              null,
      ip_same_elig_status       char(1)              null,
      ip_diff_elig_status       char(1)              null,
      ip_dnq_elig_status        char(1)              null,
      closed_notification       char(1)              null,
      qualifies_for_cb          char(1)              null,
      dnq_for_cb                char(1)              null,
      bc_ten_percent		char(1)              null,
      bc_affiliate		char(1)              null,
      bc_cont_elig		char(1)              null,
      bc_shared			char(1)              null,
      document			char(1)              null,
      ip_ten_percent		char(1)              null,	
      ip_affiliate		char(1)              null, 
      ip_cont_elig		char(1)              null,
      ip_shared			char(1)              null,
      sec_date			char(10)              null,
      lien_date			char(10)              null,
      cb_ten_percent		char(1)              null,
      cb_affiliate		char(1)              null,
      cb_cont_elig		char(1)              null,
      cb_shared			char(1)              null,
	  bidding_credit_type char(1)	null,
      bidding_credit_percent char(3)	null,
      bc_spectrum_capacity char(1)	null
)


go


create table dbo.PUBACC_BE
(
      record_type                 char(2)              not null,
      unique_system_identifier    numeric(9,0)         not null,
      uls_file_number             char(14)             null,
      ebf_number                  varchar(30)          null,
      callsign                    char(10)             null,
      multichannel		  char(1)	       null,
      cable_tv			  char(1)	       null,
      programming_requirements	  char(1)	       null,
      interference_protection	  char(1)	       null

)



go

create table dbo.PUBACC_BF
(
      record_type               char(2)              null,
      unique_system_identifier  numeric(9,0)         not null,
      call_sign                 char(10)             null,
      location_number           int                  null,
      antenna_number            int                  null,
      frequency_assigned        numeric(16,8)        null,
      buildout_code             int                  null,
      buildout_deadline         char(10)             null,
      buildout_date             char(10)             null,
      status_code		char(1)		     null,
      status_date		datetime	     null,
      frequency_number	        int		     null,
      discontinuance_date             char(10)                  null
)


go


create table dbo.PUBACC_BL
(
      record_type               char(2)              null,
      unique_system_identifier  numeric(9,0)         not null,
      call_sign                 char(10)             null,
      location                  int                  null,
      buildout_code             int                  null,
      buildout_deadline         char(10)             null,
      buildout_date             char(10)             null,
      status_code		char(1)		     null,
      status_date		datetime	     null,
      discontinuance_date     char(10)                  null
)

go

create table dbo.PUBACC_BO
(
      record_type               char(2)              null,
      unique_system_identifier  numeric(9,0)         not null,
      call_sign                 char(10)             null,
      buildout_code             int                  null,
      buildout_deadline         char(10)             null,
      buildout_date             char(10)             null,
      status_code		char(1)		     null,
      status_date		datetime	     null,
      discontinuance_date     char(10)                  null
)

go

create table dbo.PUBACC_BT
(
      record_Type               char(2)              null,
      unique_system_identifier  numeric(9,0)         not null,
      ULS_File_Number           char(14)             null,
      EBF_Number                varchar(30)          null,
      Applicant_Status		char(1)              null
     			
)
go

create table dbo.PUBACC_CD
(
	Record_Type			char(2)		null,
	Unique_System_Identifier	numeric(9,0)	not null,
	ULS_File_Number			char(14)	null,
	EBF_Number			varchar(30)	null,
	Year_Sequence			smallint	not null,
	Gross_Revenues			money		null,
	Year_End_Date			datetime	null,
	Aggregate_Gros_Rvn_DE		money		null,
	Aggregate_Gros_Rvn_CB		money		null,
	Total_Assets			money		null,
	avg_gross_rvn_de_3  money null,
	avg_gross_rvn_de_5  money null

)

go

create table dbo.PUBACC_CF
(
      Record_Type                       char(2)              null,
      unique_system_identifier          numeric(9,0)         not null,
      ULS_File_Number                   char(14)             null,
      EBF_Number                        varchar(30)          null,
      Item_Type_Indicator               char(1)              null,
      Item_Type                         char(14)             null,
      Constructed                       char(1)              null,
      Location_Number                   int                  null,
      Path_Number                       int                  null,
      Frequency_Assigned                numeric(16,8)        null,
      Frequency_Upper_Band              numeric(16,8)        null,
      number_of_mobiles                 int                  null,
      action_performed                  char(1)              null,
      actual_date_of_construction       datetime	     null,
      frequency_number                  int                  null,
      Assign_Callsign                   char(10)             null 

)

go

create table dbo.PUBACC_CG
(
      record_type               char(2)              null,
      unique_system_identifier  numeric(9,0)         null,
      uls_file_number           char(14)             null,
      ebf_number                varchar(30)          null,
      call_sign                 char(10)             null,
      station_available         char(1)              null,
      public_correspondence     char(1)              null,
      station_identifier        char(12)             null,
      aeronautical_enroute_call_sign char(10)        null,
      faa_office_notified       varchar(255)         null,
      date_faa_notified         char(10)             null,
      seeking_authorization     char(1)              null,
      regularly_engaged         char(1)              null,
      engaged                   char(1)              null,
      public_mooring            char(1)              null,
      servicing                 char(1)              null,
      fixed_station             char(1)              null,
      maritime_support          char(1)              null,
      aeronautical_fixed        char(1)              null,
      unicom                    char(1)              null,
      search_and_rescue         char(1)              null,
      flight_test_uhf           char(1)              null,
      flight_test_manufacturer  char(1)              null,
      flight_test_parent_corporation char(1)         null,
      flight_test_educational   char(1)              null,
      flight_school_certitication char(1)            null,
      lighter_than_air          char(1)              null,
      ballooning                char(1)              null,
      located_at_airport        char(1)              null,
      radiodetermination_not_faa char(1)             null,
      radiodetermination_equipment char(1)           null,
      radiodetermination_public char(1)              null,
      radiodetermination_elts   char(1)              null,
      civil_air_patrol          char(1)              null,
      aeronautical_enroute      char(1)              null,
      mobile_routine            char(1)              null,
      mobile_owner_operator     char(1)              null,
      mobile_agreement          char(1)              null,
      coast_ground_identifier   char(12)             null,
      selective_call_sign_identifier char(4)         null,
      station_class             char(4)              null,
      status_code		char(1)	             null,
      status_date		datetime	     null
)

go

create table dbo.PUBACC_CO
(
      record_type               char(2)              not null,
      unique_system_identifier  numeric(9,0)         not null,
      uls_file_num              char(14)             null,
      callsign                  char(10)             null,
      comment_date              char(10)             null,
      description               varchar(255)         null,
      status_code		char(1)		     null,
      status_date		datetime             null
)

go

create table dbo.PUBACC_CP
(
      record_type               char(2)              null,
      unique_system_identifier  numeric(9,0)         not null,
      uls_file_number           char(14)             null,
      ebf_number                varchar(30)          null,
      call_sign                 char(10)             null,
      control_point_action_performed char(1)         null,
      control_point_number      int                  null,
      control_address           varchar(80)          null,
      control_city              char(20)             null,
      state_code                char(2)              null,
      control_phone             char(10)             null,
      control_county            varchar(60)          null,
      status_code		char(1)		     null,
      status_date		datetime	     null
)

go

create table dbo.PUBACC_CS
(
      record_type               char(2)              null,
      unique_system_identifier  numeric(9,0)         not null,
      uls_file_number           char(14)             null,
      call_sign                 char(10)             null,
      location_number           int                  null,
      antenna_number            int                  null,
      frequency_assigned        numeric(16,8)        null,
      coser_result              char(5)              null,
      coser_num                 char(10)             null,
      coser_activity_type       char(1)              null,
      status_code		char(1)		     null,
      status_date		datetime	     null
)

go

create table dbo.PUBACC_EM
(
      record_type               char(2)              null,
      unique_system_identifier  numeric(9,0)         not null,
      uls_file_number           char(14)             null,
      ebf_number                varchar(30)          null,
      call_sign                 char(10)             null,
      location_number           int                  null,
      antenna_number            int                  null,
      frequency_assigned        numeric(16,8)        null,
      emission_action_performed char(1)              null,
      emission_code             char(10)             null,
      digital_mod_rate          numeric(8,1)         null,
      digital_mod_type          char(255)            null,
      frequency_number          int                  null,
      status_code		char(1)		     null,
     status_date		datetime	     null,
     emission_sequence_id       int                  null
)

go

create table dbo.PUBACC_EN
(
      record_type               char(2)              not null,
      unique_system_identifier  numeric(9,0)         not null,
      uls_file_number           char(14)             null,
      ebf_number                varchar(30)          null,
      call_sign                 char(10)             null,
      entity_type               char(2)              null,
      licensee_id               char(9)              null,
      entity_name               varchar(200)         null,
      first_name                varchar(20)          null,
      mi                        char(1)              null,
      last_name                 varchar(20)          null,
      suffix                    char(3)              null,
      phone                     char(10)             null,
      fax                       char(10)             null,
      email                     varchar(50)          null,
      street_address            varchar(60)          null,
      city                      varchar(20)          null,
      state                     char(2)              null,
      zip_code                  char(9)              null,
      po_box                    varchar(20)          null,
      attention_line            varchar(35)          null,
      sgin                      char(3)              null,
      frn                       char(10)             null,
      applicant_type_code       char(1)              null,
      applicant_type_other      char(40)             null,
      status_code               char(1)		     null,
      status_date		datetime		null,
      lic_category_code	char(1)		null,
      linked_license_id	numeric(9,0)	null,
      linked_callsign		char(10)		null
)

go


create table dbo.PUBACC_F2
(
      record_type               char(2)              null,
      unique_system_identifier  numeric(9,0)         not null,
      uls_file_number           char(14)             null,
      ebf_number                varchar(30)          null,
      call_sign                 char(10)             null,
      action_performed          char(1)              null, 
      location_number           int                  null,
      antenna_number            int                  null,
      frequency_number          int                  null,
      frequency_assigned        numeric(16,8)        null,
      frequency_upper_band      numeric(16,8)        null,
      offset                    char(3)              null,
      frequency_channel_block   char(4)              null,
      equipment_class	        char(2)              null, 
      minimum_power_output	numeric(15,3)        null,
      status_code		char(1)		     null,
      status_date		datetime	     null,
      trans_meth                char(1)              null
)



go


create table dbo.PUBACC_FA
(
      record_type               char(2)              null,
      unique_system_identifier  numeric(9,0)         null,
      uls_file_number           char(14)             null,
      ebf_number                varchar(30)          null,
      call_sign                 char(10)             null,
      operator_class            char(2)              null,
      ship_radar_endorsement    char(1)              null,
      six_month_endorsement     char(1)              null,
      date_of_birth             char(10)             null,
      certification_not_restricted char(1)           null,
      cert_restricted_permit    char(1)              null,
      restricted_permit_limited_use char(1)          null,
      cole_manager_code         char(5)              null,
      dm_call_sign              char(10)             null,
      proof_of_passing          char(1)              null 
)

go

create table dbo.PUBACC_FC
(
      record_type               char(2)              null,
      unique_system_identifier  numeric(9,0)         not null,
      uls_file_number           char(14)             null,
      ebf_number                varchar(30)          null,
      coordination_number       char(25)             null,
      coordinator_name          varchar(40)          null,
      coordinator_phone         char(10)             null,
      freq_coordination_date    char(10)             null,
      action_performed          char(1)              null
)

go

create table dbo.PUBACC_FF
(
      record_type               char(2)              null ,
      unique_system_identifier  numeric(9,0)         null ,
      callsign                  char(10)             null ,
      location_number           int                  null ,
      antenna_number            int                  null ,
      frequency                 numeric(16,8)        null ,
      frequency_number          int                  null ,
      freq_freeform_cond_type   char(1)              null ,
      unique_freq_freeform_id   numeric(9,0)         null ,
      sequence_number           int                  null ,
      freq_freeform_condition   varchar(255)         null,
      status_code		char(1)		     null,
      status_date		datetime	     null
)
 
go

create table dbo.PUBACC_FR
(
      record_type               char(2)              null,
      unique_system_identifier  numeric(9,0)         not null,
      uls_file_number           char(14)             null,
      ebf_number                varchar(30)          null,
      call_sign                 char(10)             null,
      frequency_action_performed char(1)             null,
      location_number           int                  null,
      antenna_number            int                  null,
      class_station_code        char(4)              null,
      op_altitude_code          char(2)              null,
      frequency_assigned        numeric(16,8)        null,
      frequency_upper_band      numeric(16,8)        null,
      frequency_carrier         numeric(16,8)        null,
      time_begin_operations     int                  null,
      time_end_operations       int                  null,
      power_output              numeric(15,3)        null,
      power_erp                 numeric(15,3)        null,
      tolerance                 numeric(6,5)         null,
      frequency_ind             char(1)              null,
      status                    char(1)              null,
      eirp                      numeric(7,1)         null,
      transmitter_make          varchar(25)          null,
      transmitter_model         varchar(25)          null,
      auto_transmitter_power_control char(1)         null,
      cnt_mobile_units          int                  null,
      cnt_mob_pagers            int                  null,
      freq_seq_id               int                  null,
      status_code		char(1)		     null,
      status_date		datetime	     null,
      date_first_used		datetime	     null
)

go

create table dbo.PUBACC_FS
(
      record_type               char(2)              null,
      unique_system_identifier  numeric(9,0)         not null,
      call_sign                 char(10)             null,
      location_number           int                  null,
      antenna_number            int                  null,
      frequency                 numeric(16,8)        null,
      frequency_number          int                  null,
      special_condition_type    char(1)              null,
      special_condition_code    int                  null,
      status_code		char(1)		     null,
      status_date		datetime	     null
      
)

go

 create table dbo.PUBACC_FT
(
      record_type               char(2)              null,
      unique_system_identifier  numeric(9,0)         not null,
      uls_file_number           char(14)             null,
      ebf_number                varchar(30)          null,
      call_sign                 char(10)             null,
      freq_type_action_performed char(1)             null,
      location_number           int                  null,
      antenna_number            int                  null,
      frequency_assigned        numeric(16,8)        null,
      frequency_type_number     int                  null,
      frequency_type_code       char(2)              null

)

go

create table dbo.PUBACC_HD
(
      record_type               char(2)              not null,
      unique_system_identifier  numeric(9,0)         not null,
      uls_file_number           char(14)             null,
      ebf_number                varchar(30)          null,
      call_sign                 char(10)             null,
      license_status            char(1)              null,
      radio_service_code        char(2)              null,
      grant_date                char(10)             null,
      expired_date              char(10)             null,
      cancellation_date         char(10)             null,
      eligibility_rule_num      char(10)             null,
      applicant_type_code_reserved       char(1)              null,
      alien                     char(1)              null,
      alien_government          char(1)              null,
      alien_corporation         char(1)              null,
      alien_officer             char(1)              null,
      alien_control             char(1)              null,
      revoked                   char(1)              null,
      convicted                 char(1)              null,
      adjudged                  char(1)              null,
      involved_reserved      	char(1)              null,
      common_carrier            char(1)              null,
      non_common_carrier        char(1)              null,
      private_comm              char(1)              null,
      fixed                     char(1)              null,
      mobile                    char(1)              null,
      radiolocation             char(1)              null,
      satellite                 char(1)              null,
      developmental_or_sta      char(1)              null,
      interconnected_service    char(1)              null,
      certifier_first_name      varchar(20)          null,
      certifier_mi              char(1)              null,
      certifier_last_name       varchar(20)          null,
      certifier_suffix          char(3)              null,
      certifier_title           char(40)             null,
      gender                    char(1)              null,
      african_american          char(1)              null,
      native_american           char(1)              null,
      hawaiian                  char(1)              null,
      asian                     char(1)              null,
      white                     char(1)              null,
      ethnicity                 char(1)              null,
      effective_date            char(10)             null,
      last_action_date          char(10)             null,
      auction_id                int                  null,
      reg_stat_broad_serv       char(1)              null,
      band_manager              char(1)              null,
      type_serv_broad_serv      char(1)              null,
      alien_ruling              char(1)              null,
      licensee_name_change	char(1)		     null,
      whitespace_ind            char(1)              null,
      additional_cert_choice    char(1)              null,
      additional_cert_answer    char(1)              null,
      discontinuation_ind       char(1)              null,
      regulatory_compliance_ind char(1)              null,
      eligibility_cert_900        char(1)              null,
      transition_plan_cert_900    char(1)              null,
      return_spectrum_cert_900  char(1)              null,
      payment_cert_900        char(1)              null
)

go

create table dbo.PUBACC_HS
(
      record_type               char(2)              not null,
      unique_system_identifier  numeric(9,0)         not null,
      uls_file_number           char(14)             null,
      callsign                  char(10)             null,
      log_date                  char(10)             null,
      code                      char(6)              null
)

go

create table dbo.PUBACC_IA
(
      record_type               char(2)              not null,
      unique_system_identifier  numeric(9,0)         not null,
      uls_file_number           char(14)             null,
      ebf_number                varchar(30)          null,
      call_sign                 char(10)             null,
      international_address_1   varchar(60)          null,
      international_address_2   varchar(60)          null,
      international_city        varchar(30)          null,
      country                   varchar(20)          null,
      international_zip_code    varchar(20)          null,
      international_phone       char(20)             null,
      international_fax         char(20)             null
)

go

create table dbo.PUBACC_IR
(
      record_type               char(2)              null,
      unique_system_identifier  numeric(9,0)         not null,
      uls_file_number           char(14)             null,
      call_sign                 char(10)             null,
      location_number           int                  null,
      antenna_number            int                  null,
      frequency_assigned        numeric(16,8)        null,
      irac_result               char(2)              null,
      fas_docket_num            char(8)              null,
      fccm_num                  char(10)             null,
      faa_ng_num                char(11)             null,
      status_code		char(1)		     null,
      status_date		datetime	     null,
      a_irac_status_code 	smallint     	     null
)

go

create table dbo.PUBACC_LA
 (
      record_type               char(2)              null ,
      unique_system_identifier  numeric(9,0)         null ,
      callsign                  char(10)             null ,
      attachment_code           char(1)              Null ,
      attachment_desc           varchar(60)          Null ,
      attachment_date           char(10)             Null ,
      attachment_filename       varchar(60)          Null ,
      action_performed          char(1)              Null
)

go

create table dbo.PUBACC_L2
(
      record_type               char(2)              not null,
      unique_system_identifier  numeric(9,0)         not null,
      uls_file_number           char(14)             null,
      ebf_number                varchar(30)          null,
      call_sign                 char(10)             null,
      location_action_performed	char(1)              null,
      location_number           int                  null,
      registration_required	char(1)              null,
      protection_date		datetime             null,
      link_reg_num              varchar(30)          null,
      link_reg_action_performed char(1)              null,
      mexico_clearance_indicator char(1)	     null,
      quiet_zone_consent        char(1)		     null,	
      status_code		char(1)		     null,
      status_date		datetime	     null
	
)

go

create table dbo.PUBACC_LF
 (
      record_type               char(2)              null ,
      unique_system_identifier  numeric(9,0)         null ,
      callsign                  char(10)             null ,
      location_number           int                  null ,
      loc_freeform_cond_type    char(1)              null ,
      unique_loc_freeform_id    numeric(9,0)         null ,
      sequence_number           int                  null ,
      loc_freeform_condition    varchar(255)         null,
      status_code		char(1)		     null,
      status_date		datetime	     null
)

go

create table dbo.PUBACC_LH
(
      record_type		char(2)		     not null,
      unique_system_identifier  numeric(9,0)         null,
      call_sign                 char(10)             null,
      attachment_desc           varchar(60)          null, 
      attachment_file_id        char(18)             null
)

go

create table dbo.PUBACC_LM
(
      record_type               char(2)              not null,
      unique_system_identifier  numeric(9,0)         null,
      uls_file_number           char(14)             null,
      ebf_number                varchar(30)          null,
      callsign                  char(10)             null,
      ext_implement_appr        char(1)              null,
      lm_eligibility_activity   varchar(255)         null,
      status_code		char(1)		     null,
      status_date		datetime	     null
)

go

create table dbo.PUBACC_LO
(
      record_type               char(2)              not null,
      unique_system_identifier  numeric(9,0)         not null,
      uls_file_number           char(14)             null,
      ebf_number                varchar(30)          null,
      call_sign                 char(10)             null,
      location_action_performed char(1)              null,
      location_type_code        char(1)              null,
      location_class_code       char(1)              null,
      location_number           int                  null,
      site_status               char(1)              null,
      corresponding_fixed_location int               null,
      location_address          varchar(80)          null,
      location_city             char(20)             null,
      location_county           varchar(60)          null,
      location_state            char(2)              null,
      radius_of_operation       numeric(5,1)         null,
      area_of_operation_code    char(1)              null,
      clearance_indicator       char(1)              null,
      ground_elevation          numeric(7,1)         null,
      lat_degrees               int                  null,
      lat_minutes               int                  null,
      lat_seconds               numeric(3,1)         null,
      lat_direction             char(1)              null,
      long_degrees              int                  null,
      long_minutes              int                  null,
      long_seconds              numeric(3,1)         null,
      long_direction            char(1)              null,
      max_lat_degrees           int                  null,
      max_lat_minutes           int                  null,
      max_lat_seconds           numeric(3,1)         null,
      max_lat_direction         char(1)              null,
      max_long_degrees          int                  null,
      max_long_minutes          int                  null,
      max_long_seconds          numeric(3,1)         null,
      max_long_direction        char(1)              null,
      nepa                      char(1)              null,
      quiet_zone_notification_date char(10)          null,
      tower_registration_number char(10)             null,
      height_of_support_structure numeric(7,1)       null,
      overall_height_of_structure numeric(7,1)       null,
      structure_type            char(7)              null,
      airport_id                char(4)              null,
      location_name             char(20)             null,
      units_hand_held           int                  null,
      units_mobile              int                  null,
      units_temp_fixed          int                  null,
      units_aircraft            int                  null,
      units_itinerant           int                  null,
      status_code		char(1)		     null,
      status_date		datetime	     null,
      earth_agree               char(1)              null


)

go

create table dbo.PUBACC_LS
(
      record_type               char(2)              null,
      unique_system_identifier  numeric(9,0)         not null,
      call_sign                 char(10)             null,
      location_number           int                  null,
      special_condition_type    char(1)              null,
      special_condition_code    int                  null,
      status_code		char(1)		     null,
      status_date		datetime	     null
)

go

create table dbo.PUBACC_MC
(
      record_type               		char(2)              null,
      unique_system_identifier  		numeric(9,0)         not null,
      uls_file_number           		char(14)             null,
      ebf_number                		varchar(30)          null,
      call_sign                			char(10)             null,
      partition_area_id 				numeric(9,0)         null,
      coordinate_seq_id	 				int                  null,
      partition_lat_degrees     		int                  null,
      partition_lat_minutes     		int                  null,
      partition_lat_seconds     		numeric(3,1)         null,
      partition_lat_direction   		char(1)              null,
      partition_long_degrees    		int                  null,
      partition_long_minutes    		int                  null,
      partition_long_seconds    		numeric(3,1)         null,
      partition_long_direction  		char(1)              null,
	  undefined_partitioned_area		int					 null
)


go


create table dbo.PUBACC_ME
(
      record_type              	char(2)              not null,
      unique_system_identifier  numeric(9,0)         not null,
      uls_file_number           char(14)             null,
      ebf_number              	varchar(30)          null,
      callsign                	char(10)             null,
      mea_number		 		char(6)	     		 null,
      action_performed		 	char(1)	      		 null 
)

go


create table dbo.PUBACC_MH
(
      record_type                char(2)            not null,
      unique_system_identifier 	 numeric(9,0)       not null,
      uls_file_number          	 char(14)           null,
      ebf_number               	 varchar(30)        null,
      callsign                 	 char(10)           null,
      action_performed		 	 char(1)	    	null,
      channel_plan_number	 	 char(4)	     	null,
      channel_plan		 		 char(1)			null

)


go


create table dbo.PUBACC_MF
(
      record_type               char(2)        	null,
      unique_system_identifier  numeric(9,0)   	not null,
      uls_file_number           char(14)       	null,
      ebf_number                varchar(30)    	null,
      call_sign                 char(10)        null,
      partition_area_id			numeric(9,0)	null,
      lower_frequency          	numeric(16,8)   null,
      upper_frequency          	numeric(16,8)   null,
      def_und_ind				char(1)		null,
	  defined_partition_area 	char(6)		null
)

go

create table dbo.PUBACC_MI
(
      record_type               char(2)              null,
      unique_system_identifier  numeric(9,0)         null,
      uls_file_number           char(14)             null,
      ebf_number                varchar(30)          null,
      call_sign                 char(10)             null,
      facility_type_code        char(4)              null,
      statement_of_intention    char(1)              null,
      license_type_code         char(1)              null
)

go

create table dbo.PUBACC_MK
(
      record_type               char(2)              null,
      unique_system_identifier  numeric(9,0)         not null,
      uls_file_number           char(14)             null,
      ebf_number                varchar(30)          null,
      call_sign                 char(10)             null,
      market_code               char(6)              null,
      channel_block             char(4)              null,
      submarket_code            int                  null,
      market_name               char(30)             null,
      coverage_partitioning     char(1)              null,
      coverage_dissagregation   char(1)              null,
      cellular_phase_id         smallint             null,
      population                numeric(9,0)         null,
      tribal_credit_indicator   char(1)              null,
      tribal_credit_calculation money                null,
      additional_credit_requested money              null,
      tribal_credit_awarded     money                null,
      additiona_credit_awarded  money                null,
      bc_pct			numeric(5,4)	     null,
      open_closed_bidding	char(1)		     null,
      bidding_credit_type       char(1)	             null,
      claiming_unserved_area	char(1)              null,
      incumbent_relocation      char(1)              null
)

go

create table dbo.PUBACC_MP
(
      record_type            		char(2)         null,
      unique_system_identifier 		numeric(9,0)    not null,
      uls_file_number          		char(14)        null,
      ebf_number               		varchar(30)     null,
      call_sign                 	char(10)        null,
      market_partition_code   		char(6)         null,
      defined_partition_area    	varchar(60)     null,
      defined_area_population  		numeric(9,0) 	null,
      include_exclude_ind       	char(1)         null,
      partition_area_id				numeric(9,0)    null,
      action_performed				char(1)         null,
      census_figures              	int				null,
      def_undef_ind					char(1)			null,
	  partition_sequence_number 	int 			null,
	  whitespace_ind 	        	char(1) 		null
	  
)

go

create table dbo.PUBACC_MW
(
      record_type               char(2)              not null,
      unique_system_identifier  numeric(9,0)         not null,
      uls_file_number           char(14)             null,
      ebf_number                varchar(30)          null,
      call_sign                 char(10)             null,
      pack_indicator            char(1)              null,
      pack_registration_num     int                  null,
      pack_name                 varchar(50)          null,
      type_of_operation         varchar(45)          null,
      smsa_code                 char(6)              null,
      station_class             char(4)              null,
      cum_effect_is_major       char(1)              null,
      status_code		char(1)		     null,
      status_date		datetime	     null
)

go

create table dbo.PUBACC_OP



(
      record_type               char(2)              not null,
      unique_system_identifier  numeric(9,0)         not null,
      uls_file_number           char(14)             null,
      ebf_number                varchar(30)          null,
      callsign                  char(10)             null,
      location_number           int                  null,
      area_text_sequence_num    int                  null,
      area_of_operation         varchar(255)         null,
      status_code		char(1)		     null,
      status_date		datetime	     null
)

go

create table dbo.PUBACC_PA
(
      record_type               char(2)              null,
      unique_system_identifier  numeric(9,0)         not null,
      uls_file_number           char(14)             null,
      ebf_number                varchar(30)          null,
      callsign                  char(10)             null,
      path_action_performed     char(1)              null,
      path_number               int                  null,
      transmit_location_number  int                  null,
      transmit_antenna_number   int                  null,
      receiver_location_number  int                  null,
      receiver_antenna_number   int                  null,
      mas_dems_subtype          char(2)              null,
      path_type_desc            char(20)             null,
      passive_receiver_indicator char(1)             null,
      country_code              char(3)              null,
      interference_to_gso       char(1)              null,
      receiver_callsign         varchar(10)          null,
      angular_sep		numeric(3,2)         null,
      cert_no_alternative	char(1)		     null,
      cert_no_interference	char(1)  	     null,
	  status_code			char(1)			null,
	  status_date			datetime		null,
	  link_start_date		datetime		null,
	  link_end_date			datetime		null
)

go

create table dbo.PUBACC_PC
(
      record_type               char(2)              null,
      unique_system_identifier  numeric(9,0)         not null,
      uls_file_number           char(14)             null,
      ebf_number                varchar(30)          null,
      call_sign                 char(10)             null,
      action_performed          char(1)              null,
      location_number           int                  null,
      antenna_number            int                  null,
      frequency                 numeric(16,8)        null,
      subscriber_call_sign      char(10)             null,
      city                      varchar(20)          null,
      state                     char(2)              null,
      lat_degrees               int                  null,
      lat_minutes               int                  null,
      lat_seconds               numeric(3,1)         null,
      lat_direction             char(1)              null,
      long_degrees              int                  null,
      long_minutes              int                  null,
      long_seconds              numeric(3,1)         null,
      long_direction            char(1)              null,
      point_of_com_frequency    numeric(16,8)        null,
	status_code		char(1)			null,
	status_date		datetime		null
)

go

create table dbo.PUBACC_RA
(
      record_type               char(2)              null,
      unique_system_identifier  numeric(9,0)         not null,
      uls_file_number           char(14)             null,
      ebf_number                varchar(30)          null,
      call_sign                 char(10)             null,
      radial_action_performed   char(1)              null,
      location_number           int                  null,
      antenna_number            int                  null,
      frequency_number	        int                  null,
      frequency_assigned        numeric(16,8)        null,
      frequency_upper_band      numeric(16,8)        null,
      radial_direction          numeric(3,0)         null,
      radial_haat               numeric(5,1)         null,
      radial_erp                numeric(7,3)         null,
      dist_to_sab               numeric(4,1)         null,
      dist_to_cgsa              numeric(4,1)         null,
	status_code		char(1)			null,
	status_date		datetime		null
)

go

create table dbo.PUBACC_RC
(
      record_type               char(2)              null,
      unique_system_identifier  numeric(9,0)         not null,
      uls_file_number           char(14)             null,
      ebf_number                varchar(30)          null,
      call_sign                 char(10)             null,
      radial_action_performed   char(1)              null,
      location_number           int                  null,
      antenna_number            int                  null,
      receiver_make             varchar(25)          null,
      receiver_model            varchar(25)          null,
      receiver_stability	numeric(6,5)         null,
      receiver_noise_figure	numeric(5,2)         null,
	status_code		char(1)			null,
	status_date		datetime		null

)

go

create table dbo.PUBACC_RE
(
      Record_Type               char(2)              not null,
      unique_system_identifier  numeric(9,0)         not null,
      ULS_File_Number           char(14)             null,
      EBF_Number                varchar(30)          null,
      Reason                    varchar(255)         null
)

go

create table dbo.PUBACC_RI
(
	Record_Type               	char(2)              	not null,
     	unique_system_identifier  	numeric(9,0)         	not null,
      	ULS_File_Number         	char(14)             	null,
      	EBF_Number               	varchar(30)          	null,
	Entity_Type			char(1)			not null,
	Year_Sequence_ID		smallint		not null,
        Gross_Revenues			money			null,
	Year_End_Date			datetime		not null,
	Average_Gross_Revenues		money			null,
	Asset_Disclosure		money			null,
        Statement_Type                  char(1)                 not null,
        In_Existence                    char(1)                 null
)
go


create table dbo.PUBACC_RZ
(
      record_type               char(2)              null,
      unique_system_identifier  numeric(9,0)         not null,
      uls_file_number           char(14)             null,
      ebf_number                varchar(30)          null,
      call_sign                 char(10)             null,
      antenna_action_performed  char(1)              null,
      location_number           int                  null,
      antenna_number            int                  null,
      receive_zone_number       int                  null,
      receive_zone              char(6)              null
)

go

create table dbo.PUBACC_SC
(


      record_type               char(2)              null,
      unique_system_identifier  numeric(9,0)         not null,
      uls_file_number           char(14)              null,
      ebf_number                varchar(30)           null, 
      callsign                  char(10)             null ,
      special_condition_type    char(1)              null,
      special_condition_code    int                  null,
	status_code		char(1)			null,
	status_date		datetime		null
)

go

create table dbo.PUBACC_SE

(

      record_type              char(2)               null,
      unique_system_identifier numeric(9,0)          null,
      uls_file_number          char(14)              null,
      ebf_number               varchar(30)           null,
      call_sign                char(10)              null,
      ship_call_sign           char(10)              null,
      port_registry            varchar(35)           null,
      owner                    char(1)               null,
      operater                 char(1)               null,
      charter                  char(1)               null,
      agent                    char(1)               null,
      radiotelephone_exempt_req char(1)              null,
      gmdss_exemp_req          char(1)               null,
      radio_dir_exempt_req     char(1)               null,
      prev_exempt_file_number  varchar(10)           null,
      foreign_port             char(1)               null,
      vessel_size_exempt       char(1)               null,
      equipment_exempt          char(1)               null,
      ltd_routes_exempt        char(1)               null,
      cond_voyages_exempt      char(1)               null,
      other_exempt             char(1)               null,
      other_exempt_desc        varchar(50)           null,
      ship_type                char(1)               null,
      number_of_crew           int               null,
      number_passengers        int               null,
      number_others            int               null,
      count_vhf                int               null,
      count_vhf_dsc            char(1)               null,
      count_epirb              int               null,
      count_survival           int               null,
      count_earth_station      int               null,
      count_auto_alarm         int               null,
      count_single_side_band   int               null,
      single_side_band_type_mf char(1)               null,
      single_side_band_type_hf char(1)               null,
      single_side_band_type_dsc char(1)              null,
      count_of_navtex          int               null,
      count_of_9_ghz_radar     int               null,
      count_of_500_khz_distress int              null,
      count_of_reserve_power   int               null,
      count_of_other           int               null,
      description_of_other     varchar(50)           null
)

go



create table dbo.PUBACC_SF
(
      record_type               char(2)              null ,
      unique_system_identifier  numeric(9,0)         null ,
      uls_file_number           char(14)              null,
      ebf_number                varchar(30)           null, 
      callsign                  char(10)             null ,
      lic_freeform_cond_type    char(1)              null ,
      unique_lic_freeform_id    numeric(9,0)         null ,
      sequence_number           int              null ,
      lic_freeform_condition    varchar(255)         null,
	status_code		char(1)			null,
	status_date		datetime		null
)
 

go


create table dbo.PUBACC_SG
(
      record_type               char(2)              null,
      unique_system_identifier  numeric(9,0)         not null,
      uls_file_number           char(14)             null,
      ebf_number                varchar(30)          null,
      call_sign                 char(10)             null,
      segment_action_performed  char(1)              null,
      path_number               int              null,
      transmit_location         int              null,
      transmit_antenna          int              null,
      receiver_location         int              null,
      receiver_antenna          int              null,
      segment_number            int              null,
      segment_length            numeric(12,6)        null,
	status_code		char(1)			null,
	status_date		datetime		null
)

go

create table dbo.PUBACC_SH
(
      record_type               char(2)              null,
      unique_system_identifier  numeric(9,0)         null,
      uls_file_number           char(14)             null,
      ebf_number                varchar(30)          null,
      callsign                  char(10)             null,
      type_of_authorization     char(1)              null,
      count_in_fleet            int              null,
      general_class             char(3)              null,
      special_class             char(3)              null,
      ship_name                 varchar(35)          null,
      ship_number               char(12)             null,
      international_voyages     char(1)              null,
      foreign_communications    char(1)              null,
      radiotelegraph            char(1)              null,
      mmsi_request              char(1)              null,
      gross_tonnage             int              null,
      ship_length               int              null,
      working_freq_s1           char(3)              null,
      working_freq_s2           char(3)              null,
      self_id_number            char(5)              null,
      comsat_id_number          char(7)              null,
      station_number            numeric(9,0)         null,
      required_cat_a            char(1)              null,
      required_cat_b            char(1)              null,
      required_cat_c            char(1)              null,
      required_cat_d            char(1)              null,
      required_cat_e            char(1)              null
)

go

create table dbo.PUBACC_SI
(
      record_type               char(2)              null,
      unique_system_identifier  numeric(9,0)         not null,
      uls_file_number           char(14)             null,
      ebf_number                varchar(30)          null,
      call_sign                 char(10)             null,
      sid                       char(4)              null,
      action_performed          char(1)              null
)

go

create table dbo.PUBACC_SR
(
      record_type               char(2)             null,
      unique_system_identifier	numeric(9,0)	    not null,
      uls_file_number    	char(14)            null,
      ebf_number	        varchar(30)         null,
      call_sign	                char(10)            null,
      epirb_identification_code char(15)            null,  
      INMARSAT_A            	char(1)             null,   
      INMARSAT_B                char(1)             null,     
      INMARSAT_C                char(1)             null, 
      INMARSAT_M                char(1)             null,   
      INMARSAT_Mini             char(1)             null,    
      VHF                       char(1)             null,        
      MF                        char(1)             null,          
      HF                        char(1)             null,           
      DSC                       char(1)             null,        
      epirb_406_mhz             char(1)             null,    
      epirb_121_5_mhz           char(1)             null,  
      sart                      char(1)             null,           
      raft_count                numeric(6,0)        null,  
      lifeboat_count            numeric(6,0)        null,
      vessel_capacity           numeric(6,0)        null
)

go

create table dbo.PUBACC_ST
(
      record_type               char(2)             null,
      unique_system_identifier  numeric(9,0)        not null,
      uls_file_number           char(14)            null,
      ebf_number                varchar(30)         null,
      call_sign                 char(10)            null,
      sector_action_performed   char(1)             null,
      sector_number             int                 null,
      psd_nonpsd_methodology    varchar(10)         null,
      maximum_erp               numeric(15,3)       null,
      psd_attachment            char(1)             null,
      location_number           int                 null,
      location_name             varchar(20)         null,
      status_code               char(1)             null,
      status_date               datetime            null
)

go


create table dbo.PUBACC_SV

(

      record_type               char(2)              null,
      unique_system_identifier  numeric(9,0)         null,
      uls_file_number           char(14)             null,
      ebf_number                varchar(30)          null,
      call_sign                 char(10)             null,
      voyage_number             int              null,
      voyage_description        varchar(255)         null
)

go



create table dbo.PUBACC_TA
(
      Record_Type               char(2)              null,
      unique_system_identifier  numeric(9,0)         not null,
      uls_file_number           char(14)             null,
      ebf_number                varchar(30)          null,
      pro_forma                 char(1)              null,
      full_assignment           char(1)              null,
      method_of_accomplishment  char(1)              null,
      method_other_description  varchar(30)          null,
      voluntary_involuntary     char(1)              null,
      assignor_certifier_first_name varchar(20)      null,
      assignor_certifier_mi     varchar(1)           null,
      assignor_certifier_last_name char(20)          null,
      assignor_certifier_suffix varchar(3)           null,
      assignor_certifier_title  varchar(40)          null,
      assignee_gross_rev_1      money                null,
      assignee_gross_rev_2      money                null,
      assignee_gross_rev_3      money                null,
      assignee_tot_assets       money                null,
      same_small_category       char(1)              null,
      applying_for_installments char(1)              null,
      notification_of_forebearance char(1)           null,
      wireless_need_approval    char(1)              null,
      non_wireless_need_approval char(1)             null,
      male_or_female            char(1)              null,
      african_american          char(1)              null,
      native_american           char(1)              null,
      native_pacific_islander   char(1)              null,
      asian                     char(1)              null,
      white                     char(1)              null,
      ethnicity                 char(1)              null,
      consent_date              char(10)             null,
      consummation_date         char(10)             null,
      consummation_deadline     char(10)             null,
      eligibility_category      varchar(30)          null,
      other_wireless_file_num   char(14)             null,
      occurred                  char(1)              null,
      involuntary_date          char(10)             null,
      forbearance_date          char(10)             null,
      full_partial_ind          char(1)              null,
      geo_overlap               char(1)              null,
      tenmhz_geo_overlap        char(1)              null,
      reduced_providers         char(1)              null,
      multichannel              char(1)              null,
      cable_tv                  char(1)              null,
      program_reqs              char(1)              null,
      ownership_required        char(1)              null,
      ownership_file_num        char(14)             null,
      other_wireless_app	char(1)              null,
      public_safety		char(1)              null,
      tl_short_term		char(1)              null,
      mw_link_ind		char(1)              null,
	  geo_overlap_600 char(1) null,
      rsv_spectrum_600 char(1) null,
      seek_rural_bc char(1) null,
	  PL_certification  char(1)  null,
	  PK_certification  char(1) null
)
go

create table dbo.PUBACC_TL
(
      record_type               char(2)              null,
      unique_system_identifier  numeric(9,0)         null,
      uls_file_number           char(14)             null,
      ebf_number                varchar(30)          null,
      call_sign                 char(10)             null,
      market_code               char(6)              null,
      channel_block             char(4)              null,
      action_performed          char(1)              null,
      tribal_land_name          varchar(80)          null,
      tribal_certification      char(1)              null,
      tribal_land_type          varchar(10)          null,
      square_kilometers         numeric(8,0)         null
)

go

create table dbo.PUBACC_UA
(
      record_type               char(2)              not null,
      unique_system_identifier  numeric(9,0)         null,
      uls_file_number           char(14)             null,
      ebf_number                varchar(30)          null,
      call_sign                 char(10)             null,
      action_performed          char(1)              null,
      cellular_phase            int                  null,
      market_code               char(6)              null,
      submarket_code            int                  null,
      channel_block             char(4)              null,
      claiming_unserved_area    char(1)              null
)

go

create table dbo.PUBACC_VC
(


      record_type               char(2)              null,
      unique_system_identifier  numeric(9,0)         not null,
      uls_file_number           char(14)             null,
      ebf_number                varchar(30)          null,
      request_sequence          int              null,
      callsign_requested        char(10)             null
)

go


create table dbo.PUBACC_LC
(
      record_type               char(2)              null,
      unique_system_identifier  numeric(9,0)         not null,
      uls_file_number           char(14)             null,
      ebf_number                varchar(30)          null,
      call_sign			char(10)             null,		
      a_ls_class_code		char(2)              null,		
      a_ls_allocation_type	char(1)              null, 
      a_ls_term			char(1)              null
      
)

go


create table dbo.PUBACC_LD
(
      record_type               char(2)              null,
      unique_system_identifier  numeric(9,0)         not null,
      uls_file_number           char(14)             null,
      ebf_number                varchar(30)          null,
      lease_id			char(10)             null,		
      issue_date                datetime	     null,
      expired_date              datetime             null, 
      cancellation_date         datetime             null,
      lease_never_comm_ind      char(1)              null
      
)

go


create table dbo.PUBACC_LL
(
      record_type               char(2)              null,
      unique_system_identifier  numeric(9,0)         not null,
      uls_file_number           char(14)             null,
      ebf_number                varchar(30)          null,
      call_sign			char(10)             null,		
      lease_id			char(10)             null,      
      unique_system_identifier_2   numeric(9,0)         null     /*(the licensee) */
)

go

create table dbo.PUBACC_L3
(
      record_type               char(2)              not null,
      unique_system_identifier  numeric(9,0)         not null,
      uls_file_number           char(14)             null,
      ebf_number                varchar(30)          null,
      call_sign                 char(10)             null,
      lease_id                 char(10)             null, 
      ls_site_link_id           numeric(9,0)         null,  
      location_action_performed char(1)              null,
      location_type_code        char(1)              null,
      location_class_code       char(1)              null,
      location_number           int              null,
      site_status               char(1)              null,
      corresponding_fixed_location int           null,
      location_address          varchar(80)          null,
      location_city             char(20)             null,
      location_county           varchar(60)          null,
      location_state            char(2)              null,
      radius_of_operation       numeric(5,1)         null,
      area_of_operation_code    char(1)              null,
      clearance_indicator       char(1)              null,
      ground_elevation          numeric(7,1)         null,
      lat_degrees               int              null,
      lat_minutes               int              null,
      lat_seconds               numeric(3,1)         null,
      lat_direction             char(1)              null,
      long_degrees              int              null,
      long_minutes              int              null,
      long_seconds              numeric(3,1)         null,
      long_direction            char(1)              null,
      max_lat_degrees           int              null,
      max_lat_minutes           int              null,
      max_lat_seconds           numeric(3,1)         null,
      max_lat_direction         char(1)              null,
      max_long_degrees          int              null,
      max_long_minutes          int              null,
      max_long_seconds          numeric(3,1)         null,
      max_long_direction        char(1)              null,
      nepa                      char(1)              null,
      quiet_zone_notification_date char(10)          null,
      tower_registration_number char(10)             null,
      height_of_support_structure numeric(7,1)       null,
      overall_height_of_structure numeric(7,1)       null,
      structure_type            char(7)              null,
      airport_id                char(4)              null,
      location_name             char(20)             null,
      units_hand_held           int              null,
      units_mobile              int              null,
      units_temp_fixed          int              null,
      units_aircraft            int              null,
      units_itinerant           int              null,
      status_code		char(1)			null,
      status_date		datetime		null
)

go


create table dbo.PUBACC_L4
(
      record_type               char(2)              not null,
      unique_system_identifier  numeric(9,0)         not null,
      uls_file_number           char(14)             null,
      ebf_number                varchar(30)          null,
      call_sign                 char(10)             null,
     lease_id                 char(10)             null, 
      ls_site_link_id           numeric(9,0)         null, 
      location_action_performed	char(1)              null,
      location_number           int              null,
      registration_required	char(1)              null,
      protection_date		datetime             null,
      link_reg_num              varchar(30)          null,
      link_reg_action_performed char(1)              null,
      mexico_clearance_indicator char(1)	     null,
      quiet_zone_consent        char(1)	             null,	
	status_code		char(1)		     null,
	status_date		datetime	     null
	
)

go


create table dbo.PUBACC_O2

(
      record_type               char(2)              not null,
      unique_system_identifier  numeric(9,0)         not null,
      uls_file_number           char(14)             null,
      ebf_number                varchar(30)          null,
      callsign                  char(10)             null,
     lease_id                 char(10)             null, 
      ls_site_link_id           numeric(9,0)         null, 
      location_number           int              null,
      area_text_sequence_num    int              null,
      area_of_operation         varchar(255)         null,
      status_code		char(1)			null,
      status_date		datetime		null
)

go



create table dbo.PUBACC_L5
(
      record_type               char(2)              null,
      unique_system_identifier  numeric(9,0)         not null,
      call_sign                 char(10)             null,
      uls_file_number           char(14)             null,
      ebf_number                varchar(30)          null,
      lease_id                 char(10)             null, 
      ls_site_link_id           numeric(9,0)         null, 
      location_number           int              null,
      special_condition_type    char(1)              null,
      special_condition_code    int              null,
      status_code		char(1)			null,
      status_date		datetime		null
)

go


create table dbo.PUBACC_L6
 (
      record_type               char(2)              null ,
      unique_system_identifier  numeric(9,0)         null ,
      callsign                  char(10)             null ,
      uls_file_number           char(14)             null,
      ebf_number                varchar(30)          null,
      lease_id                 char(10)             null, 
      ls_site_link_id           numeric(9,0)         null, 
      location_number           int              null ,
      loc_freeform_cond_type    char(1)              null ,
      unique_loc_freeform_id    numeric(9,0)         null ,
      sequence_number           int              null ,
      loc_freeform_condition    varchar(255)         null,
      status_code		char(1)			null,
      status_date		datetime		null
)

go


create table dbo.PUBACC_A3
(
      record_type              	 char(2)              null,
      unique_system_identifier  numeric(9,0)         not null,
      uls_file_number           char(14)             null,
      ebf_number                varchar(30)          null,
      call_sign                	char(10)             null,
      lease_id                 char(10)             null, 
      ls_site_link_id           numeric(9,0)         null, 
      antenna_action_performed  char(1)              null,
      antenna_number            int              null,
      location_number          	int              null,
      receive_zone_code         char(6)              null,
      antenna_type_code         char(1)              null,
      height_to_tip            	numeric(5,1)         null,
      height_to_center_raat     numeric(5,1)         null,
      antenna_make              varchar(25)          null,
      antenna_model             varchar(25)          null,
      tilt                      numeric(3,1)         null,
      polarization_code         char(5)              null,
      beamwidth                 numeric(4,1)         null,
      gain                      numeric(4,1)         null,
      azimuth                   numeric(4,1)         null,
      height_above_avg_terrain  numeric(5,1)         null,
      diversity_height          numeric(5,1)         null,
      diversity_gain            numeric(4,1)         null,
      diversity_beam            numeric(4,1)         null,
      reflector_height          numeric(5,1)         null,
      reflector_width           numeric(4,1)         null,
      reflector_separation      numeric(5,1)         null,
      repeater_seq_num          int              null,
      back_to_back_tx_dish_gain numeric(4,1)         null,
      back_to_back_rx_dish_gain numeric(4,1)         null,
      location_name             varchar(20)          null,
      passive_repeater_id       int              null,
      alternative_cgsa_method   char(1)              null,
      path_number               int              null,
      line_loss                 numeric(3,1)         null,
	status_code		char(1)			null,
	status_date		datetime		null      
)

go

create table dbo.PUBACC_F3
(
      record_type               char(2)              null,
      unique_system_identifier  numeric(9,0)         not null,
      uls_file_number           char(14)             null,
      ebf_number                varchar(30)          null,
      call_sign                 char(10)             null,
      lease_id                 char(10)             null, 
      ls_site_link_id           numeric(9,0)         null, 
      frequency_action_performed char(1)             null,
      location_number           int              null,
      antenna_number            int              null,
      class_station_code        char(4)              null,
      op_altitude_code          char(2)              null,
      frequency_assigned        numeric(16,8)        null,
      frequency_upper_band      numeric(16,8)        null,
      frequency_carrier         numeric(16,8)        null,
      time_begin_operations     int              null,
      time_end_operations       int              null,
      power_output              numeric(15,3)        null,
      power_erp                 numeric(15,3)        null,
      tolerance                 numeric(6,5)         null,
      frequency_ind             char(1)              null,
      status                    char(1)              null,
      eirp                      numeric(7,1)         null,
      transmitter_make          varchar(25)          null,
      transmitter_model         varchar(25)          null,
      auto_transmitter_power_control char(1)         null,
      cnt_mobile_units          int              null,
      cnt_mob_pagers            int              null,
      freq_seq_id               int              null,
	status_code		char(1)			null,
	status_date		datetime		null      

)

go


create table dbo.PUBACC_F4
(
      record_type               char(2)              null,
      unique_system_identifier  numeric(9,0)         not null,
      uls_file_number           char(14)             null,
      ebf_number                varchar(30)          null,
      call_sign                 char(10)             null,
      lease_id                 char(10)              null, 
      ls_site_link_id           numeric(9,0)         null, 
      action_performed          char(1)              null, 
      location_number           int                  null,
      antenna_number            int		     null,
      frequency_number          int                  null,
      frequency_assigned        numeric(16,8)        null,
      frequency_upper_band      numeric(16,8)        null,
      frequency_offset          char(3)              null,
      frequency_channel_block   char(4)              null,
      equipment_class	        char(2)              null, 
      minimum_power_output	numeric(15,3)        null,
      date_first_use            datetime             null,
	status_code		char(1)		     null,
	status_date		datetime	     null
)

go


create table dbo.PUBACC_F5
(
      record_type               char(2)              null,
      unique_system_identifier  numeric(9,0)         not null,
      call_sign                 char(10)             null,
      lease_id                 char(10)             null, 
      ls_site_link_id           numeric(9,0)         null, 
      location_number           int              null,
      antenna_number            int              null,
      frequency                 numeric(16,8)        null,
      frequency_number          int              null,
      special_condition_type    char(1)              null,
      special_condition_code    int              null,
	status_code		char(1)			null,
	status_date		datetime		null
      
)

go


create table dbo.PUBACC_F6
(
      record_type               char(2)              null ,
      unique_system_identifier  numeric(9,0)         null ,
      uls_file_number           char(14)             null,
      ebf_number                varchar(30)          null,
      callsign                  char(10)             null ,
      lease_id                 char(10)             null, 
      ls_site_link_id           numeric(9,0)         null, 
      location_number           int              null ,
      antenna_number            int              null ,
      frequency_number          int              null ,
      frequency                numeric(16,8)         null ,
      freq_freeform_cond_type   char(1)              null ,
      unique_freq_freeform_id   numeric(9,0)         null ,
      sequence_number           int              null ,
      freq_freeform_condition   varchar(255)         null,
      status_code		char(1)			null,
      status_date		datetime		null
)
 
go


create table dbo.PUBACC_P2
(
      record_type               char(2)              null,
      unique_system_identifier  numeric(9,0)         not null,
      uls_file_number           char(14)             null,
      ebf_number                varchar(30)          null,
      callsign                  char(10)             null,
      lease_id                 char(10)              null, 
      ls_site_link_id           numeric(9,0)         null, 
      path_action_performed     char(1)              null,
      path_number               int                  null,
      transmit_location_number  int                  null,
      transmit_antenna_number   int                  null,
      receiver_location_number  int                  null,
      receiver_antenna_number   int                  null,
      mas_dems_subtype          char(2)              null,
      path_type_desc            char(20)             null,
      passive_receiver_indicator char(1)             null,
      country_code              char(3)              null,
      interference_to_gso       char(1)              null,
      receiver_callsign         varchar(10)          null,
      angular_sep		numeric(3,2)         null,
      cert_no_alternative	char(1)		     null,
      cert_no_interference	char(1)  	     null,
	status_code		char(1)			null,
	status_date		datetime		null
)

go

create table dbo.PUBACC_TP
(
      record_type               char(2)              null,
      unique_system_identifier  numeric(9,0)         not null,
      uls_file_number           char(14)             null,
      ebf_number                varchar(30)          null,
      call_sign                 char(10)             null,
      location_number           integer              null,
      antenna_number            integer              null,
      frequency_number          integer              null,
      freq_proto_seq_id         integer              null,
      freq_proto_code_desc	varchar(40)	     not null,
      a_freq_proto_other        varchar(255)         null,
      action_performed          char(1)              null, 
      status_code		char(1)			null,
      status_date		datetime		null
)

go


A  => modules/__init__.py +0 -0
A  => modules/uls.py +273 -0
@@ 1,273 @@
import os
import re
import csv
import tempfile

try:
    from sqlite_utils import Database
except ImportError:
    print("Please install sqlite-utils with: pip install sqlite-utils")
    exit(1)


def parse_definitions_file(definitions_file, encoding="utf-8"):
    """
    Parses SQL file and generates SQLite schema
    :param definitions_file: valid ULS definitions file
    :param encoding: file encoding
    :returns: SQLite schema
    """

    # Read valid ULS definitions file
    with open(definitions_file, "r", encoding=encoding) as f:
        sql_text = f.read()

    # Define regex patterns
    table_pattern = re.compile(r"create table (dbo\..*?)\n(.*?)\n\)", re.DOTALL)
    column_pattern = re.compile(r"^\s*([a-zA-Z_][a-zA-Z0-9_]*)\s.*$", re.MULTILINE)

    # Extract table definitions
    table_matches = table_pattern.findall(sql_text)

    # Build dictionary of table definitions
    table_definitions = {}
    for table_match in table_matches:
        table_name = table_match[0]
        column_definitions = column_pattern.findall(table_match[1])
        table_definitions[table_name] = column_definitions

    # Transform into SQLite schema definitions
    sqlite_schemas = {}
    for table_name, column_names in table_definitions.items():
        sqlite_schemas[table_name] = ", ".join(column_names)

    return sqlite_schemas


def get_record_types_from_table_names(table_names):
    """
    Returns a list of record types from a list of table names
    :param table_names: list of ULS record types
    :returns: list of record types
    """
    record_types = [table_name.split("_")[-1].upper() for table_name in table_names]

    return record_types


def add_definitions_to_file(filename, record_type, definitions, encoding="utf-8"):
    """
    Adds definitions as header to a file
    :param filename: file to add definitions header to
    :param record_type: ULS record type
    :param definitions: ULS definitions
    :param encoding: file encoding
    """

    # Generate file path
    file_path = filename

    # Split definitions into list
    headers = definitions.split(", ")

    # Check if file already exists
    if os.path.isfile(file_path):
        # Read existing data into a list
        with open(file_path, "r", encoding=encoding) as f:
            existing_data = f.readlines()

        # If the file only contains headers, return without overwriting
        if len(existing_data) <= 1:
            return

        # Write headers and existing data to a temporary list
        data_to_write = [headers]

        for line in existing_data:
            data_to_write.append(line.strip().split("|"))  # should set to var

        # Write the combined data back to the file
        with open(file_path, "w", newline="", encoding=encoding) as f:
            writer = csv.writer(f, delimiter="|")  # should set to var
            writer.writerows(data_to_write)
    else:
        # Write headers to new file
        with open(file_path, "w", newline="", encoding=encoding) as f:
            writer = csv.writer(f, delimiter="|")  # should set to var
            writer.writerow(headers)


def insert_csv_to_sqlite(
    db_name, table_name, csv_filename, delimiter="|", encoding="utf-8"
):
    """
    Inserts data from a CSV file into a SQLite database using the sqlite-utils Python module
    :param db_name: Database file name (e.g. test.db)
    :param table_name: SQLite database table name (the record type)
    :param csv_filename: CSV file to import to the SQLite database
    :param delimiter: Delimiter used for the file, default is pipe
    :param encoding: file encoding, default is utf-8
    """

    # Open the CSV file and read data
    with open(csv_filename, "r", encoding=encoding) as csv_file:
        reader = csv.reader(csv_file, delimiter=delimiter)
        data_list = list(reader)

    # Convert list of lists to list of dictionaries for insert_all function
    headers = data_list.pop(0)
    data_dicts = [dict(zip(headers, row)) for row in data_list]

    # Open the SQLite database
    db = Database(db_name)

    # Create the table if not exists and insert data
    table = db[table_name]
    table.insert_all(data_dicts)


def find_definitions_file():
    """
    Checks the definitions sub-directory for definitions file and returns path
    :returns: file path
    """
    search_path = os.path.join(os.getcwd(), "definitions")
    for root, dirs, files in os.walk(search_path):
        for file in files:
            if file.endswith(".txt"):
                return os.path.join(root, file)

    raise FileNotFoundError("No .txt file found in the 'definitions' directory")


def splitter(file, valid_record_types, delimiter="|", file_encoding="windows-1252"):
    """
    Takes a 'combined' type file and splits it into files according to the
    record type. For example, if a file contains records HD and AC, two files
    will be created and filled with only those record types.
    :param file: the file to split
    :param delimiter: delimiter to split on, defaults to pipe
    :param file_encoding: the file encoding, defaults to windows-1252
    :returns: Boolean
    """
    files = {}
    with open(file, "r", encoding=file_encoding) as in_file:
        for i, line in enumerate(in_file, start=1):
            fields = line.strip().split(delimiter)
            record_type = fields[0]

            # If the record type is not two letters, or not in valid record types, skip this line
            if (
                len(record_type) != 2
                or not record_type.isalpha()
                or record_type not in valid_record_types
            ):
                continue

            # If we haven't seen this record type before, open a new file
            if record_type not in files:
                # Create the "../temp" directory if it doesn't exist
                temp_path = os.path.join(".", "temp")
                if not os.path.exists(temp_path):
                    os.mkdir(temp_path)

                # Join the directory name with the filename
                filepath = os.path.join(temp_path, f"{record_type}.csv")

                # Open the file in the specified directory
                files[record_type] = open(filepath, "w", encoding=file_encoding)

            # Write the line to the appropriate file
            files[record_type].write(line)

    return files


def check_type(filename, valid_record_types, encoding="utf-8"):
    """
    Scans the file to determine what type of database is being used
    :param filename: the file to check
    :param valid_record_types: the valid ULS record types
    :param encoding: the file encoding
    :returns: type "individual" or "combined"
    """
    unique_values = set()

    with open(filename, "r", encoding=encoding) as file:
        for line in file:
            columns = line.strip().split("|")
            first_column = columns[0]

            # Check if the first column is one of the valid record types
            if first_column in valid_record_types:
                unique_values.add(first_column)

    if len(unique_values) == 1:
        return "individual"
    elif len(unique_values) > 1:
        return "combined"
    else:
        raise Exception("No valid lines found")


def validate_record_types(definitions_file, file_encoding):
    """
    Returns a list of valid record types
    :param definitions_file: the ULS definitions file
    :param file_encoding: the file encoding
    :returns: list of valid record types
    """
    sqlite_schemas = parse_definitions_file(definitions_file, file_encoding)
    valid_record_types = get_record_types_from_table_names(sqlite_schemas.keys())
    return valid_record_types


def export_file(filename, record_type, db_name, file_encoding="windows-1252"):
    """
    Processes a single pipe-delimited database file
    :param filename: the file to process
    :param record_type: the ULS record type
    :param db_name: the ULS database name (e.g. test.db)
    :param file_encoding: the file encoding
    """

    # temporary, this is redundant
    definitions_file = find_definitions_file()
    sqlite_schemas = parse_definitions_file(definitions_file, file_encoding)
    definitions = sqlite_schemas.get(
        f"dbo.PUBACC_{record_type}", "Record type not found"
    )

    # changed from 'definitions' to 'definitions_file'
    add_definitions_to_file(filename, record_type, definitions, file_encoding)
    insert_csv_to_sqlite(db_name, record_type, filename, "|", file_encoding)


def process_file(in_file, db_name, delimiter, file_encoding="windows-1252"):
    """
    Read in a pipe-delimited input file and identify valid lines, then write
    them to a temporary file which is then sent to the export_file() function
    for writing to DB
    :param in_file: the input file
    :param db_name: the ULS database name (e.g. test.db)
    :param delimiter: the delimiter to use
    :param file_encoding: the file encoding
    """

    record_type = os.path.splitext(os.path.basename(in_file))[0]
    definitions_file = find_definitions_file()
    valid_record_types = validate_record_types(definitions_file, file_encoding)

    with tempfile.NamedTemporaryFile(
        suffix=".csv", encoding=file_encoding, mode="w", delete=False
    ) as csv_file:
        with open(in_file, "r", encoding=file_encoding) as file:
            for line in file:
                columns = line.strip().split("|")
                first_column = columns[0]
                # Check if the first column is one of the valid record types
                if first_column in valid_record_types:
                    csv_file.write(line)
    export_file(csv_file.name, record_type, db_name, file_encoding)  # export to DB
    csv_file.close()  # close the file
    os.remove(csv_file.name)  # delete the temporary file

A  => process_pipe_files.sh +17 -0
@@ 1,17 @@
#!/bin/bash

# Check if the database filename is provided
if [ -z "$1" ]; then
  echo "Please provide the database filename as the first argument."
  exit 1
fi

# Store the provided database filename
db_filename="$1"

# Loop through all .CSV files in the "temp" sub-directory
for csv_file in ./temp/*.csv; do
  # Run the Python command with the current CSV file as an input and the provided database filename as an output
  python uls2sqlite.py -i "$csv_file" -o "$db_filename" --encoding=windows-1252
  rm $csv_file # Remove the CSV file that has been processed and is no longer needed
done

A  => process_zip_files.sh +81 -0
@@ 1,81 @@
#!/bin/bash

# Check if GNU Parallel is installed
if command -v parallel > /dev/null 2>&1; then
    USE_PARALLEL=true
else
    USE_PARALLEL=false
    echo "GNU Parallel not found. Falling back to sequential processing."
fi

process_dat() {
    zipfile="$1"
    datfile="$2"
    tempdir="$3"
    
    # Assuming zipfile is the variable containing the path to the ZIP file, e.g., "archives/l_coast.zip"
    zipfile_basename=$(basename "$zipfile" .zip) # Extracts "l_coast" from "archives/l_coast.zip"
    sqlite_filename="${zipfile_basename}.sqlite" # Constructs "l_coast.sqlite"

    # Process the .DAT file to append data to the .sqlite file
    python uls2sqlite.py -i "$datfile" -o "$tempdir/$sqlite_filename" --encoding=utf-8

    # Remove the processed DAT file
    rm "$datfile"
}

process_zip() {
    zipfile="$1"

    # Create a temporary directory for this zipfile
    tempdir=$(mktemp -d)

    # Extract .DAT files from the ZIP file into the temporary directory
    unzip -j "$zipfile" -d "$tempdir" '*.dat'
    
    if [ $? -ne 0 ]; then
        echo "Failed to extract from $zipfile"
        return
    fi
    
    # Prepare the DAT files for processing
    python tools/ready_dat_files.py "$tempdir"
    
    if [ $? -ne 0 ]; then
        echo "Failed to prepare .DAT files for processing"
        return
    fi

    # Process .DAT files sequentially
    for datfile in "$tempdir"/*.dat; do
        process_dat "$zipfile" "$datfile" "$tempdir"
    done

    # Move the .sqlite file to the databases directory
    mv "$tempdir"/*.sqlite ./databases/

    # Remove the temporary directory
    rm -r "$tempdir"
}

export -f process_dat
export -f process_zip

# Ensure the databases directory exists
mkdir -p databases

#
# If using GNU Parallel, it will spawn a "job" per available CPU core.
# This can consume a great deal of system memory. If you are at all 
# resource constrained, it is recommended to run this sequentially.
#

if $USE_PARALLEL; then
    # Use GNU Parallel to process ZIP files in parallel
    ls archives/*.zip | parallel process_zip
else
    # Sequential processing
    for zipfile in archives/*.zip; do
        process_zip "$zipfile"
    done
fi

A  => requirements.txt +1 -0
@@ 1,1 @@
sqlite-utils
\ No newline at end of file

A  => temp/README.txt +1 -0
@@ 1,1 @@
Temporary files generated by tools stored here.
\ No newline at end of file

A  => tools/download_weekly_dbs.sh +20 -0
@@ 1,20 @@
#!/bin/bash

# Create the archives directory if it doesn't exist
mkdir -p ../archives

for url in $(cat weekly_db_urls.txt); do
    # Extract the filename from the URL
    filename=$(basename "$url")

    # Download the file to the archives sub-directory using curl
    # -s: Silent mode, don't show progress or error messages
    # -S: Show error messages if an error occurs
    # -L: Follow redirects
    # -o: Specify output file
    # --progress-bar: Show download progress
    curl -s -S -L --progress-bar "$url" -o "../archives/$filename"

    # Sleep for a random time between 30 seconds and 120 seconds (2 minutes)
    sleep $((30 + RANDOM % 91))
done

A  => tools/ready_dat_files.py +18 -0
@@ 1,18 @@
'''
This small script can be used to prep .DAT files for use with the ULS2SQLite tool. It will convert all .DAT filenames to upper-case, and
convert the file extensions to lower-case.
'''

import os
import argparse

parser = argparse.ArgumentParser(description='Process .DAT files in the specified directory.')
parser.add_argument('directory', type=str, help='The directory to process .DAT files in.')
    
args = parser.parse_args()

for filename in os.listdir(args.directory):
    if filename.endswith('.DAT') or filename.endswith('.dat'):
        base, ext = os.path.splitext(filename)
        new_name = base.upper() + ext.lower()
        os.rename(os.path.join(args.directory, filename), os.path.join(args.directory, new_name))

A  => tools/uls_splitter.py +39 -0
@@ 1,39 @@
#!/usr/bin/env python

import argparse
from modules import uls

# Define command-line argument parser
parser = argparse.ArgumentParser(description="Splits combined ULS files")
# Argument for import/input file
parser.add_argument(
    "-i",
    "--in_file",
    metavar="<filename>",
    type=str,
    help="Valid ULS pipe-delimited file to import",
)

args = parser.parse_args()

"""
The combined files seem to be in windows-1252 encoding, so we'll use that
as the default
"""
file_encoding = "windows-1252"

definitions_file = uls.find_definitions_file()

valid_record_types = uls.validate_record_types(definitions_file, file_encoding)

file_type = uls.check_type(args.in_file, valid_record_types, file_encoding)

if file_type == "individual":
    print("individual")
elif file_type == "combined":
    uls.splitter(
        args.in_file, valid_record_types, delimiter="|", file_encoding="windows-1252"
    )
    print("combined")
else:
    raise Exception("No valid record types found")

A  => tools/weekly_db_urls.txt +13 -0
@@ 1,13 @@
https://data.fcc.gov/download/pub/uls/complete/l_aircr.zip
https://data.fcc.gov/download/pub/uls/complete/l_amat.zip
https://data.fcc.gov/download/pub/uls/complete/l_mdsitfs.zip
https://data.fcc.gov/download/pub/uls/complete/l_cell.zip
https://data.fcc.gov/download/pub/uls/complete/l_frc.zip
https://data.fcc.gov/download/pub/uls/complete/l_gmrs.zip
https://data.fcc.gov/download/pub/uls/complete/l_LMbcast.zip
https://data.fcc.gov/download/pub/uls/complete/l_LMcomm.zip
https://data.fcc.gov/download/pub/uls/complete/l_LMpriv.zip
https://data.fcc.gov/download/pub/uls/complete/l_coast.zip
https://data.fcc.gov/download/pub/uls/complete/l_micro.zip
https://data.fcc.gov/download/pub/uls/complete/l_paging.zip
https://data.fcc.gov/download/pub/uls/complete/l_ship.zip
\ No newline at end of file

A  => uls2sqlite.py +45 -0
@@ 1,45 @@
#!/usr/bin/env python

import argparse
import os
import tempfile
from modules import uls

# Define command-line argument parser
parser = argparse.ArgumentParser(description="Example program with -file option")
# Argument for import/input file
parser.add_argument(
    "-i",
    "--in_file",
    metavar="<filename>",
    type=str,
    help="Valid FCC ULS pipe-delimited file to import",
)
# Argument for export file
parser.add_argument(
    "-o",
    "--out_file",
    metavar="<filename>",
    type=str,
    help="SQLite database file to export data to",
)
# Argument to change the file encoding
parser.add_argument(
    "-e",
    "--encoding",
    help="File encoding",
)

args = parser.parse_args()

file_encoding = "utf-8"  # "windows-1252" and "utf-8" have been observed

if args.encoding:
    file_encoding = args.encoding

db_name = args.out_file
delimiter = "|"

# Process the ULS file and export it to an SQLite database file
if args.in_file:
    uls.process_file(args.in_file, db_name, delimiter, file_encoding)