~sschwarzer/ftputil

ref: c6d0136bf66260c302304a9c53848e08337d405d ftputil/doc/ftputil.txt -rw-r--r-- 62.9 KiB
c6d0136bStefan Schwarzer Mention that `account` and `session_factory` normally aren't needed 6 years ago
                                                                                
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
1001
1002
1003
1004
1005
1006
1007
1008
1009
1010
1011
1012
1013
1014
1015
1016
1017
1018
1019
1020
1021
1022
1023
1024
1025
1026
1027
1028
1029
1030
1031
1032
1033
1034
1035
1036
1037
1038
1039
1040
1041
1042
1043
1044
1045
1046
1047
1048
1049
1050
1051
1052
1053
1054
1055
1056
1057
1058
1059
1060
1061
1062
1063
1064
1065
1066
1067
1068
1069
1070
1071
1072
1073
1074
1075
1076
1077
1078
1079
1080
1081
1082
1083
1084
1085
1086
1087
1088
1089
1090
1091
1092
1093
1094
1095
1096
1097
1098
1099
1100
1101
1102
1103
1104
1105
1106
1107
1108
1109
1110
1111
1112
1113
1114
1115
1116
1117
1118
1119
1120
1121
1122
1123
1124
1125
1126
1127
1128
1129
1130
1131
1132
1133
1134
1135
1136
1137
1138
1139
1140
1141
1142
1143
1144
1145
1146
1147
1148
1149
1150
1151
1152
1153
1154
1155
1156
1157
1158
1159
1160
1161
1162
1163
1164
1165
1166
1167
1168
1169
1170
1171
1172
1173
1174
1175
1176
1177
1178
1179
1180
1181
1182
1183
1184
1185
1186
1187
1188
1189
1190
1191
1192
1193
1194
1195
1196
1197
1198
1199
1200
1201
1202
1203
1204
1205
1206
1207
1208
1209
1210
1211
1212
1213
1214
1215
1216
1217
1218
1219
1220
1221
1222
1223
1224
1225
1226
1227
1228
1229
1230
1231
1232
1233
1234
1235
1236
1237
1238
1239
1240
1241
1242
1243
1244
1245
1246
1247
1248
1249
1250
1251
1252
1253
1254
1255
1256
1257
1258
1259
1260
1261
1262
1263
1264
1265
1266
1267
1268
1269
1270
1271
1272
1273
1274
1275
1276
1277
1278
1279
1280
1281
1282
1283
1284
1285
1286
1287
1288
1289
1290
1291
1292
1293
1294
1295
1296
1297
1298
1299
1300
1301
1302
1303
1304
1305
1306
1307
1308
1309
1310
1311
1312
1313
1314
1315
1316
1317
1318
1319
1320
1321
1322
1323
1324
1325
1326
1327
1328
1329
1330
1331
1332
1333
1334
1335
1336
1337
1338
1339
1340
1341
1342
1343
1344
1345
1346
1347
1348
1349
1350
1351
1352
1353
1354
1355
1356
1357
1358
1359
1360
1361
1362
1363
1364
1365
1366
1367
1368
1369
1370
1371
1372
1373
1374
1375
1376
1377
1378
1379
1380
1381
1382
1383
1384
1385
1386
1387
1388
1389
1390
1391
1392
1393
1394
1395
1396
1397
1398
1399
1400
1401
1402
1403
1404
1405
1406
1407
1408
1409
1410
1411
1412
1413
1414
1415
1416
1417
1418
1419
1420
1421
1422
1423
1424
1425
1426
1427
1428
1429
1430
1431
1432
1433
1434
1435
1436
1437
1438
1439
1440
1441
1442
1443
1444
1445
1446
1447
1448
1449
1450
1451
1452
1453
1454
1455
1456
1457
1458
1459
1460
1461
1462
1463
1464
1465
1466
1467
1468
1469
1470
1471
1472
1473
1474
1475
1476
1477
1478
1479
1480
1481
1482
1483
1484
1485
1486
1487
1488
1489
1490
1491
1492
1493
1494
1495
1496
1497
1498
1499
1500
1501
1502
1503
1504
1505
1506
1507
1508
1509
1510
1511
1512
1513
1514
1515
1516
1517
1518
1519
1520
1521
1522
1523
1524
1525
1526
1527
1528
1529
1530
1531
1532
1533
1534
1535
1536
1537
1538
1539
1540
1541
1542
1543
1544
1545
1546
1547
1548
1549
1550
1551
1552
1553
1554
1555
1556
1557
1558
1559
1560
1561
1562
1563
1564
1565
1566
1567
1568
1569
1570
1571
1572
1573
1574
1575
1576
1577
1578
1579
1580
1581
1582
1583
1584
1585
1586
1587
1588
1589
1590
1591
1592
1593
1594
1595
1596
1597
1598
1599
1600
1601
1602
1603
1604
1605
1606
1607
1608
1609
1610
1611
1612
1613
1614
1615
1616
1617
1618
1619
1620
1621
1622
1623
1624
1625
1626
1627
1628
1629
1630
1631
1632
1633
1634
1635
1636
1637
1638
1639
1640
1641
1642
1643
1644
1645
1646
1647
1648
1649
1650
1651
1652
1653
1654
1655
1656
1657
1658
1659
1660
1661
1662
1663
1664
1665
1666
1667
1668
1669
1670
1671
1672
1673
1674
1675
1676
1677
1678
1679
1680
1681
1682
1683
1684
1685
1686
1687
1688
1689
1690
1691
1692
1693
1694
1695
1696
1697
1698
1699
1700
1701
1702
1703
1704
1705
1706
1707
1708
1709
1710
1711
1712
1713
1714
1715
1716
1717
1718
1719
1720
1721
1722
1723
1724
1725
1726
1727
1728
1729
1730
1731
1732
1733
1734
1735
1736
``ftputil`` -- a high-level FTP client library
==============================================

:Version:   3.2
:Date:      2014-10-12
:Summary:   high-level FTP client library for Python
:Keywords:  FTP, ``ftplib`` substitute, virtual filesystem, pure Python
:Author:    Stefan Schwarzer <sschwarzer@sschwarzer.net>

.. contents::


Introduction
------------

The ``ftputil`` module is a high-level interface to the ftplib_
module. The `FTPHost objects`_ generated from it allow many operations
similar to those of os_, `os.path`_ and `shutil`_.

.. _ftplib: https://docs.python.org/library/ftplib.html
.. _os: https://docs.python.org/library/os.html
.. _`os.stat`: https://docs.python.org/library/os.html#os.stat
.. _`os.path`: https://docs.python.org/library/os.path.html
.. _`shutil`: https://docs.python.org/library/shutil.html

Example::

    import ftputil

    # Download some files from the login directory.
    with ftputil.FTPHost("ftp.domain.com", "user", "password") as ftp_host:
        names = ftp_host.listdir(ftp_host.curdir)
        for name in names:
            if ftp_host.path.isfile(name):
                ftp_host.download(name, name)  # remote, local
        # Make a new directory and copy a remote file into it.
        ftp_host.mkdir("newdir")
        with ftp_host.open("index.html", "rb") as source:
            with ftp_host.open("newdir/index.html", "wb") as target:
                ftp_host.copyfileobj(source, target)  # similar to shutil.copyfileobj

Also, there are `FTPHost.lstat`_ and `FTPHost.stat`_ to request size and
modification time of a file. The latter can also follow links, similar
to `os.stat`_. `FTPHost.walk`_ and `FTPHost.path.walk`_ work, too.


``ftputil`` features
--------------------

* Method names are familiar from Python's ``os``, ``os.path`` and
  ``shutil`` modules. For example, use ``os.path.join`` to join
  paths for a local file system and ``ftp_host.path.join`` to join
  paths for a remote FTP file system.

* Remote file system navigation (``getcwd``, ``chdir``)

* Upload and download files (``upload``, ``upload_if_newer``,
  ``download``, ``download_if_newer``)

* Time zone synchronization between client and server (needed
  for ``upload_if_newer`` and ``download_if_newer``)

* Create and remove directories (``mkdir``, ``makedirs``, ``rmdir``,
  ``rmtree``) and remove files (``remove``)

* Get information about directories, files and links (``listdir``,
  ``stat``, ``lstat``, ``exists``, ``isdir``, ``isfile``, ``islink``,
  ``abspath``, ``split``, ``join``, ``dirname``, ``basename`` etc.)

* Iterate over remote file systems (``walk``)

* Local caching of results from ``lstat`` and ``stat`` calls to reduce
  network access (also applies to ``exists``, ``getmtime`` etc.).

* Read files from and write files to remote hosts via
  file-like objects (``FTPHost.open``; the generated file-like objects
  have the familiar methods like ``read``, ``readline``, ``readlines``,
  ``write``, ``writelines`` and ``close``. You can also iterate over
  these files line by line in a ``for`` loop.


Exception hierarchy
-------------------

The exceptions are in the namespace of the ``ftputil.error`` module, e. g.
``ftputil.error.TemporaryError``.

The exception classes are organized as follows::

    FTPError
        FTPOSError(FTPError, OSError)
            PermanentError(FTPOSError)
                CommandNotImplementedError(PermanentError)
            TemporaryError(FTPOSError)
        FTPIOError(FTPError)
        InternalError(FTPError)
            InaccessibleLoginDirError(InternalError)
            ParserError(InternalError)
            RootDirError(InternalError)
            TimeShiftError(InternalError)

and are described here:

- ``FTPError``

  is the root of the exception hierarchy of the module.

- ``FTPOSError``

  is derived from ``OSError``. This is for similarity between the
  os module and ``FTPHost`` objects. Compare

  ::

    try:
        os.chdir("nonexisting_directory")
    except OSError:
        ...

  with

  ::

    host = ftputil.FTPHost("host", "user", "password")
    try:
        host.chdir("nonexisting_directory")
    except OSError:
        ...

  Imagine a function

  ::

    def func(path, file):
        ...

  which works on the local file system and catches ``OSErrors``. If you
  change the parameter list to

  ::

    def func(path, file, os=os):
        ...

  where ``os`` denotes the ``os`` module, you can call the function also as

  ::

    host = ftputil.FTPHost("host", "user", "password")
    func(path, file, os=host)

  to use the same code for both a local and remote file system.
  Another similarity between ``OSError`` and ``FTPOSError`` is that
  the latter holds the FTP server return code in the ``errno``
  attribute of the exception object and the error text in
  ``strerror``.

- ``PermanentError``

  is raised for 5xx return codes from the FTP server. This
  corresponds to ``ftplib.error_perm`` (though ``PermanentError`` and
  ``ftplib.error_perm`` are *not* identical).

- ``CommandNotImplementedError``

  indicates that an underlying command the code tries to use is not
  implemented. For an example, see the description of the
  `FTPHost.chmod`_ method.

- ``TemporaryError``

  is raised for FTP return codes from the 4xx category. This
  corresponds to ``ftplib.error_temp`` (though ``TemporaryError`` and
  ``ftplib.error_temp`` are *not* identical).

- ``FTPIOError``

  denotes an I/O error on the remote host. This appears
  mainly with file-like objects that are retrieved by calling
  ``FTPHost.open``. Compare

  ::

    >>> try:
    ...     f = open("not_there")
    ... except IOError as obj:
    ...     print obj.errno
    ...     print obj.strerror
    ...
    2
    No such file or directory

  with

  ::

    >>> ftp_host = ftputil.FTPHost("host", "user", "password")
    >>> try:
    ...     f = ftp_host.open("not_there")
    ... except IOError as obj:
    ...     print obj.errno
    ...     print obj.strerror
    ...
    550
    550 not_there: No such file or directory.

  As you can see, both code snippets are similar. However, the error
  codes aren't the same.

- ``InternalError``

  subsumes exception classes for signaling errors due to limitations
  of the FTP protocol or the concrete implementation of ``ftputil``.

- ``InaccessibleLoginDirError``

  This exception is raised if the directory in which "you" are placed
  upon login is not accessible, i. e. a ``chdir`` call with the
  directory as argument would fail.

- ``ParserError``

  is used for errors during the parsing of directory
  listings from the server. This exception is used by the ``FTPHost``
  methods ``stat``, ``lstat``, and ``listdir``.

- ``RootDirError``

  Because of the implementation of the ``lstat`` method it is not
  possible to do a ``stat`` call on the root directory ``/``.
  If you know *any* way to do it, please let me know. :-)

  This problem does *not* affect stat calls on items *in* the root
  directory.

- ``TimeShiftError``

  is used to denote errors which relate to setting the `time shift`_.


Directory and file names
------------------------

.. note::

   Keep in mind that this section only applies to directory and file
   *names*, not file *contents*. Encoding and decoding for file
   contents is handled by the ``encoding`` argument for
   `FTPHost.open`_.

First off: If your directory and file names (both as
arguments and on the server) contain only ISO 8859-1 (latin-1)
characters, you can use such names in the form of byte strings or
unicode strings. However, you can't mix different string types (bytes
and unicode) in one call (for example in ``FTPHost.path.join``).

If you have directory or file names with characters that aren't in
latin-1, it's recommended to use byte strings. In that case,
returned paths will be byte strings, too.

Read on for details.

.. note::

   The approach described below may look awkward and in a way it is.
   The intention of ``ftputil`` is to behave like the local file
   system APIs of Python 3 as far as it makes sense. Moreover, the
   taken approach makes sure that directory and file names that were
   used with Python 3's native ``ftplib`` module will be compatible
   with ``ftputil`` and vice versa. Otherwise you may be able to use a
   file name with ``ftputil``, but get an exception when trying to
   read the same file with Python 3's ``ftplib`` module.

Methods that take names of directories and/or files can take either
byte or unicode strings. If a method got a string argument and returns
one or more strings, these strings will have the same string type as
the argument(s). Mixing different string arguments in one call (for
example in ``FTPHost.path.join``) isn't allowed and will cause a
``TypeError``. These rules are the same as for local file system
operations in Python 3. Since ``ftputil`` uses the same API for Python
2, ``ftputil`` will do the same when run on Python 2.

Byte strings for directory and file names will be sent to the server
as-is. On the other hand, unicode strings will be encoded to byte
strings, assuming latin-1 encoding. This implies that such unicode
strings must only contain code points 0-255 for the latin-1 character
set. Using any other characters will result in a
``UnicodeEncodeError`` exception.

If you have directory or file names as unicode strings with non-latin-1
characters, encode the unicode strings to byte strings yourself, using
the encoding you know the server uses. Decode received paths with the
same encoding. Encapsulate these conversions as far as you can.
Otherwise, you'd have to adapt potentially a lot of code if the server
encoding changes.

If you *don't* know the encoding on the server side,
it's probably the best to only use byte strings for directory and file
names. That said, as soon as you *show* the names to a user, you -- or
the library you use for displaying the names -- has to guess an
encoding.


``FTPHost`` objects
-------------------

.. _`FTPHost construction`:

Construction
~~~~~~~~~~~~

Introduction
````````````

``FTPHost`` instances can be created with the following call::

    ftp_host = ftputil.FTPHost(server, user, password, account,
                               session_factory=ftplib.FTP)

The first four parameters are strings with the same meaning as for the
FTP class in the ``ftplib`` module. Usually the ``account`` and
``session_factory`` arguments aren't needed though.

``FTPHost`` objects can also be used in a ``with`` statement::

    import ftputil

    with ftputil.FTPHost(server, user, password) as ftp_host:
        print ftp_host.listdir(ftp_host.curdir)

After the ``with`` block, the ``FTPHost`` instance and the
associated FTP sessions will be closed automatically.

If something goes wrong during the ``FTPHost`` construction or in the
body of the ``with`` statement, the instance is closed as well.
Exceptions will be propagated (as with ``try ... finally``).

Session factories
`````````````````

The keyword argument ``session_factory`` may be used to generate FTP
connections with other factories than the default ``ftplib.FTP``. For
example, the standard library of Python 2.7 contains a class
``ftplib.FTP_TLS`` which extends ``ftplib.FTP`` to use an encrypted
connection.

In fact, all positional and keyword arguments other than
``session_factory`` are passed to the factory to generate a new
background session. This also happens for every remote file that is
opened; see below.

This functionality of the constructor also allows to wrap
``ftplib.FTP`` objects to do something that wouldn't be possible with
the ``ftplib.FTP`` constructor alone.

As an example, assume you want to connect to another than the default
port, but ``ftplib.FTP`` only offers this by means of its ``connect``
method, not via its constructor. One solution is to use a custom
class as a session factory::

    import ftplib
    import ftputil

    EXAMPLE_PORT = 50001

    class MySession(ftplib.FTP):

        def __init__(self, host, userid, password, port):
            """Act like ftplib.FTP's constructor but connect to another port."""
            ftplib.FTP.__init__(self)
            self.connect(host, port)
            self.login(userid, password)

    # Try _not_ to use an _instance_ `MySession()` as factory, -
    # use the class itself.
    with ftputil.FTPHost(host, userid, password, port=EXAMPLE_PORT,
                         session_factory=MySession) as ftp_host:
        # Use `ftp_host` as usual.
        ...

On login, the format of the directory listings (needed for stat'ing
files and directories) should be determined automatically. If not,
please `file a bug report`_.

.. _`file a bug report`: http://ftputil.sschwarzer.net/issuetrackernotes

For the most common uses you don't need to create your own session
factory class though. The ``ftputil.session`` module has a function
``session_factory`` that can create session factories for a variety
of parameters::

    session_factory(base_class=ftplib.FTP,
                    port=21,
                    use_passive_mode=None,
                    encrypt_data_channel=True,
                    debug_level=None)

with

- ``base_class`` is a base class to inherit a new session factory
  class from. By default, this is ``ftplib.FTP`` from the Python
  standard library.

- ``port`` is the command channel port. The default is 21, used in most
  FTP server configurations.

- ``use_passive_mode`` is either a boolean that determines whether
  passive mode should be used or ``None``. ``None`` means to let the
  base class choose active or passive mode.

- ``encrypt_data_channel`` defines whether to encrypt the data channel
  for secure connections. This is only supported for the base classes
  ``ftplib.FTP_TLS`` and ``M2Crypto.ftpslib.FTP_TLS``, otherwise the
  the parameter is ignored.

- ``debug_level`` sets the debug level for FTP session instances. The
  semantics is defined by the base class. For example, a debug level
  of 2 causes the most verbose output for Python's ``ftplib.FTP``
  class.

All of these parameters can be combined. For example, you could use

::

    import ftplib

    import ftputil
    import ftputil.session


    my_session_factory = ftputil.session.session_factory(
                           base_class=ftpslib.FTP_TLS,
                           port=31,
                           encrypt_data_channel=True,
                           debug_level=2)

    with ftputil.FTPHost(server, user, password,
                         session_factory=my_session_factory) as ftp_host:
        ...

to create and use a session factory derived from ``ftplib.FTP_TLS``
that connects on command channel 31, will encrypt the data channel and
print output for debug level 2.

Note: Generally, you can achieve everything you can do with
``ftputil.session.session_factory`` with an explicit session factory
as described at the start of this section. However, the class
``M2Crypto.ftpslib.FTP_TLS`` has a limitation so that you can't use
it with ftputil out of the box. The function ``session_factory``
contains a workaround for this limitation. For details refer to `this
bug report`_.

.. _`this bug report`: http://ftputil.sschwarzer.net/trac/ticket/78

Hidden files and directories
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Whether ftputil sees "hidden" files and directories (usually files or
directories whose names start with a dot) depends on the FTP server
configuration. By default, ftputil uses the ``-a`` option in the FTP
``LIST`` command to find hidden files. However, the server may ignore
this.

If using the ``-a`` option leads to problems, for example if an
FTP server causes an exception, you may switch off the use of the
option::

    ftp_host = ftputil.FTPHost(server, user, password, account,
                               session_factory=ftplib.FTP)
    ftp_host.use_list_a_option = False

``FTPHost`` attributes and methods
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Attributes
``````````

- ``curdir``, ``pardir``, ``sep``

  are strings which denote the current and the parent directory on the
  remote server. ``sep`` holds the path separator. Though `RFC 959`_
  (File Transfer Protocol) notes that these values may depend on the
  FTP server implementation, the Unix variants seem to work well in
  practice, even for non-Unix servers.

  Nevertheless, it's recommended that you don't hardcode these values
  for remote paths, but use `FTPHost.path`_ as you would use
  ``os.path`` to write platform-independent Python code for local
  filesystems. Keep in mind that most, *but not all*, arguments of
  ``FTPHost`` methods refer to remote directories or files. For
  example, in `FTPHost.upload`_, the first argument is a local
  path and the second a remote path. Both of these should use their
  respective path separators.

.. _`FTPHost.upload`: `Uploading and downloading files`_

Remote file system navigation
`````````````````````````````

- ``getcwd()``

  returns the absolute current directory on the remote host. This
  method works like ``os.getcwd``.

- ``chdir(directory)``

  sets the current directory on the FTP server. This resembles
  ``os.chdir``, as you may have expected.

.. _`callback function`:

Uploading and downloading files
```````````````````````````````

- ``upload(source, target, callback=None)``

  copies a local source file (given by a filename, i. e. a string)
  to the remote host under the name target. Both ``source`` and
  ``target`` may be absolute paths or relative to their corresponding
  current directory (on the local or the remote host, respectively).

  The file content is always transferred in binary mode.

  The callback, if given, will be invoked for each transferred chunk
  of data::

    callback(chunk)

  where ``chunk`` is a bytestring. An example usage of a callback
  method is to display a progress indicator.

- ``download(source, target, callback=None)``

  performs a download from the remote source file to a local target
  file. Both ``source`` and ``target`` are strings. See the
  description of ``upload`` for more details.

.. _`upload_if_newer`:

- ``upload_if_newer(source, target, callback=None)``

  is similar to the ``upload`` method. The only difference is that the
  upload is only invoked if the time of the last modification for the
  source file is more recent than that of the target file or the
  target doesn't exist at all. The check for the last modification
  time considers the precision of the timestamps and transfers a file
  "if in doubt". Consequently the code

  ::

    ftp_host.upload_if_newer("source_file", "target_file")
    time.sleep(10)
    ftp_host.upload_if_newer("source_file", "target_file")

  might upload the file again if the timestamp of the target file is
  precise up to a minute, which is typically the case because the
  remote datetime is determined by parsing a directory listing from
  the server. To avoid unnecessary transfers, wait at least a minute
  between calls of ``upload_if_newer`` for the same file. If it still
  seems that a file is uploaded unnecessarily (or not when it should),
  read the subsection on `time shift`_ settings.

  If an upload actually happened, the return value of
  ``upload_if_newer`` is a ``True``, else ``False``.

  Note that the method only checks the existence and/or the
  modification time of the source and target file; it doesn't
  compare any other file properties, say, the file size.

  This also means that if a transfer is interrupted, the remote file
  will have a newer modification time than the local file, and thus
  the transfer won't be repeated if ``upload_if_newer`` is used a
  second time. There are at least two possibilities after a failed
  upload:

  - use ``upload`` instead of ``upload_if_newer``, or

  - remove the incomplete target file with ``FTPHost.remove``, then
    use ``upload`` or ``upload_if_newer`` to transfer it again.

.. _`download_if_newer`:

- ``download_if_newer(source, target, callback=None)``

  corresponds to ``upload_if_newer`` but performs a download from the
  server to the local host. Read the descriptions of download and
  ``upload_if_newer`` for more information. If a download actually
  happened, the return value is ``True``, else ``False``.

.. _`time shift`:
.. _`time zone correction`:

Time zone correction
````````````````````

If the client where ``ftputil`` runs and the server have a different
understanding of their local times, this has to be taken into account
for ``upload_if_newer`` and ``download_if_newer`` to work correctly.

Note that even if the client and the server are in the same time zone
(or even on the same computer), the time shift value (see below) may
be different from zero. For example, my computer is set to use local
time whereas the server running on the very same host insists on using
UTC time.

.. _`set_time_shift`:

- ``set_time_shift(time_shift)``

  sets the so-called time shift value, measured in seconds. The time
  shift is the difference between the local time of the server and the
  local time of the client at a given moment, i. e. by definition

  ::

    time_shift = server_time - client_time

  Setting this value is important for `upload_if_newer`_ and
  `download_if_newer`_ to work correctly even if the time zone of the
  FTP server differs from that of the client. Note that the time shift
  value *can be negative*.

  If the time shift value is invalid, for example its absolute value
  is larger than 24 hours, a ``TimeShiftError`` is raised.

  See also `synchronize_times`_ for a way to set the time shift with a
  simple method call.

- ``time_shift()``

  returns the currently-set time shift value. See ``set_time_shift``
  above for its definition.

.. _`synchronize_times`:

- ``synchronize_times()``

  synchronizes the local times of the server and the client, so that
  `upload_if_newer`_ and `download_if_newer`_ work as expected, even
  if the client and the server use different time zones. For this
  to work, *all* of the following conditions must be true:

  - The connection between server and client is established.

  - The client has write access to the directory that is current when
    ``synchronize_times`` is called.

  If you can't fulfill these conditions, you can nevertheless set the
  time shift value explicitly with `set_time_shift`_. Trying to call
  ``synchronize_times`` if the above conditions aren't met results in
  a ``TimeShiftError`` exception.

Creating and removing directories
`````````````````````````````````

- ``mkdir(path, [mode])``

  makes the given directory on the remote host. This does *not*
  construct "intermediate" directories that don't already exist. The
  ``mode`` parameter is ignored; this is for compatibility with
  ``os.mkdir`` if an ``FTPHost`` object is passed into a function
  instead of the ``os`` module. See the explanation in the subsection
  `Exception hierarchy`_.

- ``makedirs(path, [mode])``

  works similar to ``mkdir`` (see above), but also makes intermediate
  directories like ``os.makedirs``. The ``mode`` parameter is only
  there for compatibility with ``os.makedirs`` and is ignored.

- ``rmdir(path)``

  removes the given remote directory. If it's not empty, raise
  a ``PermanentError``.

- ``rmtree(path, ignore_errors=False, onerror=None)``

  removes the given remote, possibly non-empty, directory tree.
  The interface of this method is rather complex, in favor of
  compatibility with ``shutil.rmtree``.

  If ``ignore_errors`` is set to a true value, errors are ignored.
  If ``ignore_errors`` is a false value *and* ``onerror`` isn't
  set, all exceptions occurring during the tree iteration and
  processing are raised. These exceptions are all of type
  ``PermanentError``.

  To distinguish between different kinds of errors, pass in a callable
  for ``onerror``. This callable must accept three arguments:
  ``func``, ``path`` and ``exc_info``. ``func`` is a bound method
  object, *for example* ``your_host_object.listdir``. ``path`` is the
  path that was the recent argument of the respective method
  (``listdir``, ``remove``, ``rmdir``). ``exc_info`` is the exception
  info as it is gotten from ``sys.exc_info``.

  The code of ``rmtree`` is taken from Python's ``shutil`` module
  and adapted for ``ftputil``.

Removing files and links
````````````````````````

- ``remove(path)``

  removes a file or link on the remote host, similar to ``os.remove``.

- ``unlink(path)``

  is an alias for ``remove``.

Retrieving information about directories, files and links
`````````````````````````````````````````````````````````

- ``listdir(path)``

  returns a list containing the names of the files and directories
  in the given path, similar to `os.listdir`_. The special names
  ``.`` and ``..`` are not in the list.

The methods ``lstat`` and ``stat`` (and some others) rely on the
directory listing format used by the FTP server. When connecting to a
host, ``FTPHost``'s constructor tries to guess the right format, which
succeeds in most cases. However, if you get strange results or
``ParserError`` exceptions by a mere ``lstat`` call, please `file a
bug report`_.

If ``lstat`` or ``stat`` give wrong modification dates or times, look
at the methods that deal with time zone differences (`time zone
correction`_).

.. _`FTPHost.lstat`:

- ``lstat(path)``

  returns an object similar to that from `os.lstat`_. This is a
  "tuple" with additional attributes; see the documentation of the
  ``os`` module for details.

  The result is derived by parsing the output of a ``LIST`` command on
  the server. Therefore, the result from ``FTPHost.lstat`` can not
  contain more information than the received text. In particular:

  - User and group ids can only be determined as strings, not as
    numbers, and that only if the server supplies them. This is
    usually the case with Unix servers but maybe not for other FTP
    server programs.

  - Values for the time of the last modification may be rough,
    depending on the information from the server. For timestamps
    older than a year, this usually means that the precision of the
    modification timestamp value is not better than days. For newer
    files, the information may be accurate to a minute.

    If the time of the last modification is before the epoch (usually
    1970-01-01 UTC), set the time of the last modification to 0.0.

  - Links can only be recognized on servers that provide this
    information in the ``LIST`` output.

  - Stat attributes that can't be determined at all are set to
  	``None``. For example, a line of a directory listing may not
  	contain the date/time of a directory's last modification.

  - There's a special problem with stat'ing the root directory.
    (Stat'ing things *in* the root directory is fine though.) In
    this case, a ``RootDirError`` is raised. This has to do with the
    algorithm used by ``(l)stat``, and I know of no approach which
    mends this problem.

  Currently, ``ftputil`` recognizes the common Unix-style and
  Microsoft/DOS-style directory formats. If you need to parse output
  from another server type, please write to the `ftputil mailing
  list`_. You may consider `writing your own parser`_.

.. _`os.listdir`: https://docs.python.org/library/os.html#os.listdir
.. _`os.lstat`: https://docs.python.org/library/os.html#os.lstat
.. _`ftputil mailing list`: http://ftputil.sschwarzer.net/mailinglist
.. _`writing your own parser`: `Writing directory parsers`_

.. _`FTPHost.stat`:

- ``stat(path)``

  returns ``stat`` information also for files which are pointed to by a
  link. This method follows multiple links until a regular file or
  directory is found. If an infinite link chain is encountered or the
  target of the last link in the chain doesn't exist, a
  ``PermanentError`` is raised.

  The limitations of the ``lstat`` method also apply to ``stat``.

.. _`FTPHost.path`:

``FTPHost`` objects contain an attribute named ``path``, similar to
`os.path`_. The following methods can be applied to the remote host
with the same semantics as for ``os.path``:

::

    abspath(path)
    basename(path)
    commonprefix(path_list)
    dirname(path)
    exists(path)
    getmtime(path)
    getsize(path)
    isabs(path)
    isdir(path)
    isfile(path)
    islink(path)
    join(path1, path2, ...)
    normcase(path)
    normpath(path)
    split(path)
    splitdrive(path)
    splitext(path)
    walk(path, func, arg)

Like Python's counterparts under `os.path`_, ``ftputil``'s ``is...``
methods return ``False`` if they can't find the path given by their
argument.

Local caching of file system information
````````````````````````````````````````

Many of the above methods need access to the remote file system to
obtain data on directories and files. To get the most recent data,
*each* call to ``lstat``, ``stat``, ``exists``, ``getmtime`` etc.
would require to fetch a directory listing from the server, which can
make the program *very* slow. This effect is more pronounced for
operations which mostly scan the file system rather than transferring
file data.

For this reason, ``ftputil`` by default saves the results from
directory listings locally and reuses those results. This reduces
network accesses and so speeds up the software a lot. However, since
data is more rarely fetched from the server, the risk of obsolete data
also increases. This will be discussed below.

Caching can be controlled -- if necessary at all -- via the
``stat_cache`` object in an ``FTPHost``'s namespace. For example,
after calling

::

    ftp_host = ftputil.FTPHost(host, user, password)

the cache can be accessed as ``ftp_host.stat_cache``.

While ``ftputil`` usually manages the cache quite well, there are two
possible reasons for modifying cache parameters.

The first is when the number of possible entries is too low. You may
notice that when you are processing very large directories and the
program becomes much slower than before. It's common for code to read
a directory with ``listdir`` and then process the found directories
and files. This can also happen implicitly by a call to
``FTPHost.walk``. Since version 2.6 ``ftputil`` automatically
increases the cache size if directories with more entries than the
current maximum cache size are to be scanned. Most of the time, this
works fine.

However, if you need access to stat data for several directories at
the same time, you may need to increase the cache explicitly. This is
done by the ``resize`` method::

    ftp_host.stat_cache.resize(20000)

where the argument is the maximum number of ``lstat`` results to store
(the default is 5000, in versions before 2.6 it was 1000). Note that
each path on the server, e. g. "/home/schwa/some_dir", corresponds to
a single cache entry. Methods like ``exists`` or ``getmtime`` all
derive their results from a previously fetched ``lstat`` result.

The value 5000 above means that the cache will hold *at most* 5000
entries (unless increased automatically by an explicit or implicit
``listdir`` call, see above). If more are about to be stored, the
entries which haven't been used for the longest time will be deleted
to make place for newer entries.

The second possible reason to change the cache parameters is to avoid
stale cache data. Caching is so effective because it reduces network
accesses. This can also be a disadvantage if the file system data on
the remote server changes after a stat result has been retrieved; the
client, when looking at the cached stat data, will use obsolete
information.

There are two potential ways to get such out-of-date stat data. The
first happens when an ``FTPHost`` instance modifies a file path for
which it has a cache entry, e. g. by calling ``remove`` or ``rmdir``.
Such changes are handled transparently; the path will be deleted from
the cache. A different matter are changes unknown to the ``FTPHost``
object which inspects its cache. Obviously, for example, these are
changes by programs running on the remote host. On the other hand,
cache inconsistencies can also occur if two ``FTPHost`` objects change
a file system simultaneously::

    with (
      ftputil.FTPHost(server, user1, password1) as ftp_host1,
      ftputil.FTPHost(server, user1, password1) as ftp_host2
    ):
        stat_result1 = ftp_host1.stat("some_file")
        stat_result2 = ftp_host2.stat("some_file")
        ftp_host2.remove("some_file")
        # `ftp_host1` will still see the obsolete cache entry!
        print ftp_host1.stat("some_file")
        # Will raise an exception since an `FTPHost` object
        # knows of its own changes.
        print ftp_host2.stat("some_file")

At first sight, it may appear to be a good idea to have a shared cache
among several ``FTPHost`` objects. After some thinking, this turns out
to be very error-prone. For example, it won't help with different
processes using ``ftputil``. So, if you have to deal with concurrent
write/read accesses to a server, you have to handle them explicitly.

The most useful tool for this is the ``invalidate`` method. In the
example above, it could be used like this::

    with (
      ftputil.FTPHost(server, user1, password1) as ftp_host1,
      ftputil.FTPHost(server, user1, password1) as ftp_host2
    ):
        stat_result1 = ftp_host1.stat("some_file")
        stat_result2 = ftp_host2.stat("some_file")
        ftp_host2.remove("some_file")
        # Invalidate using an absolute path.
        absolute_path = ftp_host1.path.abspath(
                          ftp_host1.path.join(ftp_host1.getcwd(), "some_file"))
        ftp_host1.stat_cache.invalidate(absolute_path)
        # Will now raise an exception as it should.
        print ftp_host1.stat("some_file")
        # Would raise an exception since an `FTPHost` object
        # knows of its own changes, even without `invalidate`.
        print ftp_host2.stat("some_file")

The method ``invalidate`` can be used on any *absolute* path, be it a
directory, a file or a link.

By default, the cache entries (if not replaced by newer ones) are
stored for an infinite time. That is, if you start your Python process
using ``ftputil`` and let it run for three days a stat call may still
access cache data that old. To avoid this, you can set the ``max_age``
attribute::

    with ftputil.FTPHost(server, user, password) as ftp_host:
        ftp_host.stat_cache.max_age = 60 * 60  # = 3600 seconds

This sets the maximum age of entries in the cache to an hour. This
means any entry older won't be retrieved from the cache but its data
instead fetched again from the remote host and then again stored for
up to an hour. To reset `max_age` to the default of unlimited age,
i. e. cache entries never expire, use ``None`` as value.

If you are certain that the cache will be in the way, you can disable
and later re-enable it completely with ``disable`` and ``enable``::

    with ftputil.FTPHost(server, user, password) as ftp_host:
        ftp_host.stat_cache.disable()
        ...
        ftp_host.stat_cache.enable()

During that time, the cache won't be used; all data will be fetched
from the network. After enabling the cache again, its entries will be
the same as when the cache was disabled, that is, entries won't get
updated with newer data during this period. Note that even when the
cache is disabled, the file system data in the code can become
inconsistent::

    with ftputil.FTPHost(server, user, password) as ftp_host:
        ftp_host.stat_cache.disable()
        if ftp_host.path.exists("some_file"):
            mtime = ftp_host.path.getmtime("some_file")

In that case, the file ``some_file`` may have been removed by another
process between the calls to ``exists`` and ``getmtime``!

Iteration over directories
``````````````````````````

.. _`FTPHost.walk`:

- ``walk(top, topdown=True, onerror=None, followlinks=False)``

  iterates over a directory tree, similar to `os.walk`_. Actually,
  ``FTPHost.walk`` uses the code from Python with just the necessary
  modifications, so see the linked documentation.

.. _`os.walk`: https://docs.python.org/2/library/os.html#os.walk

.. _`FTPHost.path.walk`:

- ``path.walk(path, func, arg)``

  Similar to ``os.path.walk``, the ``walk`` method in
  `FTPHost.path`_ can be used, though ``FTPHost.walk`` is probably
  easier to use.

Other methods
`````````````

- ``close()``

  closes the connection to the remote host. After this, no more
  interaction with the FTP server is possible with this ``FTPHost``
  object. Usually you don't need to close an ``FTPHost`` instance
  with ``close`` if you set up the instance in a ``with`` statement.

- ``rename(source, target)``

  renames the source file (or directory) on the FTP server.

.. _`FTPHost.chmod`:

- ``chmod(path, mode)``

  sets the access mode (permission flags) for the given path. The mode
  is an integer as returned for the mode by the ``stat`` and ``lstat``
  methods. Be careful: Usually, mode values are written as octal
  numbers, for example 0755 to make a directory readable and writable
  for the owner, but not writable for the group and others. If you
  want to use such octal values, rely on Python's support for them::

    ftp_host.chmod("some_directory", 0o755)

  Not all FTP servers support the ``chmod`` command. In case of
  an exception, how do you know if the path doesn't exist or if
  the command itself is invalid? If the FTP server complies with
  `RFC 959`_, it should return a status code 502 if the ``SITE CHMOD``
  command isn't allowed. ``ftputil`` maps this special error
  response to a ``CommandNotImplementedError`` which is derived from
  ``PermanentError``.

  So you need to code like this::

    with ftputil.FTPHost(server, user, password) as ftp_host:
        try:
            ftp_host.chmod("some_file", 0o644)
        except ftputil.error.CommandNotImplementedError:
            # `chmod` not supported
            ...
        except ftputil.error.PermanentError:
            # Possibly a non-existent file
            ...

  Because the ``CommandNotImplementedError`` is more specific, you
  have to test for it first.

.. _`RFC 959`: `RFC 959 - File Transfer Protocol (FTP)`_

- ``copyfileobj(source, target, length=64*1024)``

  copies the contents from the file-like object ``source`` to the
  file-like object ``target``. The only difference to
  ``shutil.copyfileobj`` is the default buffer size. Note that
  arbitrary file-like objects can be used as arguments (e. g. local
  files, remote FTP files).

  However, the interfaces of ``source`` and ``target`` have to match;
  the string type read from ``source`` must be an accepted string type
  when written to ``target``. For example, if you open ``source`` in
  Python 3 as a local text file and ``target`` as a remote file object
  in binary mode, the transfer will fail since ``source.read`` gives
  unicode strings whereas ``target.write`` only accepts byte strings.

  See `File-like objects`_ for the construction and use of remote
  file-like objects.

.. _`set_parser`:

- ``set_parser(parser)``

  sets a custom parser for FTP directories. Note that you have to pass
  in a parser *instance*, not the class.

  An `extra section`_ shows how to write own parsers if the default
  parsers in ``ftputil`` don't work for you.

.. _`extra section`: `Writing directory parsers`_

.. _`keep_alive`:

- ``keep_alive()``

  attempts to keep the connection to the remote server active in order
  to prevent timeouts from happening. This method is primarily
  intended to keep the underlying FTP connection of an ``FTPHost``
  object alive while a file is uploaded or downloaded. This will
  require either an extra thread while the upload or download is in
  progress or calling ``keep_alive`` from a `callback function`_.

  The ``keep_alive`` method won't help if the connection has already
  timed out. In this case, a ``ftputil.error.TemporaryError`` is raised.

  If you want to use this method, keep in mind that FTP servers define
  a timeout for a reason. A timeout prevents running out of server
  connections because of clients that never disconnect on their own.

  Note that the ``keep_alive`` method does *not* affect the "hidden"
  FTP child connections established by ``FTPHost.open`` (see section
  `FTPHost instances vs. FTP connections`_ for details). You *can't*
  use ``keep_alive`` to avoid a timeout in a stalling transfer like
  this::

      with ftputil.FTPHost(server, userid, password) as ftp_host:
          with ftp_host.open("some_remote_file", "rb") as fobj:
              data = fobj.read(100)
              # _Futile_ attempt to avoid file connection timeout.
              for i in xrange(15):
                  time.sleep(60)
                  ftp_host.keep_alive()
              # Will raise an `ftputil.error.TemporaryError`.
              data += fobj.read()


.. _`FTPHost.open`:

File-like objects
-----------------

Construction
~~~~~~~~~~~~

Basics
``````

``FTPFile`` objects are returned by a call to ``FTPHost.open``;
never use the ``FTPFile`` constructor directly.

The API of remote file-like objects are is modeled after the API of
the io_ module in Python 3, which has also been backported to Python
2.6 and 2.7.

.. _io: http://docs.python.org/library/io.html

- ``FTPHost.open(path, mode="r", buffering=None, encoding=None,
  errors=None, newline=None, rest=None)``

  returns a file-like object that refers to the path on the remote
  host. This path may be absolute or relative to the current directory
  on the remote host (this directory can be determined with the
  ``getcwd`` method). As with local file objects, the default mode is
  "r", i. e. reading text files. Valid modes are "r", "rb", "w", and
  "wb".

  If a file is opened in binary mode, you *must not* specify an
  encoding. On the other hand, if you open a file in text mode, an
  encoding is used. By default, this is the return value of
  ``locale.getpreferredencoding``, but you can (and probably should)
  specify a distinct encoding.

  If you open a file in binary mode, the read and write operations use
  byte strings (``str`` in Python 2, ``bytes`` in Python 3). That is,
  read operations return byte strings and write operations only accept
  byte strings.

  Similarly, text files always work with unicode strings (``unicode``
  in Python 2, ``str`` in Python 3). Here, read operations return
  unicode strings and write operations only accept unicode strings.

  .. warning::

     Note that the semantics of "text mode" has changed fundamentally
     from ftputil 2.8 and earlier. Previously, "text mode" implied
     converting newline characters to ``\r\n`` when writing remote
     files and converting newlines to ``\n`` when reading remote
     files. This is in line with the "text mode" notion of FTP command
     line clients. Now, "text mode" follows the semantics in Python's
     ``io`` module.

  The arguments ``errors`` and ``newline`` have the same semantics as
  in `io.open`_. The argument ``buffering`` currently is ignored.
  It's only there for compatibility with the ``io.open`` interface.

  If the file is opened in binary mode, you may pass 0 or a positive
  integer for the ``rest`` argument. The argument is passed to the
  underlying FTP session instance (for example an instance of
  ``ftplib.FTP``) to start reading or writing at the given byte
  offset. For example, if a remote file contains the letters
  "abcdef" in ASCII encoding, ``rest=3`` will start reading at "d".

  .. warning::

     If you pass ``rest`` values which point *after* the file, the
     behavior is undefined and may even differ from one FTP server to
     another. Therefore, use the ``rest`` argument only for error
     recovery in case of interrupted transfers. You need to keep track
     of the transferred data so that you can provide a valid ``rest``
     argument for a resumed transfer.

.. _`io.open`: http://docs.python.org/library/io.html#io.open

``FTPHost.open`` can also be used in a ``with`` statement::

    import ftputil

    with ftputil.FTPHost(...) as ftp_host:
        ...
        with ftp_host.open("new_file", "w", encoding="utf8") as fobj:
            fobj.write("This is some text.")

At the end of the ``with`` block, the remote file will be closed
automatically.

If something goes wrong during the construction of the file or in the
body of the ``with`` statement, the file will be closed as well.
Exceptions will be propagated as with ``try ... finally``.

Attributes and methods
~~~~~~~~~~~~~~~~~~~~~~

The methods

::

    close()
    read([count])
    readline([count])
    readlines()
    write(data)
    writelines(string_sequence)

and the attribute ``closed`` have the same semantics as for file
objects of a local disk file system. The iterator protocol is
supported as well, i. e. you can use a loop to read a file line by
line::

    with ftputil.FTPHost(server, user, password) as ftp_host:
        with ftp_host.open("some_file") as input_file:
            for line in input_file:
                # Do something with the line, e. g.
                print line.strip().replace("ftplib", "ftputil")

For more on file objects, see the section `File objects`_ in the
Python Library Reference.

.. _`file objects`: https://docs.python.org/2.7/library/stdtypes.html#file-objects


.. _`child_connections`:

``FTPHost`` instances vs. FTP connections
-----------------------------------------

This section explains why keeping an ``FTPHost`` instance "alive"
without timing out sometimes isn't trivial. If you always finish your
FTP operations in time, you don't need to read this section.

The file transfer protocol is a stateful protocol. That means an FTP
connection always is in a certain state. Each of these states can only
change to certain other states under certain conditions triggered by
the client or the server.

One of the consequences is that a single FTP connection can't be used
at the same time, say, to transfer data on the FTP data channel and to
create a directory on the remote host.

For example, consider this::

    >>> import ftplib
    >>> ftp = ftplib.FTP(server, user, password)
    >>> ftp.pwd()
    '/'
    >>> # Start transfer. `CONTENTS` is a text file on the server.
    >>> socket = ftp.transfercmd("RETR CONTENTS")
    >>> socket
    <socket._socketobject object at 0x7f801a6386e0>
    >>> ftp.pwd()
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "/usr/lib64/python2.7/ftplib.py", line 578, in pwd
        return parse257(resp)
      File "/usr/lib64/python2.7/ftplib.py", line 842, in parse257
        raise error_reply, resp
    ftplib.error_reply: 226-File successfully transferred
    226 0.000 seconds (measured here), 5.60 Mbytes per second
    >>>

Note that ``ftp`` is a single FTP connection, represented by an
``ftplib.FTP`` instance, not an ``ftputil.FTPHost`` instance.

On the other hand, consider this::

    >>> import ftputil
    >>> ftp_host = ftputil.FTPHost(server, user, password)
    >>> ftp_host.getcwd()
    >>> fobj = ftp_host.open("CONTENTS")
    >>> fobj
    <ftputil.file.FTPFile object at 0x7f8019d3aa50>
    >>> ftp_host.getcwd()
    u'/'
    >>> fobj.readline()
    u'Contents of FTP test directory\n'
    >>> fobj.close()
    >>>

To be able to start a file transfer (i. e. open a remote file for
reading or writing) and still be able to use other FTP commands,
ftputil uses a trick. For every remote file, ftputil creates a new FTP
connection, called a child connection in the ftputil source code.
(Actually, FTP connections belonging to closed remote files are
re-used if they haven't timed out yet.)

In most cases this approach isn't noticeable by code using ftputil.
However, the nice abstraction of dealing with a single FTP connection
falls apart if one of the child connections times out. For example, if
you open a remote file and work only with the initial "main"
connection to navigate the file system, the FTP connection for the
remote file may eventually time out.

While it's often relatively easy to prevent the "main" connection from
timing out it's unfortunately practically impossible to do this for a
remote file connection (apart from transferring some data, of course).
For this reason, `FTPHost.keep_alive`_ affects only the main
connection. Child connections may still time out if they're idle for
too long.

.. _`FTPHost.keep_alive`: `keep_alive`_

Some more details:

- A kind of "straightforward" way of keeping the main connection alive
  would be to call ``ftp_host.getcwd()``. However, this doesn't work
  because ftputil caches the current directory and returns it without
  actually contacting the server. That's the main reason why there's
  a ``keep_alive`` method since it calls ``pwd`` on the FTP connection
  (i. e. the session object), which isn't a public attribute.

- Some servers define not only an idle timeout but also a transfer
  timeout. This means the connection times out unless there's some
  transfer on the data channel for this connection. So ftputil's
  ``keep_alive`` doesn't prevent this timeout, but an
  ``ftp_host.listdir(ftp_host.curdir)`` call should do it. However,
  this transfers the data for the whole directory listing which might
  take some time if the directory has many entries.

Bottom line: If you can, you should organize your FTP actions so that
you finish everything before a timeout happens.


Writing directory parsers
-------------------------

``ftputil`` recognizes the two most widely-used FTP directory formats,
Unix and MS style, and adjusts itself automatically. Almost every FTP
server uses one of these formats.

However, if your server uses a format which is different from the two
provided by ``ftputil``, you can plug in a custom parser with a single
method call and have ``ftputil`` use this parser.

For this, you need to write a parser class by inheriting from the
class ``Parser`` in the ``ftputil.stat`` module. Here's an example::

    import ftputil.error
    import ftputil.stat

    class XyzParser(ftputil.stat.Parser):
        """
        Parse the default format of the FTP server of the XYZ
        corporation.
        """

        def parse_line(self, line, time_shift=0.0):
            """
            Parse a `line` from the directory listing and return a
            corresponding `StatResult` object. If the line can't
            be parsed, raise `ftputil.error.ParserError`.

            The `time_shift` argument can be used to fine-tune the
            parsing of dates and times. See the class
            `ftputil.stat.UnixParser` for an example.
            """
            # Split the `line` argument and examine it further; if
            # something goes wrong, raise an `ftputil.error.ParserError`.
            ...
            # Make a `StatResult` object from the parts above.
            stat_result = ftputil.stat.StatResult(...)
            # `_st_name`, `_st_target` and `_st_mtime_precision` are optional.
            stat_result._st_name = ...
            stat_result._st_target = ...
            stat_result._st_mtime_precision = ...
            return stat_result

        # Define `ignores_line` only if the default in the base class
        # doesn't do enough!
        def ignores_line(self, line):
            """
            Return a true value if the line should be ignored. For
            example, the implementation in the base class handles
            lines like "total 17". On the other hand, if the line
            should be used for stat'ing, return a false value.
            """
            is_total_line = super(XyzParser, self).ignores_line(line)
            my_test = ...
            return is_total_line or my_test

A ``StatResult`` object is similar to the value returned by
`os.stat`_ and is usually built with statements like

::

    stat_result = StatResult(
                    (st_mode, st_ino, st_dev, st_nlink, st_uid,
                     st_gid, st_size, st_atime, st_mtime, st_ctime))
    stat_result._st_name = ...
    stat_result._st_target = ...
    stat_result._st_mtime_precision = ...

with the arguments of the ``StatResult`` constructor described in
the following table.

===== =================== ============ =================== =======================
Index Attribute           os.stat type ``StatResult`` type Notes
===== =================== ============ =================== =======================
0     st_mode             int          int
1     st_ino              long         long
2     st_dev              long         long
3     st_nlink            int          int
4     st_uid              int          str                 usually only available as string
5     st_gid              int          str                 usually only available as string
6     st_size             long         long
7     st_atime            int/float    float
8     st_mtime            int/float    float
9     st_ctime            int/float    float
\-    _st_name            \-           str                 file name without directory part
\-    _st_target          \-           str                 link target (may be absolute or relative)
\-    _st_mtime_precision \-           int                 ``st_mtime`` precision in seconds
===== =================== ============ =================== =======================

If you can't extract all the desirable data from a line (for
example, the MS format doesn't contain any information about the
owner of a file), set the corresponding values in the ``StatResult``
instance to ``None``.

Parser classes can use several helper methods which are defined in
the class ``Parser``:

- ``parse_unix_mode`` parses strings like "drwxr-xr-x" and returns
  an appropriate ``st_mode`` integer value.

- ``parse_unix_time`` returns a float number usable for the
  ``st_...time`` values by parsing arguments like "Nov"/"23"/"02:33" or
  "May"/"26"/"2005". Note that the method expects the timestamp string
  already split at whitespace.

- ``parse_ms_time`` parses arguments like "10-23-01"/"03:25PM" and
  returns a float number like from ``time.mktime``. Note that the
  method expects the timestamp string already split at whitespace.

Additionally, there's an attribute ``_month_numbers`` which maps
lowercase three-letter month abbreviations to integers.

For more details, see the two "standard" parsers ``UnixParser`` and
``MSParser`` in the module ``ftputil/stat.py``.

To actually *use* the parser, call the method `set_parser`_ of the
``FTPHost`` instance.

If you can't write a parser or don't want to, please ask on the
`ftputil mailing list`_. Possibly someone has already written a parser
for your server or can help with it.


FAQ / Tips and tricks
---------------------

Where can I get the latest version?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

See the `download page`_. Announcements will be sent to the `mailing
list`_. Announcements on major updates will also be posted to the
newsgroup `comp.lang.python.announce`_ .

.. _`download page`: http://ftputil.sschwarzer.net/download
.. _`mailing list`: http://ftputil.sschwarzer.net/mailinglist
.. _`comp.lang.python.announce`: news:comp.lang.python.announce

Is there a mailing list on ``ftputil``?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Yes, please visit http://ftputil.sschwarzer.net/mailinglist to
subscribe or read the archives.

Though you can *technically* post without subscribing first I can't
recommend it: The mails from non-subscribers have to be approved by
me and because the arriving mails contain *lots* of spam, I rarely go
through these mails.

I found a bug! What now?
~~~~~~~~~~~~~~~~~~~~~~~~

Before reporting a bug, make sure that you already read this manual
and tried the `latest version`_ of ``ftputil``. There the bug might
have already been fixed.

.. _`latest version`: http://ftputil.sschwarzer.net/download

Please see http://ftputil.sschwarzer.net/issuetrackernotes for
guidelines on entering a bug in ``ftputil``'s ticket system. If you
are unsure if the behaviour you found is a bug or not, you should write
to the `ftputil mailing list`_. In *either* case you *must not*
include confidential information (user id, password, file names, etc.)
in the problem report! Be careful!

Does ``ftputil`` support SSL/TLS?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

``ftputil`` has no *built-in* SSL/TLS support.

On the other hand, there are two ways to get TLS support with
ftputil:

- In Python 2.7 and Python 3.2 and up, the ``ftplib`` library has a
  class ``FTP_TLS`` that you can use for the ``session_factory``
  keyword argument in the ``FTPHost`` constructor. You can't use the
  class directly though if you need additional setup code in
  comparison to ``ftplib.FTP``, for example calling ``prot_p``, to
  secure the data connection. On the other hand,
  `ftputil.session.session_factory`_ can be used to create a custom
  session factory.
  
  If you have other requirements that ``session_factory`` can't
  fulfill, you may create your own session factory by inheriting from
  ``ftplib.FTP_TLS``::

    import ftplib

    import ftputil


    class FTPTLSSession(ftplib.FTP_TLS):

        def __init__(self, host, user, password):
            ftplib.FTP_TLS.__init__(self)
            self.connect(host, port)
            self.login(user, password)
            # Set up encrypted data connection.
            self.prot_p()
            ...

    # Note the `session_factory` parameter. Pass the class, not
    # an instance.
    with ftputil.FTPHost(server, user, password,
                         session_factory=FTPTLSSession) as ftp_host:
        # Use `ftp_host` as usual.
        ...

.. _`ftputil.session.session_factory`: `Session factories`_

- If you need to work with Python 2.6, you can use the
  ``ftpslib.FTP_TLS`` class from the M2Crypto_ project. Again, you
  can't use the class directly but need to use
  ``ftputil.session.session_factory`` or a recipe similar to that
  above.

  Unfortunately, ``M2Crypto.ftpslib.FTP_TLS`` (at least in version
  0.22.3) doesn't work correctly if you pass unicode strings to its
  methods. Since ``ftputil`` does exactly that at some point (even if
  you used byte strings in ``ftputil`` calls) you need a workaround in
  the session factory class::

    import M2Crypto

    import ftputil
    import ftputil.tool


    class M2CryptoSession(M2Crypto.ftpslib.FTP_TLS):

        def __init__(self, host, user, password):
            M2Crypto.ftpslib.FTP_TLS.__init__(self)
            # Change the port number if needed.
            self.connect(host, 21)
            self.auth_tls()
            self.login(user, password)
            self.prot_p()
            self._fix_socket()
            ...

        def _fix_socket(self):
            """
            Change the socket object so that arguments to `sendall`
            are converted to byte strings before being used.
            """
            original_sendall = self.sock.sendall
            # Bound method, therefore no `self` argument.
            def sendall(data):
                data = ftputil.tool.as_bytes(data)
                return original_sendall(data)
            self.sock.sendall = sendall

    # Note the `session_factory` parameter. Pass the class, not
    # an instance.
    with ftputil.FTPHost(server, user, password,
                         session_factory=M2CryptoSession) as ftp_host:
        # Use `ftp_host` as usual.
        ...

  That said, ``session_factory`` has this workaround built in, so
  normally you don't need to define the session factory yourself!

.. _M2Crypto: https://github.com/martinpaljak/M2Crypto

How do I connect to a non-default port?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

By default, an instantiated ``FTPHost`` object connects on the usual
FTP port. If you have to use a different port, refer to the section
`Session factories`_.

How do I set active or passive mode?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Please see the section `Session factories`_.

How can I debug an FTP connection problem?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

You can do this with a session factory. See `Session factories`_.

If you want to change the debug level only temporarily after the
connection is established, you can reach the `session object`_ as the
``_session`` attribute of the ``FTPHost`` instance and call
``_session.set_debuglevel``. Note that the ``_session`` attribute
should *only* be accessed for debugging. Calling arbitrary
``ftplib.FTP`` methods on the session object may *cause* bugs!

.. _`session object`: `Session factories`_

Conditional upload/download to/from a server in a different time zone
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

You may find that ``ftputil`` uploads or downloads files
unnecessarily, or not when it should. This can happen when the FTP
server is in a different time zone than the client on which
``ftputil`` runs. Please see the section on `time zone correction`_.
It may even be sufficient to call `synchronize_times`_.

When I use ``ftputil``, all I get is a ``ParserError`` exception
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The FTP server you connect to may use a directory format that
``ftputil`` doesn't understand. You can either write and
`plug in an own parser`_ or ask on the `mailing list`_ for
help.

.. _`plug in an own parser`: `Writing directory parsers`_

``isdir``, ``isfile`` or ``islink`` incorrectly return ``False``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Like Python's counterparts under `os.path`_, ``ftputil``'s methods
return ``False`` if they can't find the given path.

Probably you used ``listdir`` on a directory and called ``is...()`` on
the returned names. But if the argument for ``listdir`` wasn't the
current directory, the paths won't be found and so all ``is...()``
variants will return ``False``.

I don't find an answer to my problem in this document
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Please send an email with your problem report or question to the
`ftputil mailing list`_, and we'll see what we can do for you. :-)


Bugs and limitations
--------------------

- ``ftputil`` needs at least Python 2.6 to work.

- Whether ``ftputil`` "sees" "hidden" directory and file names (i. e.
  names starting with a dot) depends on the configuration of the FTP
  server. See `Hidden files and directories`_ for details.

- Due to the implementation of ``lstat`` it can not return a sensible
  value for the root directory ``/`` though stat'ing entries *in* the
  root directory isn't a problem. If you know an implementation that
  can do this, please let me know. The root directory is handled
  appropriately in ``FTPHost.path.exists/isfile/isdir/islink``, though.

- In multithreaded programs, you can have each thread use one or more
  ``FTPHost`` instances as long as no instance is shared with other
  threads.

- Currently, it is not possible to continue an interrupted upload or
  download. Contact me if this causes problems for you.

- There's exactly one cache for ``lstat`` results for each ``FTPHost``
  object, i. e. there's no sharing of cache results determined by
  several ``FTPHost`` objects. See `Local caching of file system
  information`_ for the reasons.


Files
-----

If not overwritten via installation options, the ``ftputil`` files
reside in the ``ftputil`` package. There's also documentation in
`reStructuredText`_ and in HTML format. The locations of these
files after installation is system-dependent.

.. _`reStructuredText`: http://docutils.sourceforge.net/rst.html

The files ``test_*.py`` and ``mock_ftplib.py`` are for unit-testing.
If you only *use* ``ftputil``, i. e. *don't* modify it, you can
delete these files.


References
----------

- Mackinnon T, Freeman S, Craig P. 2000. `Endo-Testing:
  Unit Testing with Mock Objects`_.

- Postel J, Reynolds J. 1985. `RFC 959 - File Transfer Protocol (FTP)`_.

- Van Rossum G et al. 2013. `Python Library Reference`_.

.. _`Endo-Testing: Unit Testing with Mock Objects`:
   http://www.connextra.com/aboutUs/mockobjects.pdf
.. _`RFC 959 - File Transfer Protocol (FTP)`: http://www.ietf.org/rfc/rfc959.txt
.. _`Python Library Reference`: https://docs.python.org/library/index.html


Authors
-------

``ftputil`` is written by Stefan Schwarzer
<sschwarzer@sschwarzer.net> and contributors (see
``doc/contributors.txt``).

The original ``lrucache`` module was written by Evan Prodromou
<evan@prodromou.name>.

Feedback is appreciated. :-)