The current task I have been assigned to do is create a program which can download files from an FTP server, parse the file (in this case XML), and insert the data into the PostgreSQL database. I have no previous knowledge of working with FTP servers outside of a SEC601 class which covered FTP and how exploitable it was. After searching ddg for FTP libraries in Python3 I quickly found ftplib a simple ftp client which had all the functionality I needed for the job. In this case the functionality being mainly the FTP.retrbinary (which allows us to retrieve files from the server) and FTP.nlst (allows us to view all the files in the current directory). Time to make the first iteration of what I am going to call FTP Ingestor (FTPI). I really like test driven development so I will be getting into the habit of writing tests as I code.
The download function is going to be what I am making first since we need the files in order to actually do things with them. The code itself is simple enough (again using the 2 functions we discussed above) which can be seen below.
for f in self.ftpconn.nlst():
fHandler = open(‘%s/%s’ % (self.filespath, f), ‘wb’)
self.ftpconn.retrbinary(‘RETR %s’ % (f), fHandler.write)
The for loop is pretty self explanatory, we are looping over every file return from the nlst() method. fHandler is what we are going to be using for the callback which is the second parameter required for retrbinary. This one just creates a new file under the the name of the path joined from the values of self.filespath and f. Finally we have the retrbinary call which takes we pass the appropriate RETR command (‘RETR %s’ % (f) ~ passing the current file) and the write function of fHandler (which executed for each block of data received), I would mention fHandler.close() but it should be pretty self explanatory as well ~ it closes the file. Now to write a quick test and run this bad boy.
For this entire architecture I want to stick to the standard libraries that way none of it is reliant on a 3rd party developer (unless it is a crucial open source technology), so fittingly I will be using unittest for the TDD side of things. I haven’t used this library before, but after a quick browse through the documentation it looks to be one that is easy to pick up and streamlined for TDD. I have done a software testing class during my bachelor so I am a little experienced in unit tests, but I wrote them in C# at the time so this will be a nice refresher.