Upload script #1
Conversation
|
@marcofucci @SteveMarshall give it a review, although it's maybe not a candidate for merging yet as we are still uncertain about how we will be retrieving the files depending on what is going on with the existing MoJ thing |
| from . import settings | ||
|
|
||
| ref_pattern = re.compile('([A-Za-z][0-9]{4}[A-Za-z]{2}).*?([0-9]{1,2}).*?([0-9]{1,2}).*?([0-9]{2,4})') | ||
| file_pattern = re.compile('YO1A\.REC\.#D\.(%(code)s)\.D([0-9]{6})' % {'code': settings.ACCOUNT_CODE}) |
There was a problem hiding this comment.
I wonder if it's worth breaking these out using re.VERBOSE and adding some comments to describe the format more clearly?
|
I wonder if there's some way we can do some unit/integration testing of the main uploader? Might be nice to have some guarantees it's doing what it's supposed to… (Though all the code looks good and correct to me, mind you…) |
|
Re: testing, as you probably realised I didn't do much in the way of unit testing because it's basically glue code - it would end up just being data passed between mock objects. Integration testing once we have access to an RBS test environment is probably a good idea though. |
| os.mkdir(settings.DS_NEW_FILES_DIR) | ||
|
|
||
| # check if we recorded date of last file downloaded | ||
| last_date = None |
There was a problem hiding this comment.
Would be worth setting a default, a value really far in the past. This is because the check https://github.com/ministryofjustice/money-to-prisoners-transaction-uploader/pull/1/files#diff-8d4a24f3bb40368c078fa6bfdde77673R56 wouldn't work properly otherwise I think
There was a problem hiding this comment.
Well spotted, although personally I would prefer changing the check to:
if last_date is None or date > last_date
|
This looks good to me! @ibrechin wanna rebase? |
This script downloads files which fit a certain format to a temp directory, and then parses them to extract the relevant transaction details. These transactions are then uploaded to an API. Options (listed in settings.py) need to be set as env variables on the host.
Create an initial upload script
This script will: