Feature/azure container

[AlexAxthelm]

Contains infrastrucure to Build a docker image suitable to run on Azure Container Instances, as well as a deploy template.

Coincidentally:
Closes #3
Closes #17

[jdhoffa]

I think I'm just out of sync from being away, or maybe you and CJ already discussed this in person, but could you ELI5 this in the PR description?

I'm having a hard time following what this does.

That said, will try to test it in my machine later today, I wanted to try to get data prep running anyway to make sure I can still do it 😀

[cjyetman]

I feel like completely swapping out one dependency for another ({rlog} to {logger}) should be in a separate PR, not one titled "Feature/azure container".

[AlexAxthelm]

@jdhoffa still a fair bit of testing to do on my side, but while it's cooking, I'll be writing docs for it.

[jdhoffa]

Sweet sounds good. Let me know when it's ready for a test run!

Also agree with CJ. Very happy to swap rlog with logger, but prob deserves it's own PR.

[AlexAxthelm]

Switching to logger available as #76. Basically the same changes as have been implemented here, but isolated.

[AlexAxthelm]

@cjyetman @jdhoffa Following the discussion in #73 I'm thinking that changing the strategy for this PR makes sense.

I'd been not trying to change the logic in run_data_preparation.R (just add logging), or touch config.yml too much, which is why there's two scripts run in this PR: copy_raw_data.R (in the ACI directory), and run_data_preparation.R. Where the first one copies the raw data files from Azure files to what is the "inputs" directory for run_data_preparation.R.

I'm now thinking that the way to address the "how do we track the source files" question might best be addressed by including the paths to the raw data files in the config (see example below), and including a copy_inputs option in the config, so that if we want to isolate what were the inputs for this particular set of outputs, we can.

Sketch:

Here's a (simplified) sketch of what I'm thinking:

#config.yaml
default:
  copy_inputs: false
  # these two replace the data_prep_inputs_path:
  factset_data_path: /inputs
  asset_impact_data_path: /inputs

2022Q4_CICD
  copy_inputs: true
  factset_data_path: "/mnt/factset-extracted/factset-pacta_timestamp-20221231T000000Z_pulled-20231221T195325Z"
  asset_impact_data_path: "/mnt/rawdata/AssetImpact"

# run_data_preparation.R
# Loading config...
masterdata_ownership_path <- file.path(asset_impact_data_path, masterdata_ownership_filename) #and friends
factset_financial_data_path <- file.path(factset_data_path, "factset_financial_data.rds") # and friends

# Do all the normal data_prep stuff

if (copy_inputs) {
  inputs_copy_path <- file.path(data_prep_outputs_path, "data_prep_inputs")
  dir.create(inputs_copy_path)
  file.copy(input_files, inputs_copy_path)
}

log_info("Done!)

[jdhoffa]

Sounds reasonable to me?

[cjyetman]

@AlexAxthelm is this superseded by one or more of the other recent PRs? can/should we close it?

[cjyetman]

@AlexAxthelm can this be closed now?

[AlexAxthelm]

Closing this, but not deleting the branch yet, since there's some elements from that that I'd like to bring into main, I just need to get some time to do that.